2024-09-19T06:29:43.001 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-19T06:29:43.025 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-19T06:29:43.118 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911756 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/suites/iogen}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7911756' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 6fad5dd73c93545e52f57f681e140a9c3529c11e owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1747 sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 sleep_before_teardown: 0 subset: 176/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 6fad5dd73c93545e52f57f681e140a9c3529c11e syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZdGX1pcogqo02UoeW/tlUAzEetaztMnXkCTumANDB2SeyjykIiHRiMfgJLDZTXvbpbVuaTD/WOWlGhCdi96Fg= smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEbngPd7H6CwiddIBL0SLPkuWUFiJSVRIdanIE0SsPvHvFdznOzEaaDY9gZegwn1LZv7eDn32mLSiUn0qyczww8= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF1H85XL/uI0K5gm9BV5xbArlmyK9CleWUQlmtdBWaATy9rRfCklJNqgjOpGZtV8SUUDiJlFBRhkgLr3wpN/mZU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_21:24:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-19T06:29:43.118 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa; will attempt to use it 2024-09-19T06:29:43.119 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_6fad5dd73c93545e52f57f681e140a9c3529c11e/qa/tasks 2024-09-19T06:29:43.119 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-19T06:29:43.121 INFO:teuthology.task.internal:Checking packages... 2024-09-19T06:29:43.143 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2f599a46ad10e37c929e5c43da2f6907ddcd1961' 2024-09-19T06:29:43.143 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-19T06:29:43.143 INFO:teuthology.packaging:ref: None 2024-09-19T06:29:43.143 INFO:teuthology.packaging:tag: None 2024-09-19T06:29:43.143 INFO:teuthology.packaging:branch: squid 2024-09-19T06:29:43.143 INFO:teuthology.packaging:sha1: 2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:29:43.144 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-19T06:29:43.365 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-363.g6fad5dd7 2024-09-19T06:29:43.367 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-19T06:29:43.375 INFO:teuthology.task.internal:no buildpackages task found 2024-09-19T06:29:43.375 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-19T06:29:43.416 INFO:teuthology.task.internal:Saving configuration 2024-09-19T06:29:43.435 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-19T06:29:43.450 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-19T06:29:43.475 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911756', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-19 06:24:05.503367', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZdGX1pcogqo02UoeW/tlUAzEetaztMnXkCTumANDB2SeyjykIiHRiMfgJLDZTXvbpbVuaTD/WOWlGhCdi96Fg='} 2024-09-19T06:29:43.500 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911756', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-19 06:24:05.504879', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEbngPd7H6CwiddIBL0SLPkuWUFiJSVRIdanIE0SsPvHvFdznOzEaaDY9gZegwn1LZv7eDn32mLSiUn0qyczww8='} 2024-09-19T06:29:43.522 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-18_21:24:17-fs-squid-distro-default-smithi/7911756', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-19 06:24:05.506141', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF1H85XL/uI0K5gm9BV5xbArlmyK9CleWUQlmtdBWaATy9rRfCklJNqgjOpGZtV8SUUDiJlFBRhkgLr3wpN/mZU='} 2024-09-19T06:29:43.522 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-19T06:29:43.531 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-19T06:29:43.532 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-19T06:29:43.532 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-19T06:29:43.532 INFO:teuthology.run_tasks:Running task console_log... 2024-09-19T06:29:43.648 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6d5223fbe0>, signals=[15]) 2024-09-19T06:29:43.648 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-19T06:29:43.655 INFO:teuthology.task.internal:Opening connections... 2024-09-19T06:29:43.656 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-09-19T06:29:43.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:29:43.739 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-09-19T06:29:43.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:29:43.813 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-09-19T06:29:43.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:29:43.889 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-19T06:29:43.898 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-09-19T06:29:43.917 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-09-19T06:29:43.917 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-09-19T06:29:43.975 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-09-19T06:29:43.975 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-09-19T06:29:43.975 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-19T06:29:43.976 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-19T06:29:43.977 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-09-19T06:29:43.996 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-09-19T06:29:44.015 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-09-19T06:29:44.015 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-09-19T06:29:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-09-19T06:29:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-09-19T06:29:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-09-19T06:29:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-09-19T06:29:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-09-19T06:29:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-09-19T06:29:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-19T06:29:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-09-19T06:29:44.074 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-09-19T06:29:44.074 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-19T06:29:44.074 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-09-19T06:29:44.074 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-19T06:29:44.074 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-19T06:29:44.074 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-19T06:29:44.074 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-09-19T06:29:44.096 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-09-19T06:29:44.114 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-09-19T06:29:44.114 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-09-19T06:29:44.173 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-09-19T06:29:44.173 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2024-09-19T06:29:44.173 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-09-19T06:29:44.173 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-09-19T06:29:44.173 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2024-09-19T06:29:44.173 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2024-09-19T06:29:44.173 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-19T06:29:44.173 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-09-19T06:29:44.174 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2024-09-19T06:29:44.174 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-19T06:29:44.174 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-09-19T06:29:44.174 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-19T06:29:44.174 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-19T06:29:44.174 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-19T06:29:44.174 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-09-19T06:29:44.195 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-19T06:29:44.210 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-19T06:29:44.251 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-19T06:29:44.251 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-09-19T06:29:44.254 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-09-19T06:29:44.256 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-09-19T06:29:44.272 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-19T06:29:44.283 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-19T06:29:44.284 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-09-19T06:29:44.311 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-09-19T06:29:44.314 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-09-19T06:29:44.340 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-19T06:29:44.423 INFO:teuthology.run_tasks:Running task kernel... 2024-09-19T06:29:44.446 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-19T06:29:44.446 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-09-19T06:29:44.446 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-09-19T06:29:44.447 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-19T06:29:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T06:29:44.465 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-19T06:29:44.644 DEBUG:teuthology.task.kernel:sha1 for client.0 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-19T06:29:44.644 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-19T06:29:44.644 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-19T06:29:44.663 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-503.el9.x86_64 2024-09-19T06:29:44.663 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-19T06:29:44.663 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-19T06:29:44.664 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum upgrade -y linux-firmware 2024-09-19T06:29:47.143 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-19T06:29:55.627 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 20 MB 00:07 2024-09-19T06:30:00.053 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-19T06:30:01.704 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-19T06:30:02.724 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-19T06:30:08.130 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-19T06:30:10.143 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-19T06:30:10.153 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-19T06:30:10.153 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-09-19T06:30:10.153 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-19T06:30:10.153 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-09-19T06:30:10.154 INFO:teuthology.orchestra.run.smithi114.stdout: linux-firmware noarch 20240905-146.el9 baseos 388 M 2024-09-19T06:30:10.154 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:30:10.154 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-19T06:30:10.154 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-09-19T06:30:10.154 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 1 Package 2024-09-19T06:30:10.154 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:30:10.154 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 388 M 2024-09-19T06:30:10.154 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-19T06:30:22.081 INFO:teuthology.orchestra.run.smithi114.stdout:linux-firmware-20240905-146.el9.noarch.rpm 33 MB/s | 388 MB 00:11 2024-09-19T06:30:22.081 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-19T06:30:22.082 INFO:teuthology.orchestra.run.smithi114.stdout:Total 33 MB/s | 388 MB 00:11 2024-09-19T06:30:25.200 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-19T06:30:25.367 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-19T06:30:25.367 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-19T06:30:28.631 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-19T06:30:28.632 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-19T06:30:37.909 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-19T06:30:38.589 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : linux-firmware-20240905-146.el9.noarch 1/2 2024-09-19T06:30:38.671 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : linux-firmware-20240716-145.el9.noarch 2/2 2024-09-19T06:30:39.260 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: linux-firmware-20240716-145.el9.noarch 2/2 2024-09-19T06:30:39.261 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : linux-firmware-20240905-146.el9.noarch 1/2 2024-09-19T06:30:40.038 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 2/2 2024-09-19T06:30:40.039 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:30:40.039 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-09-19T06:30:40.039 INFO:teuthology.orchestra.run.smithi114.stdout: linux-firmware-20240905-146.el9.noarch 2024-09-19T06:30:40.039 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:30:40.039 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-19T06:30:40.413 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-19T06:30:40.414 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-19T06:30:40.561 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-19T06:30:40.561 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-19T06:30:40.564 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi114.front.sepia.ceph.com to complete... 2024-09-19T06:30:46.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19 06:30:46 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-19T06:30:46.895 INFO:teuthology.orchestra.run.smithi114.stderr:FINISHED --2024-09-19 06:30:46-- 2024-09-19T06:30:46.895 INFO:teuthology.orchestra.run.smithi114.stderr:Total wall clock time: 6.3s 2024-09-19T06:30:46.895 INFO:teuthology.orchestra.run.smithi114.stderr:Downloaded: 1 files, 343M in 6.0s (57.0 MB/s) 2024-09-19T06:30:46.898 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-19T06:30:46.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-19T06:30:48.953 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying... ######################################## 2024-09-19T06:30:49.010 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing... ######################################## 2024-09-19T06:30:49.069 INFO:teuthology.orchestra.run.smithi114.stdout:Updating / installing... 2024-09-19T06:30:57.054 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-19T06:31:37.370 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-19T06:31:37.370 DEBUG:teuthology.orchestra.run.smithi114:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-19T06:31:37.464 INFO:teuthology.orchestra.run.smithi114.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:37.464 INFO:teuthology.orchestra.run.smithi114.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:37.464 INFO:teuthology.orchestra.run.smithi114.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:37.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:37.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-19T06:31:37.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-19T06:31:37.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-19T06:31:37.464 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-19T06:31:37.465 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-19T06:31:37.466 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-19T06:31:37.467 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-19T06:31:37.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-19T06:31:37.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-19T06:31:37.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-19T06:31:37.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-19T06:31:37.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-19T06:31:37.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-19T06:31:37.468 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-19T06:31:37.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-19T06:31:37.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-19T06:31:37.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-19T06:31:37.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-19T06:31:37.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-19T06:31:37.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-19T06:31:37.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-19T06:31:37.469 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-19T06:31:37.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-19T06:31:37.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-19T06:31:37.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-19T06:31:37.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-19T06:31:37.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-19T06:31:37.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-19T06:31:37.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-19T06:31:37.470 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-19T06:31:37.471 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-19T06:31:37.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-19T06:31:37.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-19T06:31:37.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-19T06:31:37.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-19T06:31:37.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-19T06:31:37.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-19T06:31:37.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-19T06:31:37.472 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-19T06:31:37.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-19T06:31:37.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-19T06:31:37.473 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-19T06:31:37.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-19T06:31:37.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-19T06:31:37.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-19T06:31:37.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-19T06:31:37.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-19T06:31:37.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-19T06:31:37.476 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-19T06:31:37.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-19T06:31:37.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-19T06:31:37.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-19T06:31:37.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-19T06:31:37.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-19T06:31:37.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-19T06:31:37.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-19T06:31:37.477 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-19T06:31:37.478 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-19T06:31:37.479 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-19T06:31:37.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-19T06:31:37.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-19T06:31:37.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-19T06:31:37.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-19T06:31:37.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-19T06:31:37.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-19T06:31:37.480 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-19T06:31:37.481 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-19T06:31:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-19T06:31:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-19T06:31:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-19T06:31:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-19T06:31:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-19T06:31:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-19T06:31:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-19T06:31:37.482 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-19T06:31:37.483 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-19T06:31:37.484 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-19T06:31:37.485 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-19T06:31:37.486 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-19T06:31:37.487 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-19T06:31:37.488 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-19T06:31:37.489 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-19T06:31:37.490 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-19T06:31:37.491 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-19T06:31:37.492 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-19T06:31:37.493 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-19T06:31:37.494 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-19T06:31:37.495 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-19T06:31:37.496 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-19T06:31:37.497 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-19T06:31:37.498 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-19T06:31:37.499 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-19T06:31:37.500 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-19T06:31:37.501 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-19T06:31:37.502 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-19T06:31:37.503 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-19T06:31:37.503 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-19T06:31:37.503 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-19T06:31:37.503 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-19T06:31:37.503 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-19T06:31:37.503 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-19T06:31:37.504 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-19T06:31:37.505 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-19T06:31:37.506 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-19T06:31:37.507 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-19T06:31:37.508 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-19T06:31:37.509 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-19T06:31:37.510 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-19T06:31:37.511 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-19T06:31:37.512 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-19T06:31:37.513 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-19T06:31:37.514 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-19T06:31:37.515 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-19T06:31:37.516 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-19T06:31:37.517 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-19T06:31:37.518 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-19T06:31:37.519 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-19T06:31:37.520 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-19T06:31:37.521 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-19T06:31:37.522 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-19T06:31:37.523 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-19T06:31:37.524 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-19T06:31:37.525 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-19T06:31:37.526 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-19T06:31:37.527 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-19T06:31:37.528 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-19T06:31:37.528 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-19T06:31:37.528 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-19T06:31:37.528 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-19T06:31:37.528 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-19T06:31:37.528 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-19T06:31:37.529 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-19T06:31:37.529 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-19T06:31:37.529 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-19T06:31:37.529 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-19T06:31:37.529 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-19T06:31:37.530 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-19T06:31:37.530 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-19T06:31:37.530 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-19T06:31:37.530 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-19T06:31:37.530 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-19T06:31:37.530 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-19T06:31:37.531 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-19T06:31:37.531 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-19T06:31:37.531 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-19T06:31:37.531 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-19T06:31:37.531 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-19T06:31:37.531 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-19T06:31:37.532 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-19T06:31:37.532 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-19T06:31:37.532 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-19T06:31:37.532 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-19T06:31:37.532 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-19T06:31:37.533 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-19T06:31:37.533 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-19T06:31:37.533 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-19T06:31:37.533 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-19T06:31:37.533 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-19T06:31:37.534 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-19T06:31:37.534 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-19T06:31:37.534 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-19T06:31:37.534 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-19T06:31:37.534 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-19T06:31:37.534 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-19T06:31:37.535 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-19T06:31:37.535 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-19T06:31:37.535 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-19T06:31:37.535 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-19T06:31:37.535 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-19T06:31:37.535 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-19T06:31:37.536 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-19T06:31:37.536 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-19T06:31:37.536 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-19T06:31:37.536 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-19T06:31:37.536 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-19T06:31:37.537 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-19T06:31:37.537 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-19T06:31:37.537 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-19T06:31:37.537 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-19T06:31:37.537 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-19T06:31:37.537 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-19T06:31:37.538 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-19T06:31:37.538 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-19T06:31:37.538 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-19T06:31:37.538 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-19T06:31:37.538 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-19T06:31:37.539 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-19T06:31:37.539 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-19T06:31:37.539 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-19T06:31:37.539 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-19T06:31:37.539 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-19T06:31:37.539 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-19T06:31:37.540 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-19T06:31:37.541 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-19T06:31:37.542 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-19T06:31:37.543 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-19T06:31:37.544 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-19T06:31:37.545 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-19T06:31:37.546 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-19T06:31:37.547 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-19T06:31:37.548 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-19T06:31:37.549 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-19T06:31:37.550 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-19T06:31:37.551 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-19T06:31:37.552 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-19T06:31:37.553 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-19T06:31:37.554 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-19T06:31:37.555 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-19T06:31:37.556 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-19T06:31:37.557 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-19T06:31:37.558 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-19T06:31:37.559 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-19T06:31:37.560 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-19T06:31:37.561 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-19T06:31:37.562 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-19T06:31:37.563 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-19T06:31:37.564 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-19T06:31:37.565 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-19T06:31:37.566 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-19T06:31:37.567 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-19T06:31:37.568 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-19T06:31:37.569 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-19T06:31:37.570 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-19T06:31:37.571 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-19T06:31:37.572 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-19T06:31:37.573 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-19T06:31:37.574 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-19T06:31:37.575 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-19T06:31:37.576 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-19T06:31:37.577 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-19T06:31:37.578 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-19T06:31:37.579 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-19T06:31:37.580 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-19T06:31:37.581 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-19T06:31:37.582 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-19T06:31:37.583 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-19T06:31:37.584 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-19T06:31:37.585 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-19T06:31:37.586 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-19T06:31:37.587 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-19T06:31:37.588 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-19T06:31:37.589 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-19T06:31:37.590 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-19T06:31:37.591 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-19T06:31:37.592 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-19T06:31:37.593 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-19T06:31:37.594 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-19T06:31:37.595 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-19T06:31:37.596 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-19T06:31:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-19T06:31:37.598 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-19T06:31:37.599 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-19T06:31:37.600 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-19T06:31:37.601 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-19T06:31:37.602 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-19T06:31:37.603 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-19T06:31:37.604 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-19T06:31:37.605 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-19T06:31:37.606 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-19T06:31:37.607 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-19T06:31:37.608 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-19T06:31:37.609 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-19T06:31:37.610 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-19T06:31:37.611 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-19T06:31:37.612 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-19T06:31:37.613 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-19T06:31:37.614 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-19T06:31:37.615 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-19T06:31:37.616 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-19T06:31:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-19T06:31:37.618 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-19T06:31:37.619 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-19T06:31:37.620 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-19T06:31:37.621 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-19T06:31:37.622 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-19T06:31:37.623 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-19T06:31:37.624 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-19T06:31:37.625 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-19T06:31:37.626 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-19T06:31:37.626 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-19T06:31:37.626 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-19T06:31:37.626 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-19T06:31:37.626 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-19T06:31:37.626 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-19T06:31:37.626 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-19T06:31:37.626 INFO:teuthology.orchestra.run.smithi114.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-19T06:31:37.630 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:37.630 DEBUG:teuthology.orchestra.run.smithi114:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-19T06:31:37.657 INFO:teuthology.orchestra.run.smithi114.stdout:postinstall scriptlet (using /bin/sh): 2024-09-19T06:31:37.657 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-19T06:31:37.657 INFO:teuthology.orchestra.run.smithi114.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-19T06:31:37.657 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-09-19T06:31:37.657 INFO:teuthology.orchestra.run.smithi114.stdout:for file in vmlinuz System.map config; do 2024-09-19T06:31:37.658 INFO:teuthology.orchestra.run.smithi114.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-19T06:31:37.658 INFO:teuthology.orchestra.run.smithi114.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-19T06:31:37.658 INFO:teuthology.orchestra.run.smithi114.stdout: fi 2024-09-19T06:31:37.658 INFO:teuthology.orchestra.run.smithi114.stdout:done 2024-09-19T06:31:37.658 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-19T06:31:37.659 INFO:teuthology.orchestra.run.smithi114.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-19T06:31:37.659 INFO:teuthology.orchestra.run.smithi114.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-19T06:31:37.659 INFO:teuthology.orchestra.run.smithi114.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-19T06:31:37.659 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-09-19T06:31:37.659 INFO:teuthology.orchestra.run.smithi114.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-19T06:31:37.659 INFO:teuthology.orchestra.run.smithi114.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:37.660 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-09-19T06:31:37.660 INFO:teuthology.orchestra.run.smithi114.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-19T06:31:37.660 INFO:teuthology.orchestra.run.smithi114.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-19T06:31:37.660 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:37.660 INFO:teuthology.orchestra.run.smithi114.stdout:fi 2024-09-19T06:31:37.662 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-09-19T06:31:37.747 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-09-19T06:31:37.748 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-09-19T06:31:37.748 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-09-19T06:31:37.748 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 82.el9 2024-09-19T06:31:37.748 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-09-19T06:31:37.748 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-19T06:31:37.748 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-09-19T06:31:37.748 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8274670 2024-09-19T06:31:37.748 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-19T06:31:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-19T06:31:37.750 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-19T06:31:37.750 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-09-19T06:31:37.750 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:31:37.750 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-09-19T06:31:37.751 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-19T06:31:37.751 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:37.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-19T06:31:38.377 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-09-19T06:31:39.724 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-19T06:31:39.755 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-09-19T06:31:39.758 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-09-19T06:31:39.773 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.EvLek0R3wW 2024-09-19T06:31:39.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EvLek0R3wW 2024-09-19T06:31:39.840 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.EvLek0R3wW 2024-09-19T06:31:39.987 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.EvLek0R3wW is 6KB 2024-09-19T06:31:40.007 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.EvLek0R3wW 2024-09-19T06:31:40.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-09-19T06:31:40.090 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-19T06:31:40.090 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-19T06:31:40.090 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-19T06:31:40.090 INFO:teuthology.orchestra.run.smithi114.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-19T06:31:40.091 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:31:40.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-09-19T06:32:10.230 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-19T06:32:10.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-19T06:32:10.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:32:28.737 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-19T06:32:37.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-19T06:32:37.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:32:40.797 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-19T06:32:52.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-19T06:32:52.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:33:11.229 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-09-19T06:33:26.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-19T06:33:26.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:33:26.697 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-19T06:33:27.179 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-19T06:33:27.180 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-19T06:33:27.180 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-09-19T06:33:27.215 INFO:teuthology.orchestra.run.smithi114.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:33:27.215 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-19T06:33:27.215 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-19T06:33:27.215 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-19T06:33:28.217 DEBUG:teuthology.parallel:result is None 2024-09-19T06:33:28.217 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-19T06:33:28.229 INFO:teuthology.task.internal:Creating test directory... 2024-09-19T06:33:28.229 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-19T06:33:28.234 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-19T06:33:28.237 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-19T06:33:28.265 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-19T06:33:28.272 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-19T06:33:28.308 INFO:teuthology.task.internal:Creating archive directory... 2024-09-19T06:33:28.308 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-19T06:33:28.313 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-19T06:33:28.316 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-19T06:33:28.408 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-19T06:33:28.416 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-19T06:33:28.416 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-19T06:33:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T06:33:28.434 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-19T06:33:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T06:33:28.451 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-19T06:33:28.455 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-19T06:33:28.477 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-19T06:33:28.520 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T06:33:28.534 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T06:33:28.556 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T06:33:28.575 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T06:33:28.593 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T06:33:28.669 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-19T06:33:28.673 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-19T06:33:28.680 INFO:teuthology.task.internal:Configuring sudo... 2024-09-19T06:33:28.680 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-19T06:33:28.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-19T06:33:28.687 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-19T06:33:28.739 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-19T06:33:28.750 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-19T06:33:28.750 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-19T06:33:28.754 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-19T06:33:28.758 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-19T06:33:28.785 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T06:33:28.893 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T06:33:28.972 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T06:33:29.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T06:33:29.113 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:33:29.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-19T06:33:29.175 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T06:33:29.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T06:33:29.276 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T06:33:29.360 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T06:33:29.406 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:33:29.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-19T06:33:29.476 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T06:33:29.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-19T06:33:29.582 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T06:33:29.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-19T06:33:29.707 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:33:29.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-19T06:33:29.773 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-09-19T06:33:29.777 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-09-19T06:33:29.780 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-09-19T06:33:29.817 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T06:33:29.850 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T06:33:29.854 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-19T06:33:30.293 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-19T06:33:30.301 INFO:teuthology.task.internal:Starting timer... 2024-09-19T06:33:30.301 INFO:teuthology.run_tasks:Running task pcp... 2024-09-19T06:33:30.353 INFO:teuthology.run_tasks:Running task selinux... 2024-09-19T06:33:30.393 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-19T06:33:30.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-09-19T06:33:30.506 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-09-19T06:33:30.510 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-09-19T06:33:30.571 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-09-19T06:33:30.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-09-19T06:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-09-19T06:33:30.643 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-19T06:33:30.643 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-09-19T06:33:30.686 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-09-19T06:33:30.687 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-09-19T06:33:30.722 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-09-19T06:33:30.722 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-09-19T06:33:30.745 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-09-19T06:33:30.746 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2024-09-19T06:33:30.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T06:33:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T06:33:30.804 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T06:33:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T06:33:30.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-19T06:33:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-19T06:33:30.861 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-19T06:33:30.861 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-09-19T06:33:30.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-09-19T06:33:30.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-09-19T06:33:30.999 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-19T06:33:31.008 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-19T06:33:31.011 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-19T06:33:31.153 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-19T06:33:31.168 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-19T06:33:31.169 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi116.front.sepia.ceph.com,smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-19T06:38:05.214 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2024-09-19T06:38:05.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-19T06:38:05.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:38:05.305 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-09-19T06:38:05.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-09-19T06:38:05.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-09-19T06:38:05.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:38:05.494 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-09-19T06:38:05.573 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-09-19T06:38:05.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-19T06:38:05.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-19T06:38:05.651 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-09-19T06:38:05.735 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-19T06:38:05.735 INFO:teuthology.run_tasks:Running task clock... 2024-09-19T06:38:05.746 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-19T06:38:05.746 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-19T06:38:05.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T06:38:05.750 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-19T06:38:05.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T06:38:05.753 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-19T06:38:05.753 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-19T06:38:05.781 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-19T06:38:05.796 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-19T06:38:05.807 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-19T06:38:05.808 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-19T06:38:05.820 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-09-19T06:38:05.824 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-19T06:38:05.831 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-09-19T06:38:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-19T06:38:05.845 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-19T06:38:05.850 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-09-19T06:38:05.859 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-19T06:38:05.862 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-09-19T06:38:05.877 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-19T06:38:05.888 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-09-19T06:38:05.891 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-19T06:38:05.911 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-09-19T06:38:05.914 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-09-19T06:38:05.918 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T06:38:05.918 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-09-19T06:38:05.918 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:05.918 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:05.918 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:05.918 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:05.941 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-19T06:38:05.948 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-09-19T06:38:05.951 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T06:38:05.951 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-09-19T06:38:05.951 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:05.951 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:05.951 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:05.951 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:05.970 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-19T06:38:06.062 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-09-19T06:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-19T06:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-09-19T06:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:06.069 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-19T06:38:06.070 INFO:teuthology.run_tasks:Running task install... 2024-09-19T06:38:06.077 DEBUG:teuthology.task.install:project ceph 2024-09-19T06:38:06.077 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'}} 2024-09-19T06:38:06.078 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961'} 2024-09-19T06:38:06.078 INFO:teuthology.task.install:Using flavor: default 2024-09-19T06:38:06.083 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-19T06:38:06.083 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-19T06:38:06.083 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'tag': None, 'wait_for_package': False} 2024-09-19T06:38:06.084 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:38:06.086 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'tag': None, 'wait_for_package': False} 2024-09-19T06:38:06.086 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:38:06.087 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'tag': None, 'wait_for_package': False} 2024-09-19T06:38:06.087 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:38:06.234 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/ 2024-09-19T06:38:06.234 INFO:teuthology.task.install.rpm:Package version is 19.1.1-361.g2f599a46 2024-09-19T06:38:06.238 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/ 2024-09-19T06:38:06.238 INFO:teuthology.task.install.rpm:Package version is 19.1.1-361.g2f599a46 2024-09-19T06:38:06.241 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/ 2024-09-19T06:38:06.241 INFO:teuthology.task.install.rpm:Package version is 19.1.1-361.g2f599a46 2024-09-19T06:38:06.350 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-19T06:38:06.351 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:38:06.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-19T06:38:06.353 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-19T06:38:06.353 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:38:06.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-19T06:38:06.362 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-19T06:38:06.362 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:38:06.362 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-19T06:38:06.379 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-19T06:38:06.379 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f599a46ad10e37c929e5c43da2f6907ddcd1961/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-19T06:38:06.380 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-19T06:38:06.380 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f599a46ad10e37c929e5c43da2f6907ddcd1961/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-19T06:38:06.430 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-19T06:38:06.430 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f599a46ad10e37c929e5c43da2f6907ddcd1961/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-19T06:38:06.456 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-19T06:38:06.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-19T06:38:06.531 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-19T06:38:06.538 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-19T06:38:06.539 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-19T06:38:06.606 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-09-19T06:38:06.606 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-09-19T06:38:06.608 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-09-19T06:38:06.609 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-09-19T06:38:06.610 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-19T06:38:06.665 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-09-19T06:38:06.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-09-19T06:38:06.832 INFO:teuthology.orchestra.run.smithi118.stdout:45 files removed 2024-09-19T06:38:06.836 INFO:teuthology.orchestra.run.smithi116.stdout:45 files removed 2024-09-19T06:38:06.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-19T06:38:06.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-19T06:38:06.939 INFO:teuthology.orchestra.run.smithi114.stdout:45 files removed 2024-09-19T06:38:06.971 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-19T06:38:07.415 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 297 kB/s | 84 kB 00:00 2024-09-19T06:38:07.419 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 299 kB/s | 84 kB 00:00 2024-09-19T06:38:07.674 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 279 kB/s | 84 kB 00:00 2024-09-19T06:38:07.698 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-19T06:38:07.702 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-19T06:38:07.966 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-19T06:38:07.974 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-19T06:38:07.995 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 7.4 kB/s | 1.9 kB 00:00 2024-09-19T06:38:08.234 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-19T06:38:08.461 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.3 MB 00:00 2024-09-19T06:38:08.491 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-19T06:38:08.667 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 21 MB/s | 8.3 MB 00:00 2024-09-19T06:38:11.371 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-19T06:38:11.693 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 20 MB 00:02 2024-09-19T06:38:11.877 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 20 MB 00:02 2024-09-19T06:38:16.336 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-19T06:38:17.209 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-19T06:38:17.463 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.5 MB 00:02 2024-09-19T06:38:17.795 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 124 kB/s | 19 kB 00:00 2024-09-19T06:38:18.669 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-19T06:38:18.791 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-19T06:38:19.139 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-19T06:38:19.984 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-19T06:38:20.128 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-19T06:38:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-19T06:38:25.453 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-09-19T06:38:25.565 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-19T06:38:27.518 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.519 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.533 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-19T06:38:27.533 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.535 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-19T06:38:27.536 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.537 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-19T06:38:27.540 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.541 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.580 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.581 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.592 INFO:teuthology.orchestra.run.smithi118.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-19T06:38:27.593 INFO:teuthology.orchestra.run.smithi118.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.593 INFO:teuthology.orchestra.run.smithi118.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-19T06:38:27.594 INFO:teuthology.orchestra.run.smithi118.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.596 INFO:teuthology.orchestra.run.smithi118.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-19T06:38:27.598 INFO:teuthology.orchestra.run.smithi118.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.599 INFO:teuthology.orchestra.run.smithi118.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.608 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-09-19T06:38:27.637 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-19T06:38:27.637 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-09-19T06:38:27.637 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.1.1-361.g2f599a46.el9 ceph 6.6 k 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.4 M 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 896 k 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-361.g2f599a46.el9 ceph 145 k 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.1.1-361.g2f599a46.el9 ceph 1.5 M 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 146 k 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 3.6 M 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 7.4 M 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 49 k 2024-09-19T06:38:27.638 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 11 M 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.1.1-361.g2f599a46.el9 ceph 49 M 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 285 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 765 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 228 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell x86_64 2:19.1.1-361.g2f599a46.el9 ceph 27 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 18 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 32 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 751 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 127 k 2024-09-19T06:38:27.639 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.1.1-361.g2f599a46.el9 ceph 161 k 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.1.1-361.g2f599a46.el9 ceph 320 k 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 300 k 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 99 k 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 85 k 2024-09-19T06:38:27.640 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.1 M 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 171 k 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.4 M 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.2 M 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-19T06:38:27.641 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 21 M 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 32 k 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.1.1-361.g2f599a46.el9 ceph 2.4 M 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 249 k 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.1.1-361.g2f599a46.el9 ceph 4.7 M 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 17 M 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 16 k 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.1.1-361.g2f599a46.el9 ceph 25 k 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-19T06:38:27.642 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.1.1-361.g2f599a46.el9 ceph 163 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-19T06:38:27.643 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 506 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.3 M 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-19T06:38:27.644 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 45 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 136 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-19T06:38:27.645 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-19T06:38:27.646 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-19T06:38:27.647 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-19T06:38:27.648 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-19T06:38:27.649 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout:Install 148 Packages 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 8 Packages 2024-09-19T06:38:27.650 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:38:27.651 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 226 M 2024-09-19T06:38:27.651 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-09-19T06:38:27.661 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-19T06:38:27.683 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-09-19T06:38:27.683 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:19.1.1-361.g2f599a46.el9 ceph 6.6 k 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.4 M 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 896 k 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-361.g2f599a46.el9 ceph 145 k 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:19.1.1-361.g2f599a46.el9 ceph 1.5 M 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 146 k 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 3.6 M 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 7.4 M 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 49 k 2024-09-19T06:38:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 11 M 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:19.1.1-361.g2f599a46.el9 ceph 49 M 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 285 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 765 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 228 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-shell x86_64 2:19.1.1-361.g2f599a46.el9 ceph 27 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-top noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 18 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 32 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 751 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 127 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-19T06:38:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:19.1.1-361.g2f599a46.el9 ceph 161 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:19.1.1-361.g2f599a46.el9 ceph 320 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 300 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 99 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 85 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.1 M 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 171 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-09-19T06:38:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.4 M 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.2 M 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 21 M 2024-09-19T06:38:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 32 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:19.1.1-361.g2f599a46.el9 ceph 2.4 M 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 249 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:19.1.1-361.g2f599a46.el9 ceph 4.7 M 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 17 M 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 16 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:19.1.1-361.g2f599a46.el9 ceph 25 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-19T06:38:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:19.1.1-361.g2f599a46.el9 ceph 163 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 506 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.3 M 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-19T06:38:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 45 k 2024-09-19T06:38:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 136 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-19T06:38:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-19T06:38:27.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-19T06:38:27.693 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-19T06:38:27.694 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-19T06:38:27.695 INFO:teuthology.orchestra.run.smithi118.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout:Install 148 Packages 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 8 Packages 2024-09-19T06:38:27.696 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:38:27.697 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 226 M 2024-09-19T06:38:27.697 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-19T06:38:27.709 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.710 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.722 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-19T06:38:27.722 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-19T06:38:27.723 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-19T06:38:27.724 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.726 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-19T06:38:27.728 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.729 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-19T06:38:27.793 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-09-19T06:38:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================== 2024-09-19T06:38:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-09-19T06:38:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================== 2024-09-19T06:38:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-09-19T06:38:27.821 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.1.1-361.g2f599a46.el9 ceph 6.6 k 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.4 M 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 896 k 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-361.g2f599a46.el9 ceph 145 k 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.1.1-361.g2f599a46.el9 ceph 1.5 M 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 146 k 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 3.6 M 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 7.4 M 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 49 k 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 11 M 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.1.1-361.g2f599a46.el9 ceph 49 M 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 285 k 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 765 k 2024-09-19T06:38:27.822 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 228 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell x86_64 2:19.1.1-361.g2f599a46.el9 ceph 27 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 18 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 32 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 751 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.1.1-361.g2f599a46.el9 ceph 127 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-19T06:38:27.823 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.1.1-361.g2f599a46.el9 ceph 161 k 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.1.1-361.g2f599a46.el9 ceph 320 k 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 300 k 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.1.1-361.g2f599a46.el9 ceph 99 k 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 85 k 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.1 M 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 171 k 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-19T06:38:27.824 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.4 M 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 3.2 M 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 21 M 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 32 k 2024-09-19T06:38:27.825 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.1.1-361.g2f599a46.el9 ceph 2.4 M 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 249 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.1.1-361.g2f599a46.el9 ceph 4.7 M 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.1.1-361.g2f599a46.el9 ceph 17 M 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.1.1-361.g2f599a46.el9 ceph-noarch 16 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.1.1-361.g2f599a46.el9 ceph 25 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-19T06:38:27.826 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.1.1-361.g2f599a46.el9 ceph 163 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 506 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.1.1-361.g2f599a46.el9 ceph 5.3 M 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-19T06:38:27.827 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-19T06:38:27.828 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.1.1-361.g2f599a46.el9 ceph 45 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.1.1-361.g2f599a46.el9 ceph 136 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-19T06:38:27.829 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-19T06:38:27.830 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-19T06:38:27.831 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-19T06:38:27.832 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-19T06:38:27.833 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================== 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout:Install 148 Packages 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 8 Packages 2024-09-19T06:38:27.834 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:38:27.835 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 226 M 2024-09-19T06:38:27.835 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-09-19T06:38:28.330 INFO:teuthology.orchestra.run.smithi118.stdout:(1/156): ceph-19.1.1-361.g2f599a46.el9.x86_64.r 58 kB/s | 6.6 kB 00:00 2024-09-19T06:38:28.475 INFO:teuthology.orchestra.run.smithi116.stdout:(1/156): ceph-19.1.1-361.g2f599a46.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-19T06:38:28.564 INFO:teuthology.orchestra.run.smithi118.stdout:(2/156): ceph-fuse-19.1.1-361.g2f599a46.el9.x86 3.7 MB/s | 896 kB 00:00 2024-09-19T06:38:28.681 INFO:teuthology.orchestra.run.smithi118.stdout:(3/156): ceph-base-19.1.1-361.g2f599a46.el9.x86 12 MB/s | 5.4 MB 00:00 2024-09-19T06:38:28.690 INFO:teuthology.orchestra.run.smithi114.stdout:(1/156): ceph-19.1.1-361.g2f599a46.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-09-19T06:38:28.697 INFO:teuthology.orchestra.run.smithi118.stdout:(4/156): ceph-immutable-object-cache-19.1.1-361 1.1 MB/s | 145 kB 00:00 2024-09-19T06:38:28.840 INFO:teuthology.orchestra.run.smithi118.stdout:(5/156): ceph-mds-19.1.1-361.g2f599a46.el9.x86_ 15 MB/s | 2.4 MB 00:00 2024-09-19T06:38:28.892 INFO:teuthology.orchestra.run.smithi116.stdout:(2/156): ceph-fuse-19.1.1-361.g2f599a46.el9.x86 2.1 MB/s | 896 kB 00:00 2024-09-19T06:38:28.956 INFO:teuthology.orchestra.run.smithi118.stdout:(6/156): ceph-mgr-19.1.1-361.g2f599a46.el9.x86_ 5.9 MB/s | 1.5 MB 00:00 2024-09-19T06:38:29.067 INFO:teuthology.orchestra.run.smithi116.stdout:(3/156): ceph-immutable-object-cache-19.1.1-361 829 kB/s | 145 kB 00:00 2024-09-19T06:38:29.140 INFO:teuthology.orchestra.run.smithi118.stdout:(7/156): ceph-mon-19.1.1-361.g2f599a46.el9.x86_ 16 MB/s | 4.7 MB 00:00 2024-09-19T06:38:29.290 INFO:teuthology.orchestra.run.smithi114.stdout:(2/156): ceph-fuse-19.1.1-361.g2f599a46.el9.x86 1.5 MB/s | 896 kB 00:00 2024-09-19T06:38:29.415 INFO:teuthology.orchestra.run.smithi114.stdout:(3/156): ceph-immutable-object-cache-19.1.1-361 1.1 MB/s | 145 kB 00:00 2024-09-19T06:38:29.584 INFO:teuthology.orchestra.run.smithi116.stdout:(4/156): ceph-mds-19.1.1-361.g2f599a46.el9.x86_ 4.6 MB/s | 2.4 MB 00:00 2024-09-19T06:38:29.657 INFO:teuthology.orchestra.run.smithi118.stdout:(8/156): ceph-common-19.1.1-361.g2f599a46.el9.x 15 MB/s | 21 MB 00:01 2024-09-19T06:38:29.716 INFO:teuthology.orchestra.run.smithi118.stdout:(9/156): ceph-selinux-19.1.1-361.g2f599a46.el9. 432 kB/s | 25 kB 00:00 2024-09-19T06:38:29.826 INFO:teuthology.orchestra.run.smithi116.stdout:(5/156): ceph-mgr-19.1.1-361.g2f599a46.el9.x86_ 6.3 MB/s | 1.5 MB 00:00 2024-09-19T06:38:29.832 INFO:teuthology.orchestra.run.smithi114.stdout:(4/156): ceph-mds-19.1.1-361.g2f599a46.el9.x86_ 5.7 MB/s | 2.4 MB 00:00 2024-09-19T06:38:30.024 INFO:teuthology.orchestra.run.smithi114.stdout:(5/156): ceph-base-19.1.1-361.g2f599a46.el9.x86 3.6 MB/s | 5.4 MB 00:01 2024-09-19T06:38:30.093 INFO:teuthology.orchestra.run.smithi116.stdout:(6/156): ceph-base-19.1.1-361.g2f599a46.el9.x86 3.1 MB/s | 5.4 MB 00:01 2024-09-19T06:38:30.149 INFO:teuthology.orchestra.run.smithi118.stdout:(10/156): ceph-radosgw-19.1.1-361.g2f599a46.el9 11 MB/s | 11 MB 00:01 2024-09-19T06:38:30.174 INFO:teuthology.orchestra.run.smithi114.stdout:(6/156): ceph-mgr-19.1.1-361.g2f599a46.el9.x86_ 4.5 MB/s | 1.5 MB 00:00 2024-09-19T06:38:30.225 INFO:teuthology.orchestra.run.smithi118.stdout:(11/156): cephfs-mirror-19.1.1-361.g2f599a46.el 3.0 MB/s | 228 kB 00:00 2024-09-19T06:38:30.325 INFO:teuthology.orchestra.run.smithi118.stdout:(12/156): cephfs-shell-19.1.1-361.g2f599a46.el9 270 kB/s | 27 kB 00:00 2024-09-19T06:38:30.425 INFO:teuthology.orchestra.run.smithi118.stdout:(13/156): libcephfs-devel-19.1.1-361.g2f599a46. 317 kB/s | 32 kB 00:00 2024-09-19T06:38:30.610 INFO:teuthology.orchestra.run.smithi116.stdout:(7/156): ceph-mon-19.1.1-361.g2f599a46.el9.x86_ 6.0 MB/s | 4.7 MB 00:00 2024-09-19T06:38:30.759 INFO:teuthology.orchestra.run.smithi118.stdout:(14/156): libcephfs2-19.1.1-361.g2f599a46.el9.x 2.2 MB/s | 751 kB 00:00 2024-09-19T06:38:30.859 INFO:teuthology.orchestra.run.smithi118.stdout:(15/156): libcephsqlite-19.1.1-361.g2f599a46.el 1.6 MB/s | 163 kB 00:00 2024-09-19T06:38:30.968 INFO:teuthology.orchestra.run.smithi118.stdout:(16/156): librados-devel-19.1.1-361.g2f599a46.e 1.1 MB/s | 127 kB 00:00 2024-09-19T06:38:31.233 INFO:teuthology.orchestra.run.smithi114.stdout:(7/156): ceph-mon-19.1.1-361.g2f599a46.el9.x86_ 3.9 MB/s | 4.7 MB 00:01 2024-09-19T06:38:31.418 INFO:teuthology.orchestra.run.smithi118.stdout:(17/156): libradosstriper1-19.1.1-361.g2f599a46 1.1 MB/s | 506 kB 00:00 2024-09-19T06:38:31.593 INFO:teuthology.orchestra.run.smithi118.stdout:(18/156): ceph-osd-19.1.1-361.g2f599a46.el9.x86 6.4 MB/s | 17 MB 00:02 2024-09-19T06:38:31.693 INFO:teuthology.orchestra.run.smithi118.stdout:(19/156): python3-ceph-argparse-19.1.1-361.g2f5 452 kB/s | 45 kB 00:00 2024-09-19T06:38:31.810 INFO:teuthology.orchestra.run.smithi118.stdout:(20/156): python3-ceph-common-19.1.1-361.g2f599 1.1 MB/s | 136 kB 00:00 2024-09-19T06:38:31.919 INFO:teuthology.orchestra.run.smithi118.stdout:(21/156): python3-cephfs-19.1.1-361.g2f599a46.e 1.5 MB/s | 161 kB 00:00 2024-09-19T06:38:32.535 INFO:teuthology.orchestra.run.smithi116.stdout:(8/156): ceph-common-19.1.1-361.g2f599a46.el9.x 5.1 MB/s | 21 MB 00:04 2024-09-19T06:38:32.583 INFO:teuthology.orchestra.run.smithi114.stdout:(8/156): ceph-common-19.1.1-361.g2f599a46.el9.x 5.2 MB/s | 21 MB 00:04 2024-09-19T06:38:32.586 INFO:teuthology.orchestra.run.smithi118.stdout:(22/156): librgw2-19.1.1-361.g2f599a46.el9.x86_ 4.6 MB/s | 5.3 MB 00:01 2024-09-19T06:38:32.611 INFO:teuthology.orchestra.run.smithi118.stdout:(23/156): python3-rados-19.1.1-361.g2f599a46.el 462 kB/s | 320 kB 00:00 2024-09-19T06:38:32.626 INFO:teuthology.orchestra.run.smithi114.stdout:(9/156): ceph-selinux-19.1.1-361.g2f599a46.el9. 607 kB/s | 25 kB 00:00 2024-09-19T06:38:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:(9/156): ceph-radosgw-19.1.1-361.g2f599a46.el9. 5.2 MB/s | 11 MB 00:02 2024-09-19T06:38:32.645 INFO:teuthology.orchestra.run.smithi118.stdout:(24/156): python3-rbd-19.1.1-361.g2f599a46.el9. 5.0 MB/s | 300 kB 00:00 2024-09-19T06:38:32.669 INFO:teuthology.orchestra.run.smithi116.stdout:(10/156): ceph-selinux-19.1.1-361.g2f599a46.el9 188 kB/s | 25 kB 00:00 2024-09-19T06:38:32.670 INFO:teuthology.orchestra.run.smithi118.stdout:(25/156): python3-rgw-19.1.1-361.g2f599a46.el9. 1.6 MB/s | 99 kB 00:00 2024-09-19T06:38:32.744 INFO:teuthology.orchestra.run.smithi116.stdout:(11/156): cephfs-mirror-19.1.1-361.g2f599a46.el 3.0 MB/s | 228 kB 00:00 2024-09-19T06:38:32.794 INFO:teuthology.orchestra.run.smithi116.stdout:(12/156): cephfs-shell-19.1.1-361.g2f599a46.el9 542 kB/s | 27 kB 00:00 2024-09-19T06:38:32.836 INFO:teuthology.orchestra.run.smithi116.stdout:(13/156): libcephfs-devel-19.1.1-361.g2f599a46. 761 kB/s | 32 kB 00:00 2024-09-19T06:38:32.893 INFO:teuthology.orchestra.run.smithi114.stdout:(10/156): ceph-osd-19.1.1-361.g2f599a46.el9.x86 6.2 MB/s | 17 MB 00:02 2024-09-19T06:38:32.928 INFO:teuthology.orchestra.run.smithi116.stdout:(14/156): libcephfs2-19.1.1-361.g2f599a46.el9.x 8.0 MB/s | 751 kB 00:00 2024-09-19T06:38:32.959 INFO:teuthology.orchestra.run.smithi114.stdout:(11/156): cephfs-mirror-19.1.1-361.g2f599a46.el 3.3 MB/s | 228 kB 00:00 2024-09-19T06:38:32.986 INFO:teuthology.orchestra.run.smithi116.stdout:(15/156): libcephsqlite-19.1.1-361.g2f599a46.el 2.7 MB/s | 163 kB 00:00 2024-09-19T06:38:33.037 INFO:teuthology.orchestra.run.smithi116.stdout:(16/156): librados-devel-19.1.1-361.g2f599a46.e 2.5 MB/s | 127 kB 00:00 2024-09-19T06:38:33.104 INFO:teuthology.orchestra.run.smithi116.stdout:(17/156): libradosstriper1-19.1.1-361.g2f599a46 7.4 MB/s | 506 kB 00:00 2024-09-19T06:38:33.109 INFO:teuthology.orchestra.run.smithi114.stdout:(12/156): ceph-radosgw-19.1.1-361.g2f599a46.el9 5.7 MB/s | 11 MB 00:01 2024-09-19T06:38:33.126 INFO:teuthology.orchestra.run.smithi114.stdout:(13/156): cephfs-shell-19.1.1-361.g2f599a46.el9 162 kB/s | 27 kB 00:00 2024-09-19T06:38:33.160 INFO:teuthology.orchestra.run.smithi114.stdout:(14/156): libcephfs-devel-19.1.1-361.g2f599a46. 632 kB/s | 32 kB 00:00 2024-09-19T06:38:33.236 INFO:teuthology.orchestra.run.smithi114.stdout:(15/156): libcephsqlite-19.1.1-361.g2f599a46.el 2.1 MB/s | 163 kB 00:00 2024-09-19T06:38:33.237 INFO:teuthology.orchestra.run.smithi118.stdout:(26/156): ceph-test-19.1.1-361.g2f599a46.el9.x8 14 MB/s | 49 MB 00:03 2024-09-19T06:38:33.254 INFO:teuthology.orchestra.run.smithi118.stdout:(27/156): rbd-fuse-19.1.1-361.g2f599a46.el9.x86 140 kB/s | 85 kB 00:00 2024-09-19T06:38:33.269 INFO:teuthology.orchestra.run.smithi114.stdout:(16/156): libcephfs2-19.1.1-361.g2f599a46.el9.x 5.2 MB/s | 751 kB 00:00 2024-09-19T06:38:33.311 INFO:teuthology.orchestra.run.smithi114.stdout:(17/156): librados-devel-19.1.1-361.g2f599a46.e 1.6 MB/s | 127 kB 00:00 2024-09-19T06:38:33.313 INFO:teuthology.orchestra.run.smithi118.stdout:(28/156): rbd-nbd-19.1.1-361.g2f599a46.el9.x86_ 2.2 MB/s | 171 kB 00:00 2024-09-19T06:38:33.338 INFO:teuthology.orchestra.run.smithi118.stdout:(29/156): ceph-grafana-dashboards-19.1.1-361.g2 380 kB/s | 32 kB 00:00 2024-09-19T06:38:33.370 INFO:teuthology.orchestra.run.smithi114.stdout:(18/156): libradosstriper1-19.1.1-361.g2f599a46 4.9 MB/s | 506 kB 00:00 2024-09-19T06:38:33.380 INFO:teuthology.orchestra.run.smithi118.stdout:(30/156): ceph-mgr-cephadm-19.1.1-361.g2f599a46 2.1 MB/s | 146 kB 00:00 2024-09-19T06:38:33.420 INFO:teuthology.orchestra.run.smithi114.stdout:(19/156): python3-ceph-argparse-19.1.1-361.g2f5 904 kB/s | 45 kB 00:00 2024-09-19T06:38:33.455 INFO:teuthology.orchestra.run.smithi116.stdout:(18/156): ceph-osd-19.1.1-361.g2f599a46.el9.x86 5.0 MB/s | 17 MB 00:03 2024-09-19T06:38:33.464 INFO:teuthology.orchestra.run.smithi118.stdout:(31/156): rbd-mirror-19.1.1-361.g2f599a46.el9.x 3.9 MB/s | 3.1 MB 00:00 2024-09-19T06:38:33.479 INFO:teuthology.orchestra.run.smithi114.stdout:(20/156): python3-ceph-common-19.1.1-361.g2f599 2.3 MB/s | 136 kB 00:00 2024-09-19T06:38:33.530 INFO:teuthology.orchestra.run.smithi116.stdout:(19/156): python3-ceph-argparse-19.1.1-361.g2f5 604 kB/s | 45 kB 00:00 2024-09-19T06:38:33.547 INFO:teuthology.orchestra.run.smithi118.stdout:(32/156): ceph-mgr-modules-core-19.1.1-361.g2f5 2.9 MB/s | 249 kB 00:00 2024-09-19T06:38:33.571 INFO:teuthology.orchestra.run.smithi114.stdout:(21/156): python3-cephfs-19.1.1-361.g2f599a46.e 1.7 MB/s | 161 kB 00:00 2024-09-19T06:38:33.613 INFO:teuthology.orchestra.run.smithi116.stdout:(20/156): python3-ceph-common-19.1.1-361.g2f599 1.6 MB/s | 136 kB 00:00 2024-09-19T06:38:33.631 INFO:teuthology.orchestra.run.smithi118.stdout:(33/156): ceph-mgr-rook-19.1.1-361.g2f599a46.el 591 kB/s | 49 kB 00:00 2024-09-19T06:38:33.671 INFO:teuthology.orchestra.run.smithi114.stdout:(22/156): python3-rados-19.1.1-361.g2f599a46.el 3.1 MB/s | 320 kB 00:00 2024-09-19T06:38:33.697 INFO:teuthology.orchestra.run.smithi116.stdout:(21/156): python3-cephfs-19.1.1-361.g2f599a46.e 1.9 MB/s | 161 kB 00:00 2024-09-19T06:38:33.723 INFO:teuthology.orchestra.run.smithi118.stdout:(34/156): ceph-prometheus-alerts-19.1.1-361.g2f 179 kB/s | 16 kB 00:00 2024-09-19T06:38:33.771 INFO:teuthology.orchestra.run.smithi114.stdout:(23/156): python3-rbd-19.1.1-361.g2f599a46.el9. 2.9 MB/s | 300 kB 00:00 2024-09-19T06:38:33.855 INFO:teuthology.orchestra.run.smithi114.stdout:(24/156): python3-rgw-19.1.1-361.g2f599a46.el9. 1.2 MB/s | 99 kB 00:00 2024-09-19T06:38:33.873 INFO:teuthology.orchestra.run.smithi118.stdout:(35/156): ceph-volume-19.1.1-361.g2f599a46.el9. 1.9 MB/s | 285 kB 00:00 2024-09-19T06:38:33.897 INFO:teuthology.orchestra.run.smithi116.stdout:(22/156): librgw2-19.1.1-361.g2f599a46.el9.x86_ 6.7 MB/s | 5.3 MB 00:00 2024-09-19T06:38:33.925 INFO:teuthology.orchestra.run.smithi116.stdout:(23/156): python3-rados-19.1.1-361.g2f599a46.el 1.4 MB/s | 320 kB 00:00 2024-09-19T06:38:33.930 INFO:teuthology.orchestra.run.smithi114.stdout:(25/156): rbd-fuse-19.1.1-361.g2f599a46.el9.x86 1.1 MB/s | 85 kB 00:00 2024-09-19T06:38:33.973 INFO:teuthology.orchestra.run.smithi116.stdout:(24/156): python3-rbd-19.1.1-361.g2f599a46.el9. 3.9 MB/s | 300 kB 00:00 2024-09-19T06:38:33.998 INFO:teuthology.orchestra.run.smithi116.stdout:(25/156): python3-rgw-19.1.1-361.g2f599a46.el9. 1.3 MB/s | 99 kB 00:00 2024-09-19T06:38:34.007 INFO:teuthology.orchestra.run.smithi118.stdout:(36/156): ceph-mgr-dashboard-19.1.1-361.g2f599a 5.3 MB/s | 3.6 MB 00:00 2024-09-19T06:38:34.032 INFO:teuthology.orchestra.run.smithi116.stdout:(26/156): rbd-fuse-19.1.1-361.g2f599a46.el9.x86 1.4 MB/s | 85 kB 00:00 2024-09-19T06:38:34.090 INFO:teuthology.orchestra.run.smithi118.stdout:(37/156): ceph-mgr-diskprediction-local-19.1.1- 10 MB/s | 7.4 MB 00:00 2024-09-19T06:38:34.099 INFO:teuthology.orchestra.run.smithi116.stdout:(27/156): rbd-nbd-19.1.1-361.g2f599a46.el9.x86_ 2.5 MB/s | 171 kB 00:00 2024-09-19T06:38:34.116 INFO:teuthology.orchestra.run.smithi118.stdout:(38/156): cephfs-top-19.1.1-361.g2f599a46.el9.n 168 kB/s | 18 kB 00:00 2024-09-19T06:38:34.141 INFO:teuthology.orchestra.run.smithi118.stdout:(39/156): cephadm-19.1.1-361.g2f599a46.el9.noar 2.8 MB/s | 765 kB 00:00 2024-09-19T06:38:34.165 INFO:teuthology.orchestra.run.smithi116.stdout:(28/156): ceph-grafana-dashboards-19.1.1-361.g2 478 kB/s | 32 kB 00:00 2024-09-19T06:38:34.189 INFO:teuthology.orchestra.run.smithi114.stdout:(26/156): librgw2-19.1.1-361.g2f599a46.el9.x86_ 6.1 MB/s | 5.3 MB 00:00 2024-09-19T06:38:34.216 INFO:teuthology.orchestra.run.smithi116.stdout:(29/156): ceph-mgr-cephadm-19.1.1-361.g2f599a46 2.9 MB/s | 146 kB 00:00 2024-09-19T06:38:34.225 INFO:teuthology.orchestra.run.smithi118.stdout:(40/156): fuse-2.9.9-16.el9.x86_64.rpm 597 kB/s | 80 kB 00:00 2024-09-19T06:38:34.264 INFO:teuthology.orchestra.run.smithi114.stdout:(27/156): rbd-nbd-19.1.1-361.g2f599a46.el9.x86_ 2.2 MB/s | 171 kB 00:00 2024-09-19T06:38:34.283 INFO:teuthology.orchestra.run.smithi118.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-19T06:38:34.309 INFO:teuthology.orchestra.run.smithi118.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-09-19T06:38:34.331 INFO:teuthology.orchestra.run.smithi114.stdout:(28/156): rbd-mirror-19.1.1-361.g2f599a46.el9.x 7.7 MB/s | 3.1 MB 00:00 2024-09-19T06:38:34.342 INFO:teuthology.orchestra.run.smithi118.stdout:(43/156): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-09-19T06:38:34.356 INFO:teuthology.orchestra.run.smithi114.stdout:(29/156): ceph-grafana-dashboards-19.1.1-361.g2 347 kB/s | 32 kB 00:00 2024-09-19T06:38:34.367 INFO:teuthology.orchestra.run.smithi118.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-19T06:38:34.393 INFO:teuthology.orchestra.run.smithi118.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-19T06:38:34.398 INFO:teuthology.orchestra.run.smithi114.stdout:(30/156): ceph-mgr-cephadm-19.1.1-361.g2f599a46 2.1 MB/s | 146 kB 00:00 2024-09-19T06:38:34.418 INFO:teuthology.orchestra.run.smithi118.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-19T06:38:34.443 INFO:teuthology.orchestra.run.smithi118.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-09-19T06:38:34.494 INFO:teuthology.orchestra.run.smithi118.stdout:(48/156): python3-cryptography-36.0.1-4.el9.x86 5.8 MB/s | 1.2 MB 00:00 2024-09-19T06:38:34.508 INFO:teuthology.orchestra.run.smithi116.stdout:(30/156): rbd-mirror-19.1.1-361.g2f599a46.el9.x 6.1 MB/s | 3.1 MB 00:00 2024-09-19T06:38:34.594 INFO:teuthology.orchestra.run.smithi118.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 6.1 MB/s | 943 kB 00:00 2024-09-19T06:38:34.627 INFO:teuthology.orchestra.run.smithi118.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 689 kB 00:00 2024-09-19T06:38:34.641 INFO:teuthology.orchestra.run.smithi116.stdout:(31/156): ceph-mgr-dashboard-19.1.1-361.g2f599a 8.4 MB/s | 3.6 MB 00:00 2024-09-19T06:38:34.653 INFO:teuthology.orchestra.run.smithi118.stdout:(51/156): boost-program-options-1.75.0-8.el9.x8 672 kB/s | 107 kB 00:00 2024-09-19T06:38:34.670 INFO:teuthology.orchestra.run.smithi118.stdout:(52/156): elfutils-libelf-devel-0.191-4.el9.x86 286 kB/s | 22 kB 00:00 2024-09-19T06:38:34.704 INFO:teuthology.orchestra.run.smithi118.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 651 kB/s | 33 kB 00:00 2024-09-19T06:38:34.716 INFO:teuthology.orchestra.run.smithi116.stdout:(32/156): ceph-mgr-modules-core-19.1.1-361.g2f5 3.3 MB/s | 249 kB 00:00 2024-09-19T06:38:34.754 INFO:teuthology.orchestra.run.smithi118.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-09-19T06:38:34.771 INFO:teuthology.orchestra.run.smithi118.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 273 kB/s | 18 kB 00:00 2024-09-19T06:38:34.783 INFO:teuthology.orchestra.run.smithi116.stdout:(33/156): ceph-mgr-rook-19.1.1-361.g2f599a46.el 740 kB/s | 49 kB 00:00 2024-09-19T06:38:34.796 INFO:teuthology.orchestra.run.smithi118.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 267 kB/s | 11 kB 00:00 2024-09-19T06:38:34.842 INFO:teuthology.orchestra.run.smithi116.stdout:(34/156): ceph-prometheus-alerts-19.1.1-361.g2f 282 kB/s | 16 kB 00:00 2024-09-19T06:38:34.846 INFO:teuthology.orchestra.run.smithi118.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-09-19T06:38:34.848 INFO:teuthology.orchestra.run.smithi114.stdout:(31/156): ceph-mgr-dashboard-19.1.1-361.g2f599a 7.2 MB/s | 3.6 MB 00:00 2024-09-19T06:38:34.880 INFO:teuthology.orchestra.run.smithi118.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-09-19T06:38:34.901 INFO:teuthology.orchestra.run.smithi116.stdout:(35/156): ceph-volume-19.1.1-361.g2f599a46.el9. 4.8 MB/s | 285 kB 00:00 2024-09-19T06:38:34.924 INFO:teuthology.orchestra.run.smithi114.stdout:(32/156): ceph-mgr-modules-core-19.1.1-361.g2f5 3.3 MB/s | 249 kB 00:00 2024-09-19T06:38:34.972 INFO:teuthology.orchestra.run.smithi118.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-09-19T06:38:34.999 INFO:teuthology.orchestra.run.smithi114.stdout:(33/156): ceph-mgr-rook-19.1.1-361.g2f599a46.el 660 kB/s | 49 kB 00:00 2024-09-19T06:38:35.026 INFO:teuthology.orchestra.run.smithi116.stdout:(36/156): cephadm-19.1.1-361.g2f599a46.el9.noar 6.0 MB/s | 765 kB 00:00 2024-09-19T06:38:35.030 INFO:teuthology.orchestra.run.smithi118.stdout:(60/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.3 MB/s | 3.0 MB 00:00 2024-09-19T06:38:35.049 INFO:teuthology.orchestra.run.smithi114.stdout:(34/156): ceph-prometheus-alerts-19.1.1-361.g2f 330 kB/s | 16 kB 00:00 2024-09-19T06:38:35.056 INFO:teuthology.orchestra.run.smithi118.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 293 kB/s | 24 kB 00:00 2024-09-19T06:38:35.085 INFO:teuthology.orchestra.run.smithi116.stdout:(37/156): cephfs-top-19.1.1-361.g2f599a46.el9.n 313 kB/s | 18 kB 00:00 2024-09-19T06:38:35.102 INFO:teuthology.orchestra.run.smithi118.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 663 kB/s | 47 kB 00:00 2024-09-19T06:38:35.136 INFO:teuthology.orchestra.run.smithi118.stdout:(63/156): libtool-2.4.6-46.el9.x86_64.rpm 2.2 MB/s | 581 kB 00:00 2024-09-19T06:38:35.166 INFO:teuthology.orchestra.run.smithi114.stdout:(35/156): ceph-mgr-diskprediction-local-19.1.1- 9.6 MB/s | 7.4 MB 00:00 2024-09-19T06:38:35.170 INFO:teuthology.orchestra.run.smithi118.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 292 kB 00:00 2024-09-19T06:38:35.193 INFO:teuthology.orchestra.run.smithi116.stdout:(38/156): fuse-2.9.9-16.el9.x86_64.rpm 737 kB/s | 80 kB 00:00 2024-09-19T06:38:35.200 INFO:teuthology.orchestra.run.smithi114.stdout:(36/156): ceph-volume-19.1.1-361.g2f599a46.el9. 1.9 MB/s | 285 kB 00:00 2024-09-19T06:38:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 575 kB/s | 38 kB 00:00 2024-09-19T06:38:35.229 INFO:teuthology.orchestra.run.smithi118.stdout:(66/156): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-19T06:38:35.267 INFO:teuthology.orchestra.run.smithi114.stdout:(37/156): cephadm-19.1.1-361.g2f599a46.el9.noar 7.5 MB/s | 765 kB 00:00 2024-09-19T06:38:35.284 INFO:teuthology.orchestra.run.smithi114.stdout:(38/156): cephfs-top-19.1.1-361.g2f599a46.el9.n 218 kB/s | 18 kB 00:00 2024-09-19T06:38:35.294 INFO:teuthology.orchestra.run.smithi116.stdout:(39/156): libgfortran-11.5.0-2.el9.x86_64.rpm 7.8 MB/s | 801 kB 00:00 2024-09-19T06:38:35.335 INFO:teuthology.orchestra.run.smithi116.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-19T06:38:35.377 INFO:teuthology.orchestra.run.smithi116.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.0 MB/s | 253 kB 00:00 2024-09-19T06:38:35.379 INFO:teuthology.orchestra.run.smithi118.stdout:(67/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-09-19T06:38:35.421 INFO:teuthology.orchestra.run.smithi118.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 8.3 MB/s | 1.6 MB 00:00 2024-09-19T06:38:35.444 INFO:teuthology.orchestra.run.smithi116.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-09-19T06:38:35.486 INFO:teuthology.orchestra.run.smithi116.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-19T06:38:35.528 INFO:teuthology.orchestra.run.smithi116.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-09-19T06:38:35.570 INFO:teuthology.orchestra.run.smithi116.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-09-19T06:38:35.605 INFO:teuthology.orchestra.run.smithi118.stdout:(69/156): openblas-openmp-0.3.26-2.el9.x86_64.r 11 MB/s | 4.9 MB 00:00 2024-09-19T06:38:35.647 INFO:teuthology.orchestra.run.smithi118.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 610 kB/s | 137 kB 00:00 2024-09-19T06:38:35.654 INFO:teuthology.orchestra.run.smithi116.stdout:(46/156): ceph-mgr-diskprediction-local-19.1.1- 6.5 MB/s | 7.4 MB 00:01 2024-09-19T06:38:35.679 INFO:teuthology.orchestra.run.smithi116.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-09-19T06:38:35.705 INFO:teuthology.orchestra.run.smithi118.stdout:(71/156): postgresql-contrib-13.16-1.el9.x86_64 2.7 MB/s | 910 kB 00:00 2024-09-19T06:38:35.789 INFO:teuthology.orchestra.run.smithi118.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-09-19T06:38:35.814 INFO:teuthology.orchestra.run.smithi118.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 805 kB/s | 87 kB 00:00 2024-09-19T06:38:35.881 INFO:teuthology.orchestra.run.smithi118.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 920 kB/s | 61 kB 00:00 2024-09-19T06:38:35.893 INFO:teuthology.orchestra.run.smithi114.stdout:(39/156): ceph-test-19.1.1-361.g2f599a46.el9.x8 15 MB/s | 49 MB 00:03 2024-09-19T06:38:36.006 INFO:teuthology.orchestra.run.smithi118.stdout:(75/156): postgresql-server-13.16-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-09-19T06:38:36.040 INFO:teuthology.orchestra.run.smithi118.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-09-19T06:38:36.098 INFO:teuthology.orchestra.run.smithi118.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 441 kB/s | 26 kB 00:00 2024-09-19T06:38:36.132 INFO:teuthology.orchestra.run.smithi118.stdout:(78/156): python3-jinja2-2.11.3-6.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-09-19T06:38:36.182 INFO:teuthology.orchestra.run.smithi118.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 229 kB/s | 19 kB 00:00 2024-09-19T06:38:36.224 INFO:teuthology.orchestra.run.smithi118.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-19T06:38:36.250 INFO:teuthology.orchestra.run.smithi118.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 519 kB/s | 35 kB 00:00 2024-09-19T06:38:36.350 INFO:teuthology.orchestra.run.smithi118.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.5 MB/s | 250 kB 00:00 2024-09-19T06:38:36.355 INFO:teuthology.orchestra.run.smithi116.stdout:(48/156): ceph-test-19.1.1-361.g2f599a46.el9.x8 13 MB/s | 49 MB 00:03 2024-09-19T06:38:36.395 INFO:teuthology.orchestra.run.smithi114.stdout:(40/156): fuse-2.9.9-16.el9.x86_64.rpm 71 kB/s | 80 kB 00:01 2024-09-19T06:38:36.484 INFO:teuthology.orchestra.run.smithi118.stdout:(83/156): python3-babel-2.9.1-2.el9.noarch.rpm 8.6 MB/s | 6.0 MB 00:00 2024-09-19T06:38:36.509 INFO:teuthology.orchestra.run.smithi118.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-09-19T06:38:36.568 INFO:teuthology.orchestra.run.smithi118.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 927 kB/s | 77 kB 00:00 2024-09-19T06:38:36.593 INFO:teuthology.orchestra.run.smithi118.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 506 kB/s | 42 kB 00:00 2024-09-19T06:38:36.685 INFO:teuthology.orchestra.run.smithi118.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-09-19T06:38:36.695 INFO:teuthology.orchestra.run.smithi114.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 242 kB/s | 191 kB 00:00 2024-09-19T06:38:36.710 INFO:teuthology.orchestra.run.smithi118.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-09-19T06:38:36.777 INFO:teuthology.orchestra.run.smithi118.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 765 kB/s | 51 kB 00:00 2024-09-19T06:38:36.802 INFO:teuthology.orchestra.run.smithi118.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-19T06:38:36.869 INFO:teuthology.orchestra.run.smithi118.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 583 kB/s | 54 kB 00:00 2024-09-19T06:38:36.928 INFO:teuthology.orchestra.run.smithi118.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 715 kB/s | 42 kB 00:00 2024-09-19T06:38:36.986 INFO:teuthology.orchestra.run.smithi118.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 758 kB/s | 44 kB 00:00 2024-09-19T06:38:37.104 INFO:teuthology.orchestra.run.smithi118.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-19T06:38:37.146 INFO:teuthology.orchestra.run.smithi116.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 135 kB/s | 107 kB 00:00 2024-09-19T06:38:37.170 INFO:teuthology.orchestra.run.smithi118.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 540 kB/s | 36 kB 00:00 2024-09-19T06:38:37.237 INFO:teuthology.orchestra.run.smithi114.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 410 kB/s | 801 kB 00:01 2024-09-19T06:38:37.245 INFO:teuthology.orchestra.run.smithi118.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 633 kB/s | 47 kB 00:00 2024-09-19T06:38:37.355 INFO:teuthology.orchestra.run.smithi118.stdout:(97/156): socat-1.7.4.1-6.el9.x86_64.rpm 2.7 MB/s | 304 kB 00:00 2024-09-19T06:38:37.405 INFO:teuthology.orchestra.run.smithi114.stdout:(43/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 251 kB/s | 253 kB 00:01 2024-09-19T06:38:37.430 INFO:teuthology.orchestra.run.smithi118.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 773 kB/s | 58 kB 00:00 2024-09-19T06:38:37.530 INFO:teuthology.orchestra.run.smithi118.stdout:(99/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 955 kB/s | 96 kB 00:00 2024-09-19T06:38:37.598 INFO:teuthology.orchestra.run.smithi118.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 955 kB/s | 64 kB 00:00 2024-09-19T06:38:37.672 INFO:teuthology.orchestra.run.smithi118.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 599 kB/s | 45 kB 00:00 2024-09-19T06:38:37.770 INFO:teuthology.orchestra.run.smithi118.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 334 kB/s | 22 kB 00:00 2024-09-19T06:38:37.963 INFO:teuthology.orchestra.run.smithi116.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 298 kB/s | 689 kB 00:02 2024-09-19T06:38:38.004 INFO:teuthology.orchestra.run.smithi114.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 139 kB/s | 106 kB 00:00 2024-09-19T06:38:38.015 INFO:teuthology.orchestra.run.smithi118.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-19T06:38:38.048 INFO:teuthology.orchestra.run.smithi118.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-09-19T06:38:38.107 INFO:teuthology.orchestra.run.smithi118.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 5.8 MB/s | 548 kB 00:00 2024-09-19T06:38:38.124 INFO:teuthology.orchestra.run.smithi118.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-19T06:38:38.146 INFO:teuthology.orchestra.run.smithi114.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 182 kB/s | 135 kB 00:00 2024-09-19T06:38:38.149 INFO:teuthology.orchestra.run.smithi118.stdout:(107/156): gperftools-libs-2.9.1-3.el9.x86_64.r 3.2 MB/s | 308 kB 00:00 2024-09-19T06:38:38.174 INFO:teuthology.orchestra.run.smithi118.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-19T06:38:38.200 INFO:teuthology.orchestra.run.smithi118.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-19T06:38:38.225 INFO:teuthology.orchestra.run.smithi118.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-19T06:38:38.250 INFO:teuthology.orchestra.run.smithi118.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-09-19T06:38:38.326 INFO:teuthology.orchestra.run.smithi118.stdout:(112/156): python3-numpy-1.20.1-5.el9.x86_64.rp 2.4 MB/s | 5.1 MB 00:02 2024-09-19T06:38:38.393 INFO:teuthology.orchestra.run.smithi118.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.8 MB/s | 839 kB 00:00 2024-09-19T06:38:38.418 INFO:teuthology.orchestra.run.smithi118.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-19T06:38:38.430 INFO:teuthology.orchestra.run.smithi116.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 17 kB/s | 22 kB 00:01 2024-09-19T06:38:38.443 INFO:teuthology.orchestra.run.smithi118.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-19T06:38:38.468 INFO:teuthology.orchestra.run.smithi118.stdout:(116/156): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-09-19T06:38:38.494 INFO:teuthology.orchestra.run.smithi118.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-09-19T06:38:38.519 INFO:teuthology.orchestra.run.smithi118.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-19T06:38:38.544 INFO:teuthology.orchestra.run.smithi118.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-19T06:38:38.578 INFO:teuthology.orchestra.run.smithi118.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.0 MB/s | 303 kB 00:00 2024-09-19T06:38:38.603 INFO:teuthology.orchestra.run.smithi118.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-19T06:38:38.655 INFO:teuthology.orchestra.run.smithi114.stdout:(46/156): python3-cryptography-36.0.1-4.el9.x86 641 kB/s | 1.2 MB 00:01 2024-09-19T06:38:38.695 INFO:teuthology.orchestra.run.smithi118.stdout:(122/156): libarrow-9.0.0-11.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-09-19T06:38:38.721 INFO:teuthology.orchestra.run.smithi118.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-09-19T06:38:38.739 INFO:teuthology.orchestra.run.smithi116.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 308 kB/s | 943 kB 00:03 2024-09-19T06:38:38.746 INFO:teuthology.orchestra.run.smithi118.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-09-19T06:38:38.755 INFO:teuthology.orchestra.run.smithi114.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 166 kB/s | 125 kB 00:00 2024-09-19T06:38:38.772 INFO:teuthology.orchestra.run.smithi118.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 350 kB/s | 18 kB 00:00 2024-09-19T06:38:38.789 INFO:teuthology.orchestra.run.smithi116.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 380 kB/s | 313 kB 00:00 2024-09-19T06:38:38.797 INFO:teuthology.orchestra.run.smithi118.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-09-19T06:38:38.822 INFO:teuthology.orchestra.run.smithi114.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 689 kB 00:00 2024-09-19T06:38:38.822 INFO:teuthology.orchestra.run.smithi118.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-09-19T06:38:38.847 INFO:teuthology.orchestra.run.smithi118.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-19T06:38:38.864 INFO:teuthology.orchestra.run.smithi114.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-09-19T06:38:38.898 INFO:teuthology.orchestra.run.smithi114.stdout:(50/156): elfutils-libelf-devel-0.191-4.el9.x86 657 kB/s | 22 kB 00:00 2024-09-19T06:38:38.923 INFO:teuthology.orchestra.run.smithi118.stdout:(129/156): python3-grpcio-1.46.7-10.el9.x86_64. 5.9 MB/s | 2.0 MB 00:00 2024-09-19T06:38:38.931 INFO:teuthology.orchestra.run.smithi114.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 5.3 MB/s | 943 kB 00:00 2024-09-19T06:38:38.956 INFO:teuthology.orchestra.run.smithi118.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-09-19T06:38:38.965 INFO:teuthology.orchestra.run.smithi114.stdout:(52/156): python3-urllib3-1.26.5-6.el9.noarch.r 262 kB/s | 215 kB 00:00 2024-09-19T06:38:38.982 INFO:teuthology.orchestra.run.smithi118.stdout:(131/156): python3-logutils-0.3.5-21.el9.noarch 348 kB/s | 46 kB 00:00 2024-09-19T06:38:38.983 INFO:teuthology.orchestra.run.smithi114.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 669 kB/s | 33 kB 00:00 2024-09-19T06:38:39.007 INFO:teuthology.orchestra.run.smithi118.stdout:(132/156): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2024-09-19T06:38:39.015 INFO:teuthology.orchestra.run.smithi114.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 313 kB 00:00 2024-09-19T06:38:39.033 INFO:teuthology.orchestra.run.smithi114.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-09-19T06:38:39.041 INFO:teuthology.orchestra.run.smithi118.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 520 kB/s | 16 kB 00:00 2024-09-19T06:38:39.050 INFO:teuthology.orchestra.run.smithi114.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 331 kB/s | 11 kB 00:00 2024-09-19T06:38:39.066 INFO:teuthology.orchestra.run.smithi118.stdout:(134/156): python3-natsort-7.1.1-5.el9.noarch.r 535 kB/s | 58 kB 00:00 2024-09-19T06:38:39.074 INFO:teuthology.orchestra.run.smithi114.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-09-19T06:38:39.091 INFO:teuthology.orchestra.run.smithi118.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-09-19T06:38:39.092 INFO:teuthology.orchestra.run.smithi114.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-19T06:38:39.116 INFO:teuthology.orchestra.run.smithi118.stdout:(136/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-19T06:38:39.125 INFO:teuthology.orchestra.run.smithi114.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-09-19T06:38:39.141 INFO:teuthology.orchestra.run.smithi118.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-09-19T06:38:39.159 INFO:teuthology.orchestra.run.smithi114.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 8.5 MB/s | 581 kB 00:00 2024-09-19T06:38:39.167 INFO:teuthology.orchestra.run.smithi118.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-09-19T06:38:39.176 INFO:teuthology.orchestra.run.smithi114.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 490 kB/s | 24 kB 00:00 2024-09-19T06:38:39.192 INFO:teuthology.orchestra.run.smithi118.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-19T06:38:39.193 INFO:teuthology.orchestra.run.smithi114.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-19T06:38:39.217 INFO:teuthology.orchestra.run.smithi118.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-09-19T06:38:39.226 INFO:teuthology.orchestra.run.smithi114.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-09-19T06:38:39.243 INFO:teuthology.orchestra.run.smithi118.stdout:(141/156): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-09-19T06:38:39.252 INFO:teuthology.orchestra.run.smithi114.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-09-19T06:38:39.268 INFO:teuthology.orchestra.run.smithi118.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 989 kB/s | 74 kB 00:00 2024-09-19T06:38:39.269 INFO:teuthology.orchestra.run.smithi114.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 924 kB/s | 38 kB 00:00 2024-09-19T06:38:39.293 INFO:teuthology.orchestra.run.smithi118.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-19T06:38:39.310 INFO:teuthology.orchestra.run.smithi118.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-09-19T06:38:39.336 INFO:teuthology.orchestra.run.smithi114.stdout:(66/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.1 MB/s | 3.0 MB 00:00 2024-09-19T06:38:39.336 INFO:teuthology.orchestra.run.smithi118.stdout:(145/156): python3-webob-1.8.8-2.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-09-19T06:38:39.361 INFO:teuthology.orchestra.run.smithi118.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-19T06:38:39.386 INFO:teuthology.orchestra.run.smithi118.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-09-19T06:38:39.411 INFO:teuthology.orchestra.run.smithi114.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-09-19T06:38:39.428 INFO:teuthology.orchestra.run.smithi118.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-19T06:38:39.486 INFO:teuthology.orchestra.run.smithi114.stdout:(68/156): openblas-openmp-0.3.26-2.el9.x86_64.r 21 MB/s | 4.9 MB 00:00 2024-09-19T06:38:39.578 INFO:teuthology.orchestra.run.smithi114.stdout:(69/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-19T06:38:39.604 INFO:teuthology.orchestra.run.smithi114.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.1 MB/s | 137 kB 00:00 2024-09-19T06:38:39.628 INFO:teuthology.orchestra.run.smithi118.stdout:(149/156): NetworkManager-1.51.0-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-09-19T06:38:39.637 INFO:teuthology.orchestra.run.smithi114.stdout:(71/156): postgresql-contrib-13.16-1.el9.x86_64 3.9 MB/s | 910 kB 00:00 2024-09-19T06:38:39.679 INFO:teuthology.orchestra.run.smithi114.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-09-19T06:38:39.687 INFO:teuthology.orchestra.run.smithi118.stdout:(150/156): NetworkManager-libnm-1.51.0-1.el9.x8 31 MB/s | 1.8 MB 00:00 2024-09-19T06:38:39.704 INFO:teuthology.orchestra.run.smithi114.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-09-19T06:38:39.732 INFO:teuthology.orchestra.run.smithi118.stdout:(151/156): NetworkManager-team-1.51.0-1.el9.x86 916 kB/s | 41 kB 00:00 2024-09-19T06:38:39.747 INFO:teuthology.orchestra.run.smithi114.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-09-19T06:38:39.771 INFO:teuthology.orchestra.run.smithi118.stdout:(152/156): NetworkManager-tui-1.51.0-1.el9.x86_ 6.3 MB/s | 249 kB 00:00 2024-09-19T06:38:39.839 INFO:teuthology.orchestra.run.smithi114.stdout:(75/156): postgresql-server-13.16-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-09-19T06:38:39.855 INFO:teuthology.orchestra.run.smithi118.stdout:(153/156): openssl-3.2.2-6.el9.x86_64.rpm 16 MB/s | 1.4 MB 00:00 2024-09-19T06:38:39.863 INFO:teuthology.orchestra.run.smithi114.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-09-19T06:38:39.889 INFO:teuthology.orchestra.run.smithi114.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-09-19T06:38:39.914 INFO:teuthology.orchestra.run.smithi114.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-09-19T06:38:39.940 INFO:teuthology.orchestra.run.smithi114.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-09-19T06:38:39.973 INFO:teuthology.orchestra.run.smithi114.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-19T06:38:39.980 INFO:teuthology.orchestra.run.smithi118.stdout:(154/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-09-19T06:38:39.999 INFO:teuthology.orchestra.run.smithi114.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 593 kB/s | 35 kB 00:00 2024-09-19T06:38:40.099 INFO:teuthology.orchestra.run.smithi114.stdout:(82/156): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-19T06:38:40.124 INFO:teuthology.orchestra.run.smithi114.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-09-19T06:38:40.149 INFO:teuthology.orchestra.run.smithi114.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-19T06:38:40.174 INFO:teuthology.orchestra.run.smithi114.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-19T06:38:40.200 INFO:teuthology.orchestra.run.smithi114.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-09-19T06:38:40.225 INFO:teuthology.orchestra.run.smithi114.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-09-19T06:38:40.259 INFO:teuthology.orchestra.run.smithi114.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-19T06:38:40.284 INFO:teuthology.orchestra.run.smithi114.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-09-19T06:38:40.297 INFO:teuthology.orchestra.run.smithi118.stdout:(155/156): librados2-19.1.1-361.g2f599a46.el9.x 3.7 MB/s | 3.4 MB 00:00 2024-09-19T06:38:40.309 INFO:teuthology.orchestra.run.smithi114.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-19T06:38:40.334 INFO:teuthology.orchestra.run.smithi114.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-19T06:38:40.393 INFO:teuthology.orchestra.run.smithi114.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 720 kB/s | 42 kB 00:00 2024-09-19T06:38:40.455 INFO:teuthology.orchestra.run.smithi118.stdout:(156/156): librbd1-19.1.1-361.g2f599a46.el9.x86 3.0 MB/s | 3.2 MB 00:01 2024-09-19T06:38:40.462 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-19T06:38:40.462 INFO:teuthology.orchestra.run.smithi118.stdout:Total 18 MB/s | 226 MB 00:12 2024-09-19T06:38:40.464 INFO:teuthology.orchestra.run.smithi116.stdout:(54/156): flexiblas-3.0.4-8.el9.x86_64.rpm 16 kB/s | 33 kB 00:02 2024-09-19T06:38:40.469 INFO:teuthology.orchestra.run.smithi114.stdout:(93/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-09-19T06:38:40.494 INFO:teuthology.orchestra.run.smithi114.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-09-19T06:38:40.519 INFO:teuthology.orchestra.run.smithi114.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-19T06:38:40.536 INFO:teuthology.orchestra.run.smithi114.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 858 kB/s | 36 kB 00:00 2024-09-19T06:38:40.553 INFO:teuthology.orchestra.run.smithi114.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-19T06:38:40.587 INFO:teuthology.orchestra.run.smithi114.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-09-19T06:38:40.589 INFO:teuthology.orchestra.run.smithi116.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 89 kB/s | 11 kB 00:00 2024-09-19T06:38:40.612 INFO:teuthology.orchestra.run.smithi114.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 983 kB/s | 58 kB 00:00 2024-09-19T06:38:40.637 INFO:teuthology.orchestra.run.smithi114.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-19T06:38:40.662 INFO:teuthology.orchestra.run.smithi114.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-19T06:38:40.689 INFO:teuthology.orchestra.run.smithi114.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 897 kB/s | 45 kB 00:00 2024-09-19T06:38:40.713 INFO:teuthology.orchestra.run.smithi114.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-09-19T06:38:40.764 INFO:teuthology.orchestra.run.smithi114.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-09-19T06:38:40.781 INFO:teuthology.orchestra.run.smithi116.stdout:(56/156): flexiblas-openblas-openmp-3.0.4-8.el9 9.1 kB/s | 18 kB 00:01 2024-09-19T06:38:40.822 INFO:teuthology.orchestra.run.smithi114.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 4.9 MB/s | 548 kB 00:00 2024-09-19T06:38:40.847 INFO:teuthology.orchestra.run.smithi114.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 790 kB/s | 19 kB 00:00 2024-09-19T06:38:40.873 INFO:teuthology.orchestra.run.smithi114.stdout:(107/156): gperftools-libs-2.9.1-3.el9.x86_64.r 2.8 MB/s | 308 kB 00:00 2024-09-19T06:38:40.898 INFO:teuthology.orchestra.run.smithi114.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-19T06:38:40.923 INFO:teuthology.orchestra.run.smithi114.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-19T06:38:40.948 INFO:teuthology.orchestra.run.smithi114.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-19T06:38:40.982 INFO:teuthology.orchestra.run.smithi114.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-09-19T06:38:41.041 INFO:teuthology.orchestra.run.smithi114.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-19T06:38:41.083 INFO:teuthology.orchestra.run.smithi114.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-19T06:38:41.108 INFO:teuthology.orchestra.run.smithi114.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-19T06:38:41.133 INFO:teuthology.orchestra.run.smithi114.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-19T06:38:41.158 INFO:teuthology.orchestra.run.smithi114.stdout:(116/156): python3-certifi-2023.05.07-4.el9.noa 565 kB/s | 14 kB 00:00 2024-09-19T06:38:41.223 INFO:teuthology.orchestra.run.smithi114.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.6 MB/s | 172 kB 00:00 2024-09-19T06:38:41.276 INFO:teuthology.orchestra.run.smithi114.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 6.6 MB/s | 358 kB 00:00 2024-09-19T06:38:41.318 INFO:teuthology.orchestra.run.smithi114.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.1 MB/s | 303 kB 00:00 2024-09-19T06:38:41.360 INFO:teuthology.orchestra.run.smithi114.stdout:(120/156): python3-google-auth-2.34.0-1.el9.noa 5.1 MB/s | 220 kB 00:00 2024-09-19T06:38:41.436 INFO:teuthology.orchestra.run.smithi114.stdout:(121/156): libarrow-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-09-19T06:38:41.469 INFO:teuthology.orchestra.run.smithi114.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-09-19T06:38:41.494 INFO:teuthology.orchestra.run.smithi114.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-09-19T06:38:41.520 INFO:teuthology.orchestra.run.smithi114.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 710 kB/s | 18 kB 00:00 2024-09-19T06:38:41.545 INFO:teuthology.orchestra.run.smithi114.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 931 kB/s | 23 kB 00:00 2024-09-19T06:38:41.570 INFO:teuthology.orchestra.run.smithi114.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 780 kB/s | 19 kB 00:00 2024-09-19T06:38:41.689 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-19T06:38:41.796 INFO:teuthology.orchestra.run.smithi114.stdout:(127/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-19T06:38:41.813 INFO:teuthology.orchestra.run.smithi114.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 81 kB/s | 20 kB 00:00 2024-09-19T06:38:41.838 INFO:teuthology.orchestra.run.smithi114.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 3.2 MB/s | 46 kB 00:00 2024-09-19T06:38:41.840 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-19T06:38:41.840 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-19T06:38:41.872 INFO:teuthology.orchestra.run.smithi114.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 2.4 MB/s | 79 kB 00:00 2024-09-19T06:38:41.897 INFO:teuthology.orchestra.run.smithi114.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-19T06:38:41.931 INFO:teuthology.orchestra.run.smithi114.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 8.1 MB/s | 272 kB 00:00 2024-09-19T06:38:41.956 INFO:teuthology.orchestra.run.smithi114.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 661 kB/s | 16 kB 00:00 2024-09-19T06:38:41.981 INFO:teuthology.orchestra.run.smithi114.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-19T06:38:42.007 INFO:teuthology.orchestra.run.smithi114.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-19T06:38:42.049 INFO:teuthology.orchestra.run.smithi114.stdout:(136/156): python3-repoze-lru-0.7-16.el9.noarch 742 kB/s | 31 kB 00:00 2024-09-19T06:38:42.107 INFO:teuthology.orchestra.run.smithi114.stdout:(137/156): python3-grpcio-1.46.7-10.el9.x86_64. 2.7 MB/s | 2.0 MB 00:00 2024-09-19T06:38:42.141 INFO:teuthology.orchestra.run.smithi114.stdout:(138/156): python3-kubernetes-26.1.0-2.el9.noar 3.0 MB/s | 1.0 MB 00:00 2024-09-19T06:38:42.166 INFO:teuthology.orchestra.run.smithi114.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-19T06:38:42.191 INFO:teuthology.orchestra.run.smithi114.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-09-19T06:38:42.217 INFO:teuthology.orchestra.run.smithi114.stdout:(141/156): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-09-19T06:38:42.242 INFO:teuthology.orchestra.run.smithi114.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 986 kB/s | 74 kB 00:00 2024-09-19T06:38:42.267 INFO:teuthology.orchestra.run.smithi114.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-19T06:38:42.301 INFO:teuthology.orchestra.run.smithi114.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-19T06:38:42.326 INFO:teuthology.orchestra.run.smithi114.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-09-19T06:38:42.360 INFO:teuthology.orchestra.run.smithi114.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-19T06:38:42.385 INFO:teuthology.orchestra.run.smithi114.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-09-19T06:38:42.444 INFO:teuthology.orchestra.run.smithi114.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-19T06:38:42.623 INFO:teuthology.orchestra.run.smithi116.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 79 kB/s | 160 kB 00:02 2024-09-19T06:38:42.790 INFO:teuthology.orchestra.run.smithi116.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 23 kB/s | 45 kB 00:02 2024-09-19T06:38:43.011 INFO:teuthology.orchestra.run.smithi114.stdout:(149/156): librados2-19.1.1-361.g2f599a46.el9.x 5.3 MB/s | 3.4 MB 00:00 2024-09-19T06:38:43.086 INFO:teuthology.orchestra.run.smithi114.stdout:(150/156): librbd1-19.1.1-361.g2f599a46.el9.x86 4.5 MB/s | 3.2 MB 00:00 2024-09-19T06:38:43.642 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-19T06:38:43.643 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-19T06:38:43.728 INFO:teuthology.orchestra.run.smithi114.stdout:(151/156): NetworkManager-team-1.51.0-1.el9.x86 64 kB/s | 41 kB 00:00 2024-09-19T06:38:44.070 INFO:teuthology.orchestra.run.smithi114.stdout:(152/156): NetworkManager-1.51.0-1.el9.x86_64.r 1.4 MB/s | 2.3 MB 00:01 2024-09-19T06:38:44.112 INFO:teuthology.orchestra.run.smithi114.stdout:(153/156): NetworkManager-libnm-1.51.0-1.el9.x8 1.6 MB/s | 1.8 MB 00:01 2024-09-19T06:38:44.807 INFO:teuthology.orchestra.run.smithi116.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 303 kB/s | 662 kB 00:02 2024-09-19T06:38:44.862 INFO:teuthology.orchestra.run.smithi114.stdout:(154/156): NetworkManager-tui-1.51.0-1.el9.x86_ 220 kB/s | 249 kB 00:01 2024-09-19T06:38:45.703 INFO:teuthology.orchestra.run.smithi114.stdout:(155/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 1.5 MB/s | 2.4 MB 00:01 2024-09-19T06:38:45.879 INFO:teuthology.orchestra.run.smithi114.stdout:(156/156): openssl-3.2.2-6.el9.x86_64.rpm 785 kB/s | 1.4 MB 00:01 2024-09-19T06:38:45.886 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-19T06:38:45.886 INFO:teuthology.orchestra.run.smithi114.stdout:Total 12 MB/s | 226 MB 00:18 2024-09-19T06:38:46.065 INFO:teuthology.orchestra.run.smithi116.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 177 kB/s | 581 kB 00:03 2024-09-19T06:38:46.549 INFO:teuthology.orchestra.run.smithi116.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 14 kB/s | 24 kB 00:01 2024-09-19T06:38:47.965 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-09-19T06:38:48.057 INFO:teuthology.orchestra.run.smithi116.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 24 kB/s | 47 kB 00:01 2024-09-19T06:38:48.141 INFO:teuthology.orchestra.run.smithi116.stdout:(63/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 326 kB/s | 3.0 MB 00:09 2024-09-19T06:38:48.143 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-09-19T06:38:48.143 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-09-19T06:38:48.224 INFO:teuthology.orchestra.run.smithi116.stdout:(64/156): openblas-0.3.26-2.el9.x86_64.rpm 462 kB/s | 38 kB 00:00 2024-09-19T06:38:48.250 INFO:teuthology.orchestra.run.smithi116.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 980 kB/s | 188 kB 00:00 2024-09-19T06:38:48.650 INFO:teuthology.orchestra.run.smithi116.stdout:(66/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 139 kB/s | 292 kB 00:02 2024-09-19T06:38:50.024 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-19T06:38:50.117 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-19T06:38:50.281 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-19T06:38:50.478 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-19T06:38:50.519 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-19T06:38:50.531 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-09-19T06:38:50.532 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-09-19T06:38:50.544 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-19T06:38:50.587 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-19T06:38:50.825 INFO:teuthology.orchestra.run.smithi116.stdout:(67/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 1.7 MB/s | 4.4 MB 00:02 2024-09-19T06:38:50.867 INFO:teuthology.orchestra.run.smithi116.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 732 kB/s | 1.6 MB 00:02 2024-09-19T06:38:50.916 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-19T06:38:50.925 INFO:teuthology.orchestra.run.smithi116.stdout:(69/156): postgresql-private-libs-13.16-1.el9.x 2.3 MB/s | 137 kB 00:00 2024-09-19T06:38:50.959 INFO:teuthology.orchestra.run.smithi116.stdout:(70/156): postgresql-contrib-13.16-1.el9.x86_64 6.7 MB/s | 910 kB 00:00 2024-09-19T06:38:51.017 INFO:teuthology.orchestra.run.smithi116.stdout:(71/156): openblas-openmp-0.3.26-2.el9.x86_64.r 1.8 MB/s | 4.9 MB 00:02 2024-09-19T06:38:51.020 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:38:51.076 INFO:teuthology.orchestra.run.smithi116.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-09-19T06:38:51.109 INFO:teuthology.orchestra.run.smithi116.stdout:(73/156): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-09-19T06:38:51.142 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:38:51.171 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:38:51.262 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:38:51.301 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T06:38:51.449 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-19T06:38:51.506 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-19T06:38:51.534 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:38:51.638 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:38:51.660 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T06:38:51.723 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T06:38:51.744 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T06:38:51.770 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T06:38:51.963 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T06:38:52.095 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T06:38:52.227 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 17/164 2024-09-19T06:38:52.328 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:38:52.420 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:38:52.674 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:38:52.796 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-19T06:38:52.898 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-19T06:38:52.926 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-19T06:38:52.961 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-19T06:38:53.034 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-19T06:38:53.079 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-19T06:38:53.119 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-19T06:38:53.163 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-19T06:38:53.196 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-19T06:38:53.210 INFO:teuthology.orchestra.run.smithi116.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 29 kB/s | 61 kB 00:02 2024-09-19T06:38:53.224 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-19T06:38:53.294 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-19T06:38:53.326 INFO:teuthology.orchestra.run.smithi116.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-09-19T06:38:53.356 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-19T06:38:53.386 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-19T06:38:53.410 INFO:teuthology.orchestra.run.smithi116.stdout:(76/156): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-09-19T06:38:53.430 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-19T06:38:53.461 INFO:teuthology.orchestra.run.smithi116.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-09-19T06:38:53.508 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-19T06:38:53.510 INFO:teuthology.orchestra.run.smithi116.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 385 kB/s | 19 kB 00:00 2024-09-19T06:38:53.541 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 34/164 2024-09-19T06:38:53.580 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 35/164 2024-09-19T06:38:53.586 INFO:teuthology.orchestra.run.smithi116.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-19T06:38:53.608 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 36/164 2024-09-19T06:38:53.645 INFO:teuthology.orchestra.run.smithi116.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 597 kB/s | 35 kB 00:00 2024-09-19T06:38:53.678 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-19T06:38:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-19T06:38:53.770 INFO:teuthology.orchestra.run.smithi116.stdout:(81/156): python3-babel-2.9.1-2.el9.noarch.rpm 2.2 MB/s | 6.0 MB 00:02 2024-09-19T06:38:53.797 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-19T06:38:53.930 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-19T06:38:53.984 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-19T06:38:54.510 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-19T06:38:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-19T06:38:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-19T06:38:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-19T06:38:54.699 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 46/164 2024-09-19T06:38:54.740 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-19T06:38:54.887 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-19T06:38:54.943 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-19T06:38:54.998 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-19T06:38:55.070 INFO:teuthology.orchestra.run.smithi116.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 193 kB/s | 250 kB 00:01 2024-09-19T06:38:55.070 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-19T06:38:55.103 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-19T06:38:55.128 INFO:teuthology.orchestra.run.smithi116.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-09-19T06:38:55.139 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-19T06:38:55.179 INFO:teuthology.orchestra.run.smithi116.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-19T06:38:55.186 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-19T06:38:55.222 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-19T06:38:55.229 INFO:teuthology.orchestra.run.smithi116.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 847 kB/s | 42 kB 00:00 2024-09-19T06:38:55.265 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-19T06:38:55.288 INFO:teuthology.orchestra.run.smithi116.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-19T06:38:55.307 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-19T06:38:55.346 INFO:teuthology.orchestra.run.smithi116.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-19T06:38:55.405 INFO:teuthology.orchestra.run.smithi116.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-09-19T06:38:55.443 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-19T06:38:55.455 INFO:teuthology.orchestra.run.smithi116.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-19T06:38:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:(90/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.7 MB/s | 5.1 MB 00:01 2024-09-19T06:38:55.547 INFO:teuthology.orchestra.run.smithi116.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 584 kB/s | 54 kB 00:00 2024-09-19T06:38:55.706 INFO:teuthology.orchestra.run.smithi116.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 263 kB/s | 42 kB 00:00 2024-09-19T06:38:55.715 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-19T06:38:55.748 INFO:teuthology.orchestra.run.smithi116.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-09-19T06:38:55.774 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-19T06:38:55.807 INFO:teuthology.orchestra.run.smithi116.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-19T06:38:55.879 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-19T06:38:55.920 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-19T06:38:56.073 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-19T06:38:56.230 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-19T06:38:56.372 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-19T06:38:56.447 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-19T06:38:56.584 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-19T06:38:56.622 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-19T06:38:56.650 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-19T06:38:56.684 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-19T06:38:56.724 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-19T06:38:56.764 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-19T06:38:56.794 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-19T06:38:56.828 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-19T06:38:56.861 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-19T06:38:56.903 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-19T06:38:56.932 INFO:teuthology.orchestra.run.smithi116.stdout:(95/156): postgresql-server-13.16-1.el9.x86_64. 985 kB/s | 5.8 MB 00:06 2024-09-19T06:38:56.938 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-19T06:38:56.974 INFO:teuthology.orchestra.run.smithi116.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-09-19T06:38:57.041 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-19T06:38:57.057 INFO:teuthology.orchestra.run.smithi116.stdout:(97/156): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-09-19T06:38:57.085 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-19T06:38:57.116 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-19T06:38:57.124 INFO:teuthology.orchestra.run.smithi116.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 867 kB/s | 58 kB 00:00 2024-09-19T06:38:57.218 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-09-19T06:38:57.219 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-19T06:38:57.276 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-19T06:38:57.359 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-19T06:38:57.423 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-19T06:38:57.441 INFO:teuthology.orchestra.run.smithi116.stdout:(99/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 302 kB/s | 96 kB 00:00 2024-09-19T06:38:57.483 INFO:teuthology.orchestra.run.smithi116.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-19T06:38:57.491 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-19T06:38:57.536 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-19T06:38:57.541 INFO:teuthology.orchestra.run.smithi116.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 767 kB/s | 45 kB 00:00 2024-09-19T06:38:57.568 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-19T06:38:57.634 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-19T06:38:57.842 INFO:teuthology.orchestra.run.smithi116.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-09-19T06:38:57.959 INFO:teuthology.orchestra.run.smithi116.stdout:(103/156): qatlib-service-24.02.0-1.el9.x86_64. 17 kB/s | 36 kB 00:02 2024-09-19T06:38:57.998 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-19T06:38:58.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-19T06:38:58.009 INFO:teuthology.orchestra.run.smithi116.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 5.1 MB/s | 863 kB 00:00 2024-09-19T06:38:58.048 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-19T06:38:58.051 INFO:teuthology.orchestra.run.smithi116.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 5.8 MB/s | 548 kB 00:00 2024-09-19T06:38:58.068 INFO:teuthology.orchestra.run.smithi116.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 1.3 MB/s | 19 kB 00:00 2024-09-19T06:38:58.082 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-19T06:38:58.101 INFO:teuthology.orchestra.run.smithi116.stdout:(107/156): gperftools-libs-2.9.1-3.el9.x86_64.r 3.3 MB/s | 308 kB 00:00 2024-09-19T06:38:58.110 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:38:58.118 INFO:teuthology.orchestra.run.smithi116.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-09-19T06:38:58.136 INFO:teuthology.orchestra.run.smithi116.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 2.9 MB/s | 49 kB 00:00 2024-09-19T06:38:58.161 INFO:teuthology.orchestra.run.smithi116.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-19T06:38:58.181 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-19T06:38:58.195 INFO:teuthology.orchestra.run.smithi116.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2024-09-19T06:38:58.229 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-19T06:38:58.238 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:38:58.245 INFO:teuthology.orchestra.run.smithi116.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-09-19T06:38:58.261 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:38:58.274 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-19T06:38:58.312 INFO:teuthology.orchestra.run.smithi116.stdout:(113/156): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-09-19T06:38:58.337 INFO:teuthology.orchestra.run.smithi116.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-19T06:38:58.362 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-19T06:38:58.362 INFO:teuthology.orchestra.run.smithi116.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.2 MB/s | 32 kB 00:00 2024-09-19T06:38:58.374 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:38:58.387 INFO:teuthology.orchestra.run.smithi116.stdout:(116/156): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-09-19T06:38:58.410 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-19T06:38:58.413 INFO:teuthology.orchestra.run.smithi116.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-09-19T06:38:58.424 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T06:38:58.439 INFO:teuthology.orchestra.run.smithi116.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-19T06:38:58.463 INFO:teuthology.orchestra.run.smithi116.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-19T06:38:58.489 INFO:teuthology.orchestra.run.smithi116.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-09-19T06:38:58.514 INFO:teuthology.orchestra.run.smithi116.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-19T06:38:58.539 INFO:teuthology.orchestra.run.smithi116.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-09-19T06:38:58.564 INFO:teuthology.orchestra.run.smithi116.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 428 kB/s | 11 kB 00:00 2024-09-19T06:38:58.589 INFO:teuthology.orchestra.run.smithi116.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 716 kB/s | 18 kB 00:00 2024-09-19T06:38:58.594 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-19T06:38:58.615 INFO:teuthology.orchestra.run.smithi116.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 926 kB/s | 23 kB 00:00 2024-09-19T06:38:58.624 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-19T06:38:58.640 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-19T06:38:58.640 INFO:teuthology.orchestra.run.smithi116.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 777 kB/s | 19 kB 00:00 2024-09-19T06:38:58.702 INFO:teuthology.orchestra.run.smithi116.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 321 kB/s | 20 kB 00:00 2024-09-19T06:38:58.704 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-19T06:38:58.722 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-19T06:38:58.748 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:38:58.749 INFO:teuthology.orchestra.run.smithi116.stdout:(128/156): python3-grpcio-1.46.7-10.el9.x86_64. 7.8 MB/s | 2.0 MB 00:00 2024-09-19T06:38:58.774 INFO:teuthology.orchestra.run.smithi116.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-19T06:38:58.799 INFO:teuthology.orchestra.run.smithi116.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-19T06:38:58.816 INFO:teuthology.orchestra.run.smithi116.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 3.4 MB/s | 58 kB 00:00 2024-09-19T06:38:58.858 INFO:teuthology.orchestra.run.smithi116.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 6.5 MB/s | 1.0 MB 00:00 2024-09-19T06:38:58.875 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:38:58.884 INFO:teuthology.orchestra.run.smithi116.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-09-19T06:38:58.900 INFO:teuthology.orchestra.run.smithi116.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 392 kB/s | 16 kB 00:00 2024-09-19T06:38:58.908 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T06:38:58.917 INFO:teuthology.orchestra.run.smithi116.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.7 MB/s | 28 kB 00:00 2024-09-19T06:38:58.942 INFO:teuthology.orchestra.run.smithi116.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-09-19T06:38:58.959 INFO:teuthology.orchestra.run.smithi116.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 734 kB/s | 31 kB 00:00 2024-09-19T06:38:58.982 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T06:38:58.985 INFO:teuthology.orchestra.run.smithi116.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2024-09-19T06:38:59.002 INFO:teuthology.orchestra.run.smithi116.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-19T06:38:59.009 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T06:38:59.019 INFO:teuthology.orchestra.run.smithi116.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-09-19T06:38:59.044 INFO:teuthology.orchestra.run.smithi116.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.7 MB/s | 74 kB 00:00 2024-09-19T06:38:59.053 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T06:38:59.069 INFO:teuthology.orchestra.run.smithi116.stdout:(142/156): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-19T06:38:59.094 INFO:teuthology.orchestra.run.smithi116.stdout:(143/156): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-19T06:38:59.112 INFO:teuthology.orchestra.run.smithi116.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-09-19T06:38:59.145 INFO:teuthology.orchestra.run.smithi116.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-09-19T06:38:59.170 INFO:teuthology.orchestra.run.smithi116.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-19T06:38:59.238 INFO:teuthology.orchestra.run.smithi116.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-19T06:38:59.285 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T06:38:59.347 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-19T06:38:59.454 INFO:teuthology.orchestra.run.smithi116.stdout:(148/156): librados2-19.1.1-361.g2f599a46.el9.x 12 MB/s | 3.4 MB 00:00 2024-09-19T06:38:59.468 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T06:38:59.473 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-19T06:38:59.649 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 17/164 2024-09-19T06:38:59.655 INFO:teuthology.orchestra.run.smithi116.stdout:(149/156): librbd1-19.1.1-361.g2f599a46.el9.x86 7.6 MB/s | 3.2 MB 00:00 2024-09-19T06:38:59.705 INFO:teuthology.orchestra.run.smithi116.stdout:(150/156): NetworkManager-1.51.0-1.el9.x86_64.r 9.3 MB/s | 2.3 MB 00:00 2024-09-19T06:38:59.738 INFO:teuthology.orchestra.run.smithi116.stdout:(151/156): NetworkManager-team-1.51.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-09-19T06:38:59.769 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:38:59.780 INFO:teuthology.orchestra.run.smithi116.stdout:(152/156): NetworkManager-tui-1.51.0-1.el9.x86_ 5.8 MB/s | 249 kB 00:00 2024-09-19T06:38:59.839 INFO:teuthology.orchestra.run.smithi116.stdout:(153/156): openssl-3.2.2-6.el9.x86_64.rpm 23 MB/s | 1.4 MB 00:00 2024-09-19T06:38:59.859 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:38:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:(154/156): NetworkManager-libnm-1.51.0-1.el9.x8 8.0 MB/s | 1.8 MB 00:00 2024-09-19T06:38:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:(155/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-09-19T06:39:00.164 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:39:00.297 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-19T06:39:00.435 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-19T06:39:00.482 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-19T06:39:00.534 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-19T06:39:00.622 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-19T06:39:00.695 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-19T06:39:00.705 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-19T06:39:00.735 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-19T06:39:00.780 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-19T06:39:00.789 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-19T06:39:00.808 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-19T06:39:00.836 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-19T06:39:00.840 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-19T06:39:00.878 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-19T06:39:00.966 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-19T06:39:01.023 INFO:teuthology.orchestra.run.smithi116.stdout:(156/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-09-19T06:39:01.029 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-09-19T06:39:01.029 INFO:teuthology.orchestra.run.smithi116.stdout:Total 6.8 MB/s | 226 MB 00:33 2024-09-19T06:39:01.038 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-19T06:39:01.074 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-19T06:39:01.121 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-19T06:39:01.149 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-19T06:39:01.196 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T06:39:01.242 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-19T06:39:01.278 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 34/164 2024-09-19T06:39:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T06:39:01.323 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 35/164 2024-09-19T06:39:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 102/164 2024-09-19T06:39:01.360 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 36/164 2024-09-19T06:39:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-19T06:39:01.447 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-19T06:39:01.511 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-19T06:39:01.599 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-19T06:39:01.761 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-19T06:39:01.821 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-19T06:39:01.840 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-19T06:39:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T06:39:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T06:39:01.967 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 106/164 2024-09-19T06:39:02.242 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-09-19T06:39:02.393 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-09-19T06:39:02.393 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-09-19T06:39:02.449 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-19T06:39:02.546 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-19T06:39:02.585 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-19T06:39:02.656 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-19T06:39:02.782 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 46/164 2024-09-19T06:39:02.835 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-19T06:39:03.019 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-19T06:39:03.085 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-19T06:39:03.144 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-19T06:39:03.229 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-19T06:39:03.269 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-19T06:39:03.305 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-19T06:39:03.359 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-19T06:39:03.400 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-19T06:39:03.445 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-19T06:39:03.495 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-19T06:39:03.663 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-19T06:39:03.712 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-19T06:39:03.755 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-19T06:39:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-19T06:39:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-19T06:39:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-19T06:39:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-19T06:39:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-19T06:39:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-19T06:39:04.175 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-09-19T06:39:04.175 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-09-19T06:39:04.273 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-19T06:39:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-19T06:39:04.330 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-19T06:39:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-19T06:39:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-19T06:39:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-19T06:39:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-19T06:39:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-19T06:39:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-19T06:39:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-19T06:39:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-19T06:39:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-19T06:39:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-19T06:39:04.876 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-19T06:39:04.928 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T06:39:05.012 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T06:39:05.012 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-19T06:39:05.012 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-19T06:39:05.012 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:05.039 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-19T06:39:05.044 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T06:39:05.107 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-19T06:39:06.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-19T06:39:06.052 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-19T06:39:06.085 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-19T06:39:06.197 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-19T06:39:06.268 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-19T06:39:06.329 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-19T06:39:06.426 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-19T06:39:06.470 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-19T06:39:06.711 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-19T06:39:06.751 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-19T06:39:06.845 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-19T06:39:07.641 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-19T06:39:07.770 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-19T06:39:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-19T06:39:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-19T06:39:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-19T06:39:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-19T06:39:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-19T06:39:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T06:39:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T06:39:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 102/164 2024-09-19T06:39:10.107 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-19T06:39:10.446 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-09-19T06:39:10.568 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-19T06:39:10.606 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-19T06:39:10.620 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-19T06:39:10.648 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T06:39:10.668 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-19T06:39:10.702 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-19T06:39:10.727 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-19T06:39:10.739 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T06:39:10.766 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 106/164 2024-09-19T06:39:10.770 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-19T06:39:11.102 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-19T06:39:11.220 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:39:11.343 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:39:11.389 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:39:11.462 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:39:11.493 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T06:39:11.639 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-19T06:39:11.689 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-19T06:39:11.717 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:39:11.821 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:39:11.843 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T06:39:11.906 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 14/164 2024-09-19T06:39:11.936 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T06:39:11.960 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-19T06:39:12.154 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T06:39:12.288 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-19T06:39:12.417 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 17/164 2024-09-19T06:39:12.505 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:39:12.589 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:39:12.804 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 18/164 2024-09-19T06:39:12.909 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-19T06:39:13.005 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-19T06:39:13.044 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-19T06:39:13.078 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-19T06:39:13.142 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-19T06:39:13.196 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-19T06:39:13.228 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-19T06:39:13.272 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-19T06:39:13.305 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-19T06:39:13.332 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-19T06:39:13.401 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-19T06:39:13.455 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-19T06:39:13.494 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-19T06:39:13.537 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-19T06:39:13.613 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-19T06:39:13.646 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:13.648 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 34/164 2024-09-19T06:39:13.687 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:13.691 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 35/164 2024-09-19T06:39:13.720 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 36/164 2024-09-19T06:39:13.788 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-19T06:39:13.860 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-19T06:39:13.934 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-19T06:39:14.075 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-19T06:39:14.128 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-19T06:39:14.313 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:14.357 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T06:39:14.482 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T06:39:14.482 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-19T06:39:14.482 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-19T06:39:14.483 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:14.525 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T06:39:14.646 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-19T06:39:14.732 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-19T06:39:14.771 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-19T06:39:14.813 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-19T06:39:14.851 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 46/164 2024-09-19T06:39:14.893 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-19T06:39:15.052 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-19T06:39:15.114 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-19T06:39:15.167 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-19T06:39:15.240 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-19T06:39:15.273 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-19T06:39:15.309 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-19T06:39:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-19T06:39:15.400 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-19T06:39:15.442 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-19T06:39:15.485 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-19T06:39:15.620 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-19T06:39:15.664 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-19T06:39:15.705 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-19T06:39:15.807 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-19T06:39:15.839 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-19T06:39:15.879 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-19T06:39:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-19T06:39:15.929 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-19T06:39:16.011 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-19T06:39:16.145 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-19T06:39:16.186 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-19T06:39:16.214 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-19T06:39:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-19T06:39:16.287 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-19T06:39:16.319 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-19T06:39:16.358 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-19T06:39:16.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-19T06:39:16.425 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-19T06:39:16.467 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-19T06:39:16.501 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-19T06:39:16.586 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-19T06:39:16.640 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-19T06:39:16.671 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-19T06:39:16.773 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-19T06:39:16.822 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-19T06:39:17.535 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-19T06:39:17.594 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-19T06:39:17.645 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-19T06:39:17.743 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-19T06:39:17.791 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-19T06:39:17.836 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-19T06:39:17.930 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T06:39:17.931 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-09-19T06:39:17.931 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-09-19T06:39:17.931 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-09-19T06:39:17.931 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-09-19T06:39:17.931 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-09-19T06:39:17.931 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2024-09-19T06:39:17.931 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-09-19T06:39:17.931 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:17.944 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-19T06:39:17.997 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-19T06:39:18.101 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T06:39:18.119 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T06:39:18.119 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:18.120 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-19T06:39:18.120 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:18.180 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-19T06:39:18.220 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-19T06:39:18.297 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-19T06:39:18.933 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-19T06:39:19.051 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-19T06:39:19.587 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T06:39:19.614 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T06:39:19.615 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:19.615 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-19T06:39:19.615 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T06:39:19.615 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T06:39:19.615 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:19.846 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T06:39:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T06:39:19.870 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:19.870 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-19T06:39:19.870 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T06:39:19.870 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T06:39:19.870 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:20.258 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-19T06:39:20.322 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T06:39:20.331 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-19T06:39:20.345 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T06:39:20.346 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:20.346 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-19T06:39:20.346 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T06:39:20.346 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T06:39:20.346 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:20.359 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-19T06:39:20.393 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-19T06:39:20.460 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-19T06:39:20.517 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-19T06:39:20.573 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-19T06:39:20.703 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-19T06:39:20.756 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T06:39:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-19T06:39:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-19T06:39:20.846 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 101/164 2024-09-19T06:39:20.884 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-19T06:39:20.886 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 102/164 2024-09-19T06:39:20.922 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-19T06:39:20.972 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-19T06:39:21.128 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-19T06:39:21.173 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-19T06:39:21.206 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-19T06:39:21.233 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-19T06:39:21.401 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-19T06:39:21.442 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T06:39:21.513 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 105/164 2024-09-19T06:39:21.536 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 106/164 2024-09-19T06:39:21.688 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T06:39:21.715 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T06:39:21.747 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 126/164 2024-09-19T06:39:21.809 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 127/164 2024-09-19T06:39:21.839 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T06:39:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T06:39:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T06:39:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T06:39:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T06:39:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T06:39:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 131/164 2024-09-19T06:39:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T06:39:24.258 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T06:39:24.260 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:24.260 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-19T06:39:24.260 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T06:39:24.260 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T06:39:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:24.291 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T06:39:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T06:39:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 134/164 2024-09-19T06:39:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 135/164 2024-09-19T06:39:24.833 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 107/164 2024-09-19T06:39:24.869 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T06:39:24.947 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 108/164 2024-09-19T06:39:24.947 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-19T06:39:24.947 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-19T06:39:24.947 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:24.979 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T06:39:25.405 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-19T06:39:25.438 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T06:39:25.463 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T06:39:25.463 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:25.463 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-19T06:39:25.463 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T06:39:25.463 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T06:39:25.463 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:25.506 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T06:39:25.529 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T06:39:25.529 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:25.529 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-19T06:39:25.529 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:25.590 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T06:39:25.614 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T06:39:25.614 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:25.614 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-19T06:39:25.615 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T06:39:25.615 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T06:39:25.615 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:25.933 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T06:39:25.956 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T06:39:25.957 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:25.957 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-19T06:39:25.957 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T06:39:25.957 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T06:39:25.957 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:27.144 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T06:39:27.144 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-09-19T06:39:27.145 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-09-19T06:39:27.145 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-09-19T06:39:27.145 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-09-19T06:39:27.145 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-09-19T06:39:27.145 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-09-19T06:39:27.145 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-09-19T06:39:27.145 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:27.355 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T06:39:27.393 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T06:39:27.393 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:27.393 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-19T06:39:27.393 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:28.958 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T06:39:29.015 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T06:39:29.015 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:29.015 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-19T06:39:29.015 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T06:39:29.015 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T06:39:29.015 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:29.267 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T06:39:29.318 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T06:39:29.318 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:29.318 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-19T06:39:29.318 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T06:39:29.318 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T06:39:29.318 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:29.792 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T06:39:29.842 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T06:39:29.842 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:29.843 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-19T06:39:29.843 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T06:39:29.843 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T06:39:29.843 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:29.998 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-19T06:39:30.068 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-19T06:39:30.132 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-19T06:39:30.357 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-19T06:39:30.421 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-19T06:39:30.470 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 141/164 2024-09-19T06:39:30.511 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 142/164 2024-09-19T06:39:30.515 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-19T06:39:30.552 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 143/164 2024-09-19T06:39:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-19T06:39:30.630 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-19T06:39:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-19T06:39:30.795 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-19T06:39:30.817 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 146/164 2024-09-19T06:39:30.851 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-19T06:39:30.876 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-19T06:39:30.894 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-19T06:39:30.906 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 148/164 2024-09-19T06:39:30.919 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-19T06:39:30.936 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 149/164 2024-09-19T06:39:30.982 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 150/164 2024-09-19T06:39:31.401 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T06:39:31.435 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T06:39:31.476 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 126/164 2024-09-19T06:39:31.544 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 127/164 2024-09-19T06:39:31.560 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:31.574 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T06:39:31.630 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:31.652 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:32.491 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 152/164 2024-09-19T06:39:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-19T06:39:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-19T06:39:32.716 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-19T06:39:32.764 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-19T06:39:32.781 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T06:39:32.790 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-19T06:39:32.818 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T06:39:32.824 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-19T06:39:32.824 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T06:39:32.925 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T06:39:32.925 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T06:39:33.009 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T06:39:33.034 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-19T06:39:33.040 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:33.040 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:33.220 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:33.220 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T06:39:33.321 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T06:39:33.393 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:39:33.893 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T06:39:33.918 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T06:39:34.054 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T06:39:34.232 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 131/164 2024-09-19T06:39:34.270 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T06:39:34.321 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T06:39:34.322 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:34.322 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-19T06:39:34.322 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T06:39:34.322 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T06:39:34.322 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:34.388 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T06:39:34.414 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T06:39:34.520 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 134/164 2024-09-19T06:39:34.579 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 135/164 2024-09-19T06:39:35.563 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-19T06:39:35.599 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T06:39:35.650 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T06:39:35.650 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:35.650 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-19T06:39:35.650 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T06:39:35.650 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T06:39:35.650 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:35.699 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T06:39:35.748 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T06:39:35.749 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:35.749 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-19T06:39:35.749 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:35.816 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T06:39:35.867 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T06:39:35.867 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:35.867 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-19T06:39:35.867 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T06:39:35.867 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T06:39:35.867 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:36.200 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T06:39:36.251 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T06:39:36.252 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:36.252 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-19T06:39:36.252 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T06:39:36.252 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T06:39:36.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:36.773 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 109/164 2024-09-19T06:39:36.774 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-09-19T06:39:36.774 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-09-19T06:39:36.774 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-09-19T06:39:36.774 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-09-19T06:39:36.774 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-09-19T06:39:36.774 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-09-19T06:39:36.774 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-09-19T06:39:36.774 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:36.939 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T06:39:36.954 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 110/164 2024-09-19T06:39:36.955 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:36.955 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-19T06:39:36.955 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:38.441 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T06:39:38.467 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 111/164 2024-09-19T06:39:38.468 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:38.468 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-19T06:39:38.468 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T06:39:38.468 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-19T06:39:38.468 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:38.709 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T06:39:38.732 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 112/164 2024-09-19T06:39:38.733 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:38.733 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-19T06:39:38.733 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T06:39:38.733 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-19T06:39:38.733 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:39.178 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T06:39:39.201 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 113/164 2024-09-19T06:39:39.201 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:39.201 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-19T06:39:39.201 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T06:39:39.201 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-19T06:39:39.201 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:39.307 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-19T06:39:39.364 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-19T06:39:39.427 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-19T06:39:39.612 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-19T06:39:39.779 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-19T06:39:39.779 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-19T06:39:39.823 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-19T06:39:39.960 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-19T06:39:40.012 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-19T06:39:40.048 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-19T06:39:40.072 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-19T06:39:40.386 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:39:40.386 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 1/164 2024-09-19T06:39:40.386 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2/164 2024-09-19T06:39:40.386 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 3/164 2024-09-19T06:39:40.386 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 4/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 5/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 6/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 7/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 11/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 12/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 14/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 15/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 16/164 2024-09-19T06:39:40.387 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 17/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 18/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 19/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 20/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 21/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 22/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 23/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 24/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 25/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 26/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 27/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 28/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 29/164 2024-09-19T06:39:40.388 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 30/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 31/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 32/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 33/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 34/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 35/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 36/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 37/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 38/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 39/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-19T06:39:40.389 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-19T06:39:40.390 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-19T06:39:40.391 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-19T06:39:40.392 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-19T06:39:40.393 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-19T06:39:40.394 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-19T06:39:40.395 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-19T06:39:40.396 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-19T06:39:40.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 149/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 151/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 153/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 155/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 157/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 159/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-19T06:39:40.398 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-19T06:39:40.399 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-19T06:39:40.533 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T06:39:40.562 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 125/164 2024-09-19T06:39:40.601 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 126/164 2024-09-19T06:39:40.663 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 127/164 2024-09-19T06:39:40.693 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T06:39:41.146 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 141/164 2024-09-19T06:39:41.196 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 142/164 2024-09-19T06:39:41.239 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 143/164 2024-09-19T06:39:41.269 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-19T06:39:41.304 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-19T06:39:41.546 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 146/164 2024-09-19T06:39:41.592 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-19T06:39:41.647 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 148/164 2024-09-19T06:39:41.678 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 149/164 2024-09-19T06:39:41.722 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 150/164 2024-09-19T06:39:41.739 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 128/164 2024-09-19T06:39:41.778 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T06:39:42.491 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:42.537 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:42.579 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:42.742 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 129/164 2024-09-19T06:39:42.742 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:39:42.742 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: automake-1.16.2-8.el9.noarch 2024-09-19T06:39:42.743 INFO:teuthology.orchestra.run.smithi118.stdout: bison-3.7.4-5.el9.x86_64 2024-09-19T06:39:42.744 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-19T06:39:42.744 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.744 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.744 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.744 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.744 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.744 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.745 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.745 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.745 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.745 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.745 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.745 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.746 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.746 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.746 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.746 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.746 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.746 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.746 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.747 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.747 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.747 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.747 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.747 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:42.747 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-19T06:39:42.747 INFO:teuthology.orchestra.run.smithi118.stdout: flex-2.6.4-9.el9.x86_64 2024-09-19T06:39:42.748 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-19T06:39:42.748 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-19T06:39:42.748 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-19T06:39:42.748 INFO:teuthology.orchestra.run.smithi118.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-19T06:39:42.748 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-19T06:39:42.749 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-19T06:39:42.749 INFO:teuthology.orchestra.run.smithi118.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-19T06:39:42.749 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-19T06:39:42.749 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-19T06:39:42.749 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.749 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.749 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.749 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-19T06:39:42.750 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-19T06:39:42.750 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-19T06:39:42.750 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-19T06:39:42.750 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-19T06:39:42.750 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.750 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.751 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-19T06:39:42.751 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.751 INFO:teuthology.orchestra.run.smithi118.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-19T06:39:42.751 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-19T06:39:42.751 INFO:teuthology.orchestra.run.smithi118.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-19T06:39:42.751 INFO:teuthology.orchestra.run.smithi118.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-19T06:39:42.752 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-19T06:39:42.752 INFO:teuthology.orchestra.run.smithi118.stdout: lua-5.4.4-4.el9.x86_64 2024-09-19T06:39:42.752 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-19T06:39:42.753 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-19T06:39:42.753 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-19T06:39:42.753 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-19T06:39:42.753 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-19T06:39:42.753 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-19T06:39:42.753 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-13.16-1.el9.x86_64 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.754 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-19T06:39:42.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-19T06:39:42.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-19T06:39:42.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-19T06:39:42.758 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-19T06:39:42.759 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-19T06:39:42.760 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-19T06:39:42.760 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-19T06:39:42.760 INFO:teuthology.orchestra.run.smithi118.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-19T06:39:42.760 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-19T06:39:42.761 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-19T06:39:42.761 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-19T06:39:42.761 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-19T06:39:42.761 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-19T06:39:42.761 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-19T06:39:42.761 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-19T06:39:42.762 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.762 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.762 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:42.762 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-20.el9.x86_64 2024-09-19T06:39:42.762 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-19T06:39:42.763 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-19T06:39:42.763 INFO:teuthology.orchestra.run.smithi118.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-19T06:39:42.763 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-19T06:39:42.763 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-19T06:39:42.764 INFO:teuthology.orchestra.run.smithi118.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-19T06:39:42.764 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:39:42.764 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-19T06:39:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T06:39:42.873 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 130/164 2024-09-19T06:39:43.032 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 131/164 2024-09-19T06:39:43.064 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T06:39:43.087 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 132/164 2024-09-19T06:39:43.088 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:43.088 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-19T06:39:43.088 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T06:39:43.088 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-19T06:39:43.088 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:43.132 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T06:39:43.143 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 133/164 2024-09-19T06:39:43.185 DEBUG:teuthology.parallel:result is None 2024-09-19T06:39:43.242 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 134/164 2024-09-19T06:39:43.296 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 135/164 2024-09-19T06:39:43.856 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 152/164 2024-09-19T06:39:43.961 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-19T06:39:44.025 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-19T06:39:44.146 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-19T06:39:44.180 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-19T06:39:44.205 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-19T06:39:44.221 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-19T06:39:44.241 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-19T06:39:44.241 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T06:39:44.251 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T06:39:44.274 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 137/164 2024-09-19T06:39:44.274 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:44.274 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-19T06:39:44.274 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T06:39:44.275 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-19T06:39:44.275 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:44.318 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T06:39:44.340 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-361.g2f599a 138/164 2024-09-19T06:39:44.340 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:44.341 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-19T06:39:44.341 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:44.349 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T06:39:44.349 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T06:39:44.393 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T06:39:44.416 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 139/164 2024-09-19T06:39:44.417 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:44.417 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-19T06:39:44.417 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T06:39:44.417 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-19T06:39:44.417 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:44.467 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T06:39:44.491 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-19T06:39:44.507 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:44.507 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:44.719 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T06:39:44.742 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 140/164 2024-09-19T06:39:44.742 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-19T06:39:44.742 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-19T06:39:44.742 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T06:39:44.742 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-19T06:39:44.743 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:39:44.752 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:44.752 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T06:39:44.853 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T06:39:44.925 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:39:45.138 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:39:45.138 INFO:teuthology.orchestra.run.smithi114.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-19T06:39:45.138 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:49.298 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 141/164 2024-09-19T06:39:49.348 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 142/164 2024-09-19T06:39:49.388 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 143/164 2024-09-19T06:39:49.420 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-19T06:39:49.448 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-19T06:39:49.628 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 146/164 2024-09-19T06:39:49.669 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-19T06:39:49.700 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 148/164 2024-09-19T06:39:49.729 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 149/164 2024-09-19T06:39:49.775 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 150/164 2024-09-19T06:39:50.361 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:50.398 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:50.418 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 1/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 3/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 4/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 5/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 6/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 7/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:39:50.807 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 11/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 12/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 14/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 15/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 16/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 17/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 18/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 19/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 20/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 21/164 2024-09-19T06:39:50.808 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 22/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 23/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 24/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 25/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 26/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 27/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 28/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 29/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 30/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 31/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 32/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 33/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 34/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 35/164 2024-09-19T06:39:50.809 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 36/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 37/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 38/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 39/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-19T06:39:50.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-19T06:39:50.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-19T06:39:50.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-19T06:39:50.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-19T06:39:50.814 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-19T06:39:50.815 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-19T06:39:50.816 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-19T06:39:50.817 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 149/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 151/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 153/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 155/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 157/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 159/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-19T06:39:50.818 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-19T06:39:50.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-19T06:39:50.819 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-19T06:39:51.242 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 152/164 2024-09-19T06:39:51.336 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-19T06:39:51.377 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-19T06:39:51.457 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-19T06:39:51.498 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-19T06:39:51.523 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-19T06:39:51.549 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-19T06:39:51.549 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T06:39:51.633 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-19T06:39:51.633 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T06:39:51.726 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-19T06:39:51.750 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-19T06:39:51.756 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:51.756 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:51.952 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-19T06:39:51.953 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T06:39:52.036 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-19T06:39:52.099 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:39:53.002 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:39:53.002 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.2-8.el9.noarch 2024-09-19T06:39:53.003 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.7.4-5.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.004 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:39:53.005 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.4-9.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.006 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-19T06:39:53.007 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.008 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-19T06:39:53.008 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-19T06:39:53.008 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-19T06:39:53.008 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-19T06:39:53.008 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-19T06:39:53.008 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-09-19T06:39:53.008 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-19T06:39:53.008 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-13.16-1.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-19T06:39:53.009 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-19T06:39:53.010 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-19T06:39:53.011 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-19T06:39:53.011 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-19T06:39:53.011 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-19T06:39:53.011 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-19T06:39:53.011 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-19T06:39:53.011 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-19T06:39:53.011 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-19T06:39:53.011 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-19T06:39:53.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-19T06:39:53.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-19T06:39:53.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-19T06:39:53.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-19T06:39:53.012 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-19T06:39:53.013 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-19T06:39:53.014 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:39:53.016 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-20.el9.x86_64 2024-09-19T06:39:53.016 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-19T06:39:53.016 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-19T06:39:53.016 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-19T06:39:53.016 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-19T06:39:53.016 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-19T06:39:53.016 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-19T06:39:53.017 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:39:53.017 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-09-19T06:39:53.642 DEBUG:teuthology.parallel:result is None 2024-09-19T06:39:57.712 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:39:57.712 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.1.1-361.g2f599a46.el9.x86_64 1/164 2024-09-19T06:39:57.712 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2/164 2024-09-19T06:39:57.712 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 3/164 2024-09-19T06:39:57.712 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 4/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-361.g2f599a 5/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 6/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 7/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 8/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 9/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 10/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 11/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 12/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 13/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 14/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_6 15/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 16/164 2024-09-19T06:39:57.713 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 17/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 18/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_ 19/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 20/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9 21/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x 22/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 23/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 24/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 25/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 26/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 27/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 28/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 29/164 2024-09-19T06:39:57.714 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.e 30/164 2024-09-19T06:39:57.726 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noar 31/164 2024-09-19T06:39:57.726 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.no 32/164 2024-09-19T06:39:57.726 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-361.g2f59 33/164 2024-09-19T06:39:57.726 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9 34/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 35/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el 36/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 37/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.1.1-361.g2f599a46.el9.noarch 38/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 39/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-19T06:39:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-19T06:39:57.728 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-19T06:39:57.729 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-19T06:39:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-19T06:39:57.731 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-19T06:39:57.732 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-19T06:39:57.733 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-19T06:39:57.734 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-19T06:39:57.735 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.1.1-361.g2f599a46.el9.x86_64 149/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 151/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 153/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 155/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 157/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 159/164 2024-09-19T06:39:57.736 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-19T06:39:57.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-19T06:39:57.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-19T06:39:57.737 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-19T06:40:00.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-19T06:40:00.383 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:40:00.383 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-09-19T06:40:00.383 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-19T06:40:00.383 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-19T06:40:00.383 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-19T06:40:00.383 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.384 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.385 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:19.1.1-361.g2f599a46.el9.noarch 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-19T06:40:00.386 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-19T06:40:00.387 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-19T06:40:00.388 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.16-1.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-19T06:40:00.389 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-19T06:40:00.390 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-19T06:40:00.391 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-09-19T06:40:00.392 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-19T06:40:00.393 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-19T06:40:00.394 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.1.1-361.g2f599a46.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-20.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-19T06:40:00.395 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-19T06:40:00.396 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-19T06:40:00.396 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:40:00.396 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-09-19T06:40:00.798 DEBUG:teuthology.parallel:result is None 2024-09-19T06:40:00.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:40:00.946 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-19T06:40:00.996 INFO:teuthology.orchestra.run.smithi114.stdout:19.1.1-361.g2f599a46.el9 2024-09-19T06:40:00.996 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-361.g2f599a46.el9 2024-09-19T06:40:00.997 INFO:teuthology.task.install:The correct ceph version 19.1.1-361.g2f599a46 is installed. 2024-09-19T06:40:00.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:40:01.140 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-19T06:40:01.163 INFO:teuthology.orchestra.run.smithi116.stdout:19.1.1-361.g2f599a46.el9 2024-09-19T06:40:01.163 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-361.g2f599a46.el9 2024-09-19T06:40:01.163 INFO:teuthology.task.install:The correct ceph version 19.1.1-361.g2f599a46 is installed. 2024-09-19T06:40:01.165 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:40:01.304 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-19T06:40:01.331 INFO:teuthology.orchestra.run.smithi118.stdout:19.1.1-361.g2f599a46.el9 2024-09-19T06:40:01.331 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-361.g2f599a46.el9 2024-09-19T06:40:01.331 INFO:teuthology.task.install:The correct ceph version 19.1.1-361.g2f599a46 is installed. 2024-09-19T06:40:01.333 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-19T06:40:01.334 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:40:01.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-19T06:40:01.399 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:40:01.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-19T06:40:01.428 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:40:01.428 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-19T06:40:01.461 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-19T06:40:01.461 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:40:01.461 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-09-19T06:40:01.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-19T06:40:01.614 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:40:01.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-09-19T06:40:01.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-19T06:40:01.706 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:40:01.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-09-19T06:40:01.732 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-19T06:40:01.798 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-19T06:40:01.798 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:40:01.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-19T06:40:01.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-19T06:40:01.953 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:40:01.953 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-19T06:40:01.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-19T06:40:02.045 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:40:02.045 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-19T06:40:02.072 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-19T06:40:02.137 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-19T06:40:02.137 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:40:02.137 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-09-19T06:40:02.194 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-19T06:40:02.291 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:40:02.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-09-19T06:40:02.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-19T06:40:02.383 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:40:02.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-09-19T06:40:02.408 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-19T06:40:02.472 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-19T06:40:02.571 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-19T06:40:02.571 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:40:02.572 INFO:tasks.cephadm:Cluster fsid is fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:40:02.572 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-19T06:40:02.573 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.114', 'mon.b': '172.21.15.116', 'mon.c': '172.21.15.118'} 2024-09-19T06:40:02.573 INFO:tasks.cephadm:First mon is mon.a on smithi114 2024-09-19T06:40:02.573 INFO:tasks.cephadm:First mgr is x 2024-09-19T06:40:02.573 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-19T06:40:02.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-09-19T06:40:02.632 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-09-19T06:40:02.658 DEBUG:teuthology.orchestra.run.smithi118:> sudo hostname $(hostname -s) 2024-09-19T06:40:02.683 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-19T06:40:02.683 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:40:02.820 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2f599a46ad10e37c929e5c43da2f6907ddcd1961', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56106/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-361-g2f599a46', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-361.g2f599a46'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-18 17:52:24.287651', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-19T06:40:02.955 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 2f599a46ad10e37c929e5c43da2f6907ddcd1961 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:40:02.956 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm 2024-09-19T06:40:02.957 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm 2024-09-19T06:40:02.957 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:03.393 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 784313 Sep 19 06:40 /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:03.394 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:03.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 784313 Sep 19 06:40 /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:03.774 DEBUG:teuthology.orchestra.run.smithi118:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2f599a46ad10e37c929e5c43da2f6907ddcd1961/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:04.138 INFO:teuthology.orchestra.run.smithi118.stdout:-rw-r--r--. 1 ubuntu ubuntu 784313 Sep 19 06:40 /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:04.139 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:04.178 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:04.198 DEBUG:teuthology.orchestra.run.smithi118:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-19T06:40:04.234 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 on all hosts... 2024-09-19T06:40:04.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 pull 2024-09-19T06:40:04.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 pull 2024-09-19T06:40:04.241 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 pull 2024-09-19T06:40:05.860 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T06:40:06.024 INFO:teuthology.orchestra.run.smithi118.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T06:40:06.156 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T06:40:34.087 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-19T06:40:34.088 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T06:40:34.088 INFO:teuthology.orchestra.run.smithi118.stdout: "image_id": "d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d", 2024-09-19T06:40:34.088 INFO:teuthology.orchestra.run.smithi118.stdout: "repo_digests": [ 2024-09-19T06:40:34.088 INFO:teuthology.orchestra.run.smithi118.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e" 2024-09-19T06:40:34.088 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-19T06:40:34.088 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-19T06:40:36.511 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-09-19T06:40:36.511 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T06:40:36.512 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d", 2024-09-19T06:40:36.512 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-09-19T06:40:36.512 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e" 2024-09-19T06:40:36.512 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-09-19T06:40:36.512 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-09-19T06:40:37.319 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:40:37.320 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T06:40:37.320 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d", 2024-09-19T06:40:37.320 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-09-19T06:40:37.320 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e" 2024-09-19T06:40:37.320 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-09-19T06:40:37.320 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:40:37.351 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-09-19T06:40:37.409 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-09-19T06:40:37.443 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph 2024-09-19T06:40:37.477 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-09-19T06:40:37.532 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-09-19T06:40:37.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /etc/ceph 2024-09-19T06:40:37.589 INFO:tasks.cephadm:Writing seed config... 2024-09-19T06:40:37.591 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-19T06:40:37.592 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-09-19T06:40:37.593 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-09-19T06:40:37.593 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-09-19T06:40:37.593 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-09-19T06:40:37.593 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-19T06:40:37.593 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-19T06:40:37.593 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-19T06:40:37.593 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-19T06:40:37.593 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-19T06:40:37.594 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-19T06:40:37.595 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-19T06:40:37.596 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-19T06:40:37.596 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-19T06:40:37.596 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-19T06:40:37.596 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-19T06:40:37.596 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-19T06:40:37.596 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-19T06:40:37.596 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-09-19T06:40:37.597 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:40:37.597 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-19T06:40:37.632 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = fface640-7651-11ef-bceb-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-19T06:40:37.632 DEBUG:teuthology.orchestra.run.smithi114:mon.a> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mon.a.service 2024-09-19T06:40:37.678 DEBUG:teuthology.orchestra.run.smithi114:mgr.x> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mgr.x.service 2024-09-19T06:40:37.723 INFO:tasks.cephadm:Bootstrapping... 2024-09-19T06:40:37.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 -v bootstrap --fsid fface640-7651-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.114 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-19T06:40:37.980 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-09-19T06:40:37.980 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961', '-v', 'bootstrap', '--fsid', 'fface640-7651-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.114', '--skip-admin-label'] 2024-09-19T06:40:37.980 INFO:teuthology.orchestra.run.smithi114.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-19T06:40:37.981 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying podman|docker is present... 2024-09-19T06:40:38.025 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-19T06:40:38.025 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying lvm2 is present... 2024-09-19T06:40:38.025 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying time synchronization is in place... 2024-09-19T06:40:38.038 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-19T06:40:38.039 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-19T06:40:38.051 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-19T06:40:38.051 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-19T06:40:38.064 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-09-19T06:40:38.077 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-09-19T06:40:38.077 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-09-19T06:40:38.077 INFO:teuthology.orchestra.run.smithi114.stdout:Repeating the final host check... 2024-09-19T06:40:38.121 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 5.2.2 2024-09-19T06:40:38.121 INFO:teuthology.orchestra.run.smithi114.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-19T06:40:38.122 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl is present 2024-09-19T06:40:38.122 INFO:teuthology.orchestra.run.smithi114.stdout:lvcreate is present 2024-09-19T06:40:38.134 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-19T06:40:38.134 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-19T06:40:38.146 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-19T06:40:38.147 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-19T06:40:38.160 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-09-19T06:40:38.172 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-09-19T06:40:38.172 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-09-19T06:40:38.172 INFO:teuthology.orchestra.run.smithi114.stdout:Host looks OK 2024-09-19T06:40:38.172 INFO:teuthology.orchestra.run.smithi114.stdout:Cluster fsid: fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:40:38.172 INFO:teuthology.orchestra.run.smithi114.stdout:Acquiring lock 140380402676736 on /run/cephadm/fface640-7651-11ef-bceb-c7b262605968.lock 2024-09-19T06:40:38.173 INFO:teuthology.orchestra.run.smithi114.stdout:Lock 140380402676736 acquired on /run/cephadm/fface640-7651-11ef-bceb-c7b262605968.lock 2024-09-19T06:40:38.173 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 3300 ... 2024-09-19T06:40:38.173 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 6789 ... 2024-09-19T06:40:38.174 INFO:teuthology.orchestra.run.smithi114.stdout:Base mon IP(s) is [172.21.15.114:3300, 172.21.15.114:6789], mon addrv is [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-19T06:40:38.181 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.114 metric 100 2024-09-19T06:40:38.181 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.114 metric 100 2024-09-19T06:40:38.187 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-19T06:40:38.187 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-19T06:40:38.187 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-19T06:40:38.192 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-19T06:40:38.192 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-19T06:40:38.192 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-19T06:40:38.193 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-19T06:40:38.193 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link noprefixroute 2024-09-19T06:40:38.193 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-19T06:40:38.193 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-09-19T06:40:38.193 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-09-19T06:40:38.193 INFO:teuthology.orchestra.run.smithi114.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-19T06:40:38.194 INFO:teuthology.orchestra.run.smithi114.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-19T06:40:38.195 INFO:teuthology.orchestra.run.smithi114.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T06:40:38.798 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d 2024-09-19T06:40:38.799 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961... 2024-09-19T06:40:38.799 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Getting image source signatures 2024-09-19T06:40:38.799 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:c4221751d16d26d2a4b7031ebd71ecffffff13c8cf7401cf95ce80dab3b6d67a 2024-09-19T06:40:38.799 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-19T06:40:38.799 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying config sha256:d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d 2024-09-19T06:40:38.799 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-19T06:40:39.868 INFO:teuthology.orchestra.run.smithi114.stdout:ceph: stdout ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable) 2024-09-19T06:40:39.868 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph version: ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable) 2024-09-19T06:40:39.869 INFO:teuthology.orchestra.run.smithi114.stdout:Extracting ceph user uid/gid from container image... 2024-09-19T06:40:40.884 INFO:teuthology.orchestra.run.smithi114.stdout:stat: stdout 167 167 2024-09-19T06:40:40.884 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial keys... 2024-09-19T06:40:51.428 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQByx+tmOJG6MBAAy0WgqTtoz/o1YUVT26U3FQ== 2024-09-19T06:40:52.444 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQBzx+tmnnpbMBAAAvYS/vsygYOJXRG2fsaWlA== 2024-09-19T06:40:53.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQB0x+tmMqcqNBAApp0tUU/6H1TQTYvc9wE3iQ== 2024-09-19T06:40:53.602 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial monmap... 2024-09-19T06:40:55.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool for a [v2:172.21.15.114:3300,v1:172.21.15.114:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: set fsid to fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:40:55.248 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mon... 2024-09-19T06:40:56.546 INFO:teuthology.orchestra.run.smithi114.stdout:create mon.a on 2024-09-19T06:40:56.841 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-19T06:40:57.060 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-19T06:40:57.314 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-fface640-7651-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-fface640-7651-11ef-bceb-c7b262605968.target. 2024-09-19T06:40:57.314 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-fface640-7651-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-fface640-7651-11ef-bceb-c7b262605968.target. 2024-09-19T06:40:57.641 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fface640-7651-11ef-bceb-c7b262605968@mon.a 2024-09-19T06:40:57.641 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fface640-7651-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-fface640-7651-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-19T06:40:57.911 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fface640-7651-11ef-bceb-c7b262605968.target.wants/ceph-fface640-7651-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-fface640-7651-11ef-bceb-c7b262605968@.service. 2024-09-19T06:40:58.571 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:58 smithi114 podman[30438]: 2024-09-19 06:40:58.507903257 +0000 UTC m=+0.362717380 container start 24fb34e7dad2bda204d5282fd091ee99231cab3cdb42b31c9711e275e56b60d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-19T06:40:58.628 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-19T06:40:58.628 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-19T06:40:58.641 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-19T06:40:58.641 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-19T06:40:58.641 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-19T06:40:58.641 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-09-19T06:40:58.642 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon to start... 2024-09-19T06:40:58.642 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon... 2024-09-19T06:40:58.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:58 smithi114 ceph-mon[30457]: mkfs fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:40:58.914 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:58 smithi114 ceph-mon[30457]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-19T06:40:58.914 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:58 smithi114 bash[30438]: 24fb34e7dad2bda204d5282fd091ee99231cab3cdb42b31c9711e275e56b60d1 2024-09-19T06:40:58.914 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:58 smithi114 systemd[1]: Started Ceph mon.a for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:40:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-19T06:40:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: monmap epoch 1 2024-09-19T06:40:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:40:59.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: last_changed 2024-09-19T06:40:54.002091+0000 2024-09-19T06:40:59.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:40:59.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: min_mon_release 19 (squid) 2024-09-19T06:40:59.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: election_strategy: 1 2024-09-19T06:40:59.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:40:59.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: fsmap 2024-09-19T06:40:59.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: osdmap e1: 0 total, 0 up, 0 in 2024-09-19T06:40:59.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: mgrmap e1: no daemons active 2024-09-19T06:40:59.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:40:59 smithi114 ceph-mon[30457]: from='client.? 172.21.15.114:0/2091836846' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-19T06:40:59.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout cluster: 2024-09-19T06:40:59.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout id: fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:40:59.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-19T06:40:59.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:40:59.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout services: 2024-09-19T06:40:59.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.637007s) 2024-09-19T06:40:59.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-19T06:40:59.787 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout data: 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pgs: 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.126+0000 7f2e7de00640 1 Processor -- start 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.126+0000 7f2e7de00640 1 -- start start 2024-09-19T06:40:59.788 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.126+0000 7f2e7de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7807c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.127+0000 7f2e7de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7807c840 con 0x7f2e7807de00 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.127+0000 7f2e76a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7807c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.127+0000 7f2e76a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7807c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50646/0 (socket says 172.21.15.114:50646) 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.127+0000 7f2e76a00640 1 -- 172.21.15.114:0/1935585335 learned_addr learned my addr 172.21.15.114:0/1935585335 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.128+0000 7f2e76a00640 1 -- 172.21.15.114:0/1935585335 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7807c9c0 con 0x7f2e7807de00 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.130+0000 7f2e76a00640 1 --2- 172.21.15.114:0/1935585335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7807c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009820 tx=0x7f2e6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1858af606b9c693 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e75600640 1 -- 172.21.15.114:0/1935585335 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e68037070 con 0x7f2e7807de00 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e75600640 1 -- 172.21.15.114:0/1935585335 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e68032440 con 0x7f2e7807de00 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e75600640 1 -- 172.21.15.114:0/1935585335 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e68032740 con 0x7f2e7807de00 2024-09-19T06:40:59.789 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e7de00640 1 -- 172.21.15.114:0/1935585335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 msgr2=0x7f2e7807c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e7de00640 1 --2- 172.21.15.114:0/1935585335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7807c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009820 tx=0x7f2e6802a270 comp rx=0 tx=0).stop 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e7de00640 1 -- 172.21.15.114:0/1935585335 shutdown_connections 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e7de00640 1 --2- 172.21.15.114:0/1935585335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7807c300 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e7de00640 1 -- 172.21.15.114:0/1935585335 >> 172.21.15.114:0/1935585335 conn(0x7f2e78102e50 msgr2=0x7f2e78105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e7de00640 1 -- 172.21.15.114:0/1935585335 shutdown_connections 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.131+0000 7f2e7de00640 1 -- 172.21.15.114:0/1935585335 wait complete. 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.132+0000 7f2e7de00640 1 Processor -- start 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.132+0000 7f2e7de00640 1 -- start start 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.133+0000 7f2e7de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.133+0000 7f2e7de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7807cf00 con 0x7f2e7807de00 2024-09-19T06:40:59.790 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.133+0000 7f2e76a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.133+0000 7f2e76a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7819cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50658/0 (socket says 172.21.15.114:50658) 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.133+0000 7f2e76a00640 1 -- 172.21.15.114:0/2091836846 learned_addr learned my addr 172.21.15.114:0/2091836846 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.133+0000 7f2e76a00640 1 -- 172.21.15.114:0/2091836846 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7819d4f0 con 0x7f2e7807de00 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.134+0000 7f2e76a00640 1 --2- 172.21.15.114:0/2091836846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7819cfb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2e680090b0 tx=0x7f2e680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.134+0000 7f2e7ca00640 1 -- 172.21.15.114:0/2091836846 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e68041070 con 0x7f2e7807de00 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.134+0000 7f2e7ca00640 1 -- 172.21.15.114:0/2091836846 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6803c7a0 con 0x7f2e7807de00 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.134+0000 7f2e7ca00640 1 -- 172.21.15.114:0/2091836846 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e68037040 con 0x7f2e7807de00 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.134+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7819d780 con 0x7f2e7807de00 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.134+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e781a0470 con 0x7f2e7807de00 2024-09-19T06:40:59.791 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.134+0000 7f2e7ca00640 1 -- 172.21.15.114:0/2091836846 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2e68044070 con 0x7f2e7807de00 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.134+0000 7f2e7ca00640 1 -- 172.21.15.114:0/2091836846 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2e6803d3f0 con 0x7f2e7807de00 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.135+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e7807c380 con 0x7f2e7807de00 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.138+0000 7f2e7ca00640 1 -- 172.21.15.114:0/2091836846 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f2e68046450 con 0x7f2e7807de00 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.201+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f2e7802d910 con 0x7f2e7807de00 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.201+0000 7f2e7ca00640 1 -- 172.21.15.114:0/2091836846 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f2e68046630 con 0x7f2e7807de00 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.202+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 msgr2=0x7f2e7819cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.202+0000 7f2e7de00640 1 --2- 172.21.15.114:0/2091836846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7819cfb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2e680090b0 tx=0x7f2e680046c0 comp rx=0 tx=0).stop 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.202+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 shutdown_connections 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.202+0000 7f2e7de00640 1 --2- 172.21.15.114:0/2091836846 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e7807de00 0x7f2e7819cfb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.202+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 >> 172.21.15.114:0/2091836846 conn(0x7f2e78102e50 msgr2=0x7f2e78103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:40:59.792 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.202+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 shutdown_connections 2024-09-19T06:40:59.793 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:40:59.202+0000 7f2e7de00640 1 -- 172.21.15.114:0/2091836846 wait complete. 2024-09-19T06:40:59.793 INFO:teuthology.orchestra.run.smithi114.stdout:mon is available 2024-09-19T06:40:59.793 INFO:teuthology.orchestra.run.smithi114.stdout:Assimilating anything we can from ceph.conf... 2024-09-19T06:41:00.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:00 smithi114 ceph-mon[30457]: from='client.? 172.21.15.114:0/1018293454' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-19T06:41:00.672 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:00 smithi114 ceph-mon[30457]: from='client.? 172.21.15.114:0/1018293454' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-19T06:41:01.012 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:01.012 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-09-19T06:41:01.012 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:41:01.012 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-19T06:41:01.012 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-19T06:41:01.012 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-19T06:41:01.012 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-19T06:41:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-19T06:41:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-19T06:41:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-19T06:41:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-09-19T06:41:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-19T06:41:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-19T06:41:01.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-19T06:41:01.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-19T06:41:01.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-19T06:41:01.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.274+0000 7f1e65a00640 1 Processor -- start 2024-09-19T06:41:01.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.274+0000 7f1e65a00640 1 -- start start 2024-09-19T06:41:01.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.274+0000 7f1e65a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e60109d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:01.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.274+0000 7f1e65a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6010a310 con 0x7f1e60109920 2024-09-19T06:41:01.014 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.275+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e60109d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.275+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e60109d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50670/0 (socket says 172.21.15.114:50670) 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.275+0000 7f1e5ea00640 1 -- 172.21.15.114:0/3261224534 learned_addr learned my addr 172.21.15.114:0/3261224534 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.275+0000 7f1e5ea00640 1 -- 172.21.15.114:0/3261224534 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6010aa10 con 0x7f1e60109920 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.276+0000 7f1e5ea00640 1 --2- 172.21.15.114:0/3261224534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e60109d40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009820 tx=0x7f1e5002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0aca6734b4ecab server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.276+0000 7f1e5d600640 1 -- 172.21.15.114:0/3261224534 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e50037070 con 0x7f1e60109920 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.276+0000 7f1e5d600640 1 -- 172.21.15.114:0/3261224534 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e50032440 con 0x7f1e60109920 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.276+0000 7f1e5d600640 1 -- 172.21.15.114:0/3261224534 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e50032740 con 0x7f1e60109920 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.277+0000 7f1e65a00640 1 -- 172.21.15.114:0/3261224534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 msgr2=0x7f1e60109d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.277+0000 7f1e65a00640 1 --2- 172.21.15.114:0/3261224534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e60109d40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009820 tx=0x7f1e5002a270 comp rx=0 tx=0).stop 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.277+0000 7f1e65a00640 1 -- 172.21.15.114:0/3261224534 shutdown_connections 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.277+0000 7f1e65a00640 1 --2- 172.21.15.114:0/3261224534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e60109d40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:01.015 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.277+0000 7f1e65a00640 1 -- 172.21.15.114:0/3261224534 >> 172.21.15.114:0/3261224534 conn(0x7f1e6007cf40 msgr2=0x7f1e6007d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.277+0000 7f1e65a00640 1 -- 172.21.15.114:0/3261224534 shutdown_connections 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.277+0000 7f1e65a00640 1 -- 172.21.15.114:0/3261224534 wait complete. 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.278+0000 7f1e65a00640 1 Processor -- start 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.278+0000 7f1e65a00640 1 -- start start 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.278+0000 7f1e65a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e6019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.278+0000 7f1e65a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6010aee0 con 0x7f1e60109920 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.279+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e6019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.279+0000 7f1e5ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e6019cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50672/0 (socket says 172.21.15.114:50672) 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.279+0000 7f1e5ea00640 1 -- 172.21.15.114:0/1018293454 learned_addr learned my addr 172.21.15.114:0/1018293454 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.279+0000 7f1e5ea00640 1 -- 172.21.15.114:0/1018293454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6019d510 con 0x7f1e60109920 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.279+0000 7f1e5ea00640 1 --2- 172.21.15.114:0/1018293454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e6019cfd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1e500090b0 tx=0x7f1e500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.280+0000 7f1e57400640 1 -- 172.21.15.114:0/1018293454 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e50041070 con 0x7f1e60109920 2024-09-19T06:41:01.016 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.280+0000 7f1e57400640 1 -- 172.21.15.114:0/1018293454 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e5003c7a0 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.280+0000 7f1e57400640 1 -- 172.21.15.114:0/1018293454 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1e50037040 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.280+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6019d7a0 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.280+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e601a0490 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.281+0000 7f1e57400640 1 -- 172.21.15.114:0/1018293454 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1e50044070 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.281+0000 7f1e57400640 1 -- 172.21.15.114:0/1018293454 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1e5003d3f0 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.281+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e6010b580 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.284+0000 7f1e57400640 1 -- 172.21.15.114:0/1018293454 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f1e50046450 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.343+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1e601a1b60 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.348+0000 7f1e57400640 1 -- 172.21.15.114:0/1018293454 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7f1e50046630 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.349+0000 7f1e57400640 1 -- 172.21.15.114:0/1018293454 <== mon.0 v2:172.21.15.114:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1e50056070 con 0x7f1e60109920 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.349+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 msgr2=0x7f1e6019cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:01.017 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.349+0000 7f1e65a00640 1 --2- 172.21.15.114:0/1018293454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e6019cfd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1e500090b0 tx=0x7f1e500046c0 comp rx=0 tx=0).stop 2024-09-19T06:41:01.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.349+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 shutdown_connections 2024-09-19T06:41:01.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.349+0000 7f1e65a00640 1 --2- 172.21.15.114:0/1018293454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1e60109920 0x7f1e6019cfd0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:01.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.349+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 >> 172.21.15.114:0/1018293454 conn(0x7f1e6007cf40 msgr2=0x7f1e601912c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:01.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.350+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 shutdown_connections 2024-09-19T06:41:01.018 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:00.350+0000 7f1e65a00640 1 -- 172.21.15.114:0/1018293454 wait complete. 2024-09-19T06:41:01.018 INFO:teuthology.orchestra.run.smithi114.stdout:Generating new minimal ceph.conf... 2024-09-19T06:41:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:01 smithi114 ceph-mon[30457]: from='client.? 172.21.15.114:0/978185027' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.475+0000 7f4bbc000640 1 Processor -- start 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.475+0000 7f4bbc000640 1 -- start start 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.476+0000 7f4bbc000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb4109c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.476+0000 7f4bbc000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb410a240 con 0x7f4bb4109870 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.476+0000 7f4bb9600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb4109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.476+0000 7f4bb9600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb4109c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50682/0 (socket says 172.21.15.114:50682) 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.476+0000 7f4bb9600640 1 -- 172.21.15.114:0/3373970376 learned_addr learned my addr 172.21.15.114:0/3373970376 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.476+0000 7f4bb9600640 1 -- 172.21.15.114:0/3373970376 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb410aa10 con 0x7f4bb4109870 2024-09-19T06:41:02.315 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.477+0000 7f4bb9600640 1 --2- 172.21.15.114:0/3373970376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb4109c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009820 tx=0x7f4ba802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26b04811640b89d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bb3e00640 1 -- 172.21.15.114:0/3373970376 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8037070 con 0x7f4bb4109870 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bb3e00640 1 -- 172.21.15.114:0/3373970376 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4ba8032900 con 0x7f4bb4109870 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bb3e00640 1 -- 172.21.15.114:0/3373970376 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8030850 con 0x7f4bb4109870 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bbc000640 1 -- 172.21.15.114:0/3373970376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 msgr2=0x7f4bb4109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bbc000640 1 --2- 172.21.15.114:0/3373970376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb4109c70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009820 tx=0x7f4ba802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bbc000640 1 -- 172.21.15.114:0/3373970376 shutdown_connections 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bbc000640 1 --2- 172.21.15.114:0/3373970376 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb4109c70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bbc000640 1 -- 172.21.15.114:0/3373970376 >> 172.21.15.114:0/3373970376 conn(0x7f4bb407cf40 msgr2=0x7f4bb407d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bbc000640 1 -- 172.21.15.114:0/3373970376 shutdown_connections 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.478+0000 7f4bbc000640 1 -- 172.21.15.114:0/3373970376 wait complete. 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.479+0000 7f4bbc000640 1 Processor -- start 2024-09-19T06:41:02.316 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.479+0000 7f4bbc000640 1 -- start start 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.480+0000 7f4bbc000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.480+0000 7f4bbc000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb410ddc0 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.480+0000 7f4bb9600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.480+0000 7f4bb9600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb419d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50688/0 (socket says 172.21.15.114:50688) 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.480+0000 7f4bb9600640 1 -- 172.21.15.114:0/978185027 learned_addr learned my addr 172.21.15.114:0/978185027 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.480+0000 7f4bb9600640 1 -- 172.21.15.114:0/978185027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb419dd20 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.480+0000 7f4bb9600640 1 --2- 172.21.15.114:0/978185027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb419d7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8032ce0 tx=0x7f4ba8032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.481+0000 7f4bb2000640 1 -- 172.21.15.114:0/978185027 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8042070 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.481+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb419dfb0 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.481+0000 7f4bb2000640 1 -- 172.21.15.114:0/978185027 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4ba80308c0 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.481+0000 7f4bb2000640 1 -- 172.21.15.114:0/978185027 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8037040 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.481+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb41a0ca0 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.481+0000 7f4bb2000640 1 -- 172.21.15.114:0/978185027 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4ba8045070 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.481+0000 7f4bb2000640 1 -- 172.21.15.114:0/978185027 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4ba8047b70 con 0x7f4bb4109870 2024-09-19T06:41:02.317 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.482+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb4109c70 con 0x7f4bb4109870 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.484+0000 7f4bb2000640 1 -- 172.21.15.114:0/978185027 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f4ba803dbf0 con 0x7f4bb4109870 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.544+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f4bb41a1210 con 0x7f4bb4109870 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.545+0000 7f4bb2000640 1 -- 172.21.15.114:0/978185027 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f4ba803d430 con 0x7f4bb4109870 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.546+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 msgr2=0x7f4bb419d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.546+0000 7f4bbc000640 1 --2- 172.21.15.114:0/978185027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb419d7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8032ce0 tx=0x7f4ba8032d10 comp rx=0 tx=0).stop 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.546+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 shutdown_connections 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.546+0000 7f4bbc000640 1 --2- 172.21.15.114:0/978185027 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bb4109870 0x7f4bb419d7e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.546+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 >> 172.21.15.114:0/978185027 conn(0x7f4bb407cf40 msgr2=0x7f4bb4107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.546+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 shutdown_connections 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:01.546+0000 7f4bbc000640 1 -- 172.21.15.114:0/978185027 wait complete. 2024-09-19T06:41:02.318 INFO:teuthology.orchestra.run.smithi114.stdout:Restarting the monitor... 2024-09-19T06:41:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:02 smithi114 systemd[1]: Stopping Ceph mon.a for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:41:03.049 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a[30453]: 2024-09-19T06:41:02.667+0000 7f943e000640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-19T06:41:03.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a[30453]: 2024-09-19T06:41:02.668+0000 7f943e000640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-19T06:41:03.050 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:02 smithi114 podman[30698]: 2024-09-19 06:41:02.771435792 +0000 UTC m=+0.240647848 container died 24fb34e7dad2bda204d5282fd091ee99231cab3cdb42b31c9711e275e56b60d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-19T06:41:03.341 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 podman[30698]: 2024-09-19 06:41:03.156115714 +0000 UTC m=+0.625327740 container cleanup 24fb34e7dad2bda204d5282fd091ee99231cab3cdb42b31c9711e275e56b60d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True) 2024-09-19T06:41:03.341 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 bash[30698]: ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a 2024-09-19T06:41:03.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 podman[30709]: 2024-09-19 06:41:03.340160368 +0000 UTC m=+0.557663364 container remove 24fb34e7dad2bda204d5282fd091ee99231cab3cdb42b31c9711e275e56b60d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T06:41:03.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 systemd[1]: ceph-fface640-7651-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-19T06:41:03.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 systemd[1]: Stopped Ceph mon.a for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:41:03.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 systemd[1]: ceph-fface640-7651-11ef-bceb-c7b262605968@mon.a.service: Consumed 1.204s CPU time. 2024-09-19T06:41:03.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 systemd[1]: Starting Ceph mon.a for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:41:04.159 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 podman[30802]: 2024-09-19 06:41:03.806281054 +0000 UTC m=+0.026965919 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:41:04.160 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:03 smithi114 podman[30802]: 2024-09-19 06:41:03.908323993 +0000 UTC m=+0.129008857 container create a7a113530492ef9a9db60d145cfc7a690d5d1293a173ae822da778f8cf0786e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-19T06:41:04.160 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 podman[30802]: 2024-09-19 06:41:04.118838417 +0000 UTC m=+0.339523282 container init a7a113530492ef9a9db60d145cfc7a690d5d1293a173ae822da778f8cf0786e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-19T06:41:04.160 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 podman[30802]: 2024-09-19 06:41:04.125233049 +0000 UTC m=+0.345917923 container start a7a113530492ef9a9db60d145cfc7a690d5d1293a173ae822da778f8cf0786e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T06:41:04.220 INFO:teuthology.orchestra.run.smithi114.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-19T06:41:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: set uid:gid to 167:167 (ceph:ceph) 2024-09-19T06:41:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable), process ceph-mon, pid 2 2024-09-19T06:41:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: pidfile_write: ignore empty --pid-file 2024-09-19T06:41:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: load: jerasure load: lrc 2024-09-19T06:41:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: RocksDB version: 7.9.2 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Git sha 0 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Compile date 2024-09-18 16:29:48 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: DB SUMMARY 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: DB Session ID: G3OJK883SEZQH7GWQIO1 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: CURRENT file: CURRENT 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: IDENTITY file: IDENTITY 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105390 ; 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.error_if_exists: 0 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.create_if_missing: 0 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.paranoid_checks: 1 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-19T06:41:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.env: 0x55ef30924940 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.fs: PosixFileSystem 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.info_log: 0x55ef672d36a0 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_file_opening_threads: 16 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.statistics: (nil) 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.use_fsync: 0 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_log_file_size: 0 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.keep_log_file_num: 1000 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.recycle_log_file_num: 0 2024-09-19T06:41:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.allow_fallocate: 1 2024-09-19T06:41:04.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.allow_mmap_reads: 0 2024-09-19T06:41:04.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.allow_mmap_writes: 0 2024-09-19T06:41:04.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.use_direct_reads: 0 2024-09-19T06:41:04.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-19T06:41:04.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.create_missing_column_families: 0 2024-09-19T06:41:04.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.db_log_dir: 2024-09-19T06:41:04.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.wal_dir: 2024-09-19T06:41:04.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-19T06:41:04.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-19T06:41:04.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.advise_random_on_open: 1 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.db_write_buffer_size: 0 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.write_buffer_manager: 0x55ef672e2dc0 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.rate_limiter: (nil) 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.wal_recovery_mode: 2 2024-09-19T06:41:04.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.enable_thread_tracking: 0 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.enable_pipelined_write: 0 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.unordered_write: 0 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.row_cache: None 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.wal_filter: None 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.allow_ingest_behind: 0 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.two_write_queues: 0 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.manual_wal_flush: 0 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.wal_compression: 0 2024-09-19T06:41:04.423 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.atomic_flush: 0 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.log_readahead_size: 0 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.best_efforts_recovery: 0 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.allow_data_in_errors: 0 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.db_host_id: __hostname__ 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_background_jobs: 2 2024-09-19T06:41:04.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_background_compactions: -1 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_subcompactions: 1 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_total_wal_size: 0 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_open_files: -1 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bytes_per_sync: 0 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-19T06:41:04.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_readahead_size: 0 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_background_flushes: -1 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Compression algorithms supported: 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: kZSTD supported: 0 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: kXpressCompression supported: 0 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: kBZip2Compression supported: 0 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: kLZ4Compression supported: 1 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: kZlibCompression supported: 1 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: kLZ4HCCompression supported: 1 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: kSnappyCompression supported: 1 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-19T06:41:04.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.merge_operator: 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_filter: None 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_filter_factory: None 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.sst_partitioner_factory: None 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ef672d3300) 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-09-19T06:41:04.427 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_type: 0 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_index_type: 0 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_shortening: 1 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: checksum: 4 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: no_block_cache: 0 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache: 0x55ef672c71f0 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_options: 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: capacity : 536870912 2024-09-19T06:41:04.428 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_shard_bits : 4 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: strict_capacity_limit : 0 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_compressed: (nil) 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: persistent_cache: (nil) 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size: 4096 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size_deviation: 10 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_restart_interval: 16 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_block_restart_interval: 1 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: metadata_block_size: 4096 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: partition_filters: 0 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: use_delta_encoding: 1 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: filter_policy: bloomfilter 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: whole_key_filtering: 1 2024-09-19T06:41:04.429 INFO:journalctl@ceph.mon.a.smithi114.stdout: verify_compression: 0 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout: format_version: 5 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_index_compression: 1 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_align: 0 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_auto_readahead_size: 262144 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout: prepopulate_block_cache: 0 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout: initial_auto_readahead_size: 8192 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.write_buffer_size: 33554432 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_write_buffer_number: 2 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression: NoCompression 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression: Disabled 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.prefix_extractor: nullptr 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-19T06:41:04.430 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.num_levels: 7 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-19T06:41:04.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.level: 32767 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.strategy: 0 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.enabled: false 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.target_file_size_base: 67108864 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-19T06:41:04.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.arena_block_size: 1048576 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.disable_auto_compactions: 0 2024-09-19T06:41:04.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.inplace_update_support: 0 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-19T06:41:04.434 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.bloom_locality: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.max_successive_merges: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.paranoid_file_checks: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.force_consistency_checks: 1 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.report_bg_io_stats: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.ttl: 2592000 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.enable_blob_files: false 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.min_blob_size: 0 2024-09-19T06:41:04.435 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.blob_file_size: 268435456 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.blob_file_starting_level: 0 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 31ea2696-5af8-4d5f-92d9-964d5e12f6f1 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726728064167556, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726728064169076, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100463, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98347, "index_size": 308, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14312, "raw_average_key_size": 49, "raw_value_size": 90593, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726728064, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "31ea2696-5af8-4d5f-92d9-964d5e12f6f1", "db_session_id": "G3OJK883SEZQH7GWQIO1", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-19T06:41:04.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726728064169283, "job": 1, "event": "recovery_finished"} 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ef672f6e00 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: DB pointer 0x55ef673f2000 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** DB Stats ** 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-19T06:41:04.437 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: L0 2/0 99.97 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: Sum 2/0 99.97 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 110.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-19T06:41:04.438 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative compaction: 0.00 GB write, 16.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval compaction: 0.00 GB write, 16.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache BinnedLRUCache@0x55ef672c71f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-19T06:41:04.439 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: starting mon.a rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a@-1(???) e1 preinit fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a@-1(???).mds e1 new map 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a@-1(???).mds e1 print_map 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout: e1 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout: btime 2024-09-19T06:40:58:566495+0000 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout: legacy client fscid: -1 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout: No filesystems configured 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T06:41:04.440 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 bash[30802]: a7a113530492ef9a9db60d145cfc7a690d5d1293a173ae822da778f8cf0786e4 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 systemd[1]: Started Ceph mon.a for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: monmap epoch 1 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: last_changed 2024-09-19T06:40:54.002091+0000 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: min_mon_release 19 (squid) 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: election_strategy: 1 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: fsmap 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: osdmap e1: 0 total, 0 up, 0 in 2024-09-19T06:41:04.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:04 smithi114 ceph-mon[30820]: mgrmap e1: no daemons active 2024-09-19T06:41:05.444 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.714+0000 7efcce600640 1 Processor -- start 2024-09-19T06:41:05.444 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.714+0000 7efcce600640 1 -- start start 2024-09-19T06:41:05.444 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.715+0000 7efcce600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc8081a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:05.444 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.715+0000 7efcce600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efcc8082010 con 0x7efcc8081640 2024-09-19T06:41:05.444 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.715+0000 7efcc7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc8081a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:05.444 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.715+0000 7efcc7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc8081a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50694/0 (socket says 172.21.15.114:50694) 2024-09-19T06:41:05.444 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.715+0000 7efcc7400640 1 -- 172.21.15.114:0/519407949 learned_addr learned my addr 172.21.15.114:0/519407949 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.716+0000 7efcc7400640 1 -- 172.21.15.114:0/519407949 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc810ee10 con 0x7efcc8081640 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.718+0000 7efcc7400640 1 --2- 172.21.15.114:0/519407949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc8081a40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009820 tx=0x7efcb802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fced766ea042deaa server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.718+0000 7efcc6a00640 1 -- 172.21.15.114:0/519407949 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcb8037070 con 0x7efcc8081640 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcc6a00640 1 -- 172.21.15.114:0/519407949 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efcb8032900 con 0x7efcc8081640 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcc6a00640 1 -- 172.21.15.114:0/519407949 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcb8030850 con 0x7efcc8081640 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcce600640 1 -- 172.21.15.114:0/519407949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 msgr2=0x7efcc8081a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcce600640 1 --2- 172.21.15.114:0/519407949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc8081a40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009820 tx=0x7efcb802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcce600640 1 -- 172.21.15.114:0/519407949 shutdown_connections 2024-09-19T06:41:05.445 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcce600640 1 --2- 172.21.15.114:0/519407949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc8081a40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcce600640 1 -- 172.21.15.114:0/519407949 >> 172.21.15.114:0/519407949 conn(0x7efcc807cdd0 msgr2=0x7efcc807f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcce600640 1 -- 172.21.15.114:0/519407949 shutdown_connections 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.719+0000 7efcce600640 1 -- 172.21.15.114:0/519407949 wait complete. 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.720+0000 7efcce600640 1 Processor -- start 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.720+0000 7efcce600640 1 -- start start 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.721+0000 7efcce600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc81a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.721+0000 7efcce600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efcc81121c0 con 0x7efcc8081640 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.721+0000 7efcc7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc81a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.721+0000 7efcc7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc81a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50696/0 (socket says 172.21.15.114:50696) 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.721+0000 7efcc7400640 1 -- 172.21.15.114:0/2407703815 learned_addr learned my addr 172.21.15.114:0/2407703815 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.721+0000 7efcc7400640 1 -- 172.21.15.114:0/2407703815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc81a21a0 con 0x7efcc8081640 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.722+0000 7efcc7400640 1 --2- 172.21.15.114:0/2407703815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc81a1c60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009940 tx=0x7efcb8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.722+0000 7efcc4c00640 1 -- 172.21.15.114:0/2407703815 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcb8042070 con 0x7efcc8081640 2024-09-19T06:41:05.446 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.722+0000 7efcc4c00640 1 -- 172.21.15.114:0/2407703815 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efcb80308c0 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.722+0000 7efcc4c00640 1 -- 172.21.15.114:0/2407703815 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcb8037040 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.722+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc81a2430 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.722+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc81a5120 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.722+0000 7efcc4c00640 1 -- 172.21.15.114:0/2407703815 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efcb8045070 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.722+0000 7efcc4c00640 1 -- 172.21.15.114:0/2407703815 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efcb8047b70 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.723+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc8112340 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.726+0000 7efcc4c00640 1 -- 172.21.15.114:0/2407703815 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7efcb80327b0 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.785+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7efcc8081ac0 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.787+0000 7efcc4c00640 1 -- 172.21.15.114:0/2407703815 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7efcb803c760 con 0x7efcc8081640 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.789+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 msgr2=0x7efcc81a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.789+0000 7efcce600640 1 --2- 172.21.15.114:0/2407703815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc81a1c60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efcb8009940 tx=0x7efcb8032ce0 comp rx=0 tx=0).stop 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.789+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 shutdown_connections 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.789+0000 7efcce600640 1 --2- 172.21.15.114:0/2407703815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efcc8081640 0x7efcc81a1c60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:05.447 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.789+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 >> 172.21.15.114:0/2407703815 conn(0x7efcc807cdd0 msgr2=0x7efcc807d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:05.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.789+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 shutdown_connections 2024-09-19T06:41:05.448 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:04.789+0000 7efcce600640 1 -- 172.21.15.114:0/2407703815 wait complete. 2024-09-19T06:41:05.448 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-19T06:41:05.475 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-19T06:41:05.475 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mgr... 2024-09-19T06:41:05.476 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-19T06:41:05.476 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-19T06:41:05.824 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fface640-7651-11ef-bceb-c7b262605968@mgr.x 2024-09-19T06:41:05.824 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fface640-7651-11ef-bceb-c7b262605968@mgr.x.service: Unit ceph-fface640-7651-11ef-bceb-c7b262605968@mgr.x.service not loaded. 2024-09-19T06:41:06.057 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fface640-7651-11ef-bceb-c7b262605968.target.wants/ceph-fface640-7651-11ef-bceb-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-fface640-7651-11ef-bceb-c7b262605968@.service. 2024-09-19T06:41:06.078 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:05 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2407703815' entity='client.admin' 2024-09-19T06:41:06.413 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:06 smithi114 systemd[1]: Starting Ceph mgr.x for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:41:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:06 smithi114 podman[31029]: 2024-09-19 06:41:06.329407753 +0000 UTC m=+0.028145907 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 2024-09-19T06:41:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:06 smithi114 podman[31029]: 2024-09-19 06:41:06.43570653 +0000 UTC m=+0.134444663 container create f9558e3ea80223d48438d79f91de74da7e64298d5a68ad7a0d72ad4aeee237be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-19T06:41:06.863 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-19T06:41:06.863 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-19T06:41:06.879 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-19T06:41:06.879 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-19T06:41:06.879 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-19T06:41:06.879 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-09-19T06:41:06.898 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-19T06:41:06.898 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-19T06:41:06.914 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-19T06:41:06.914 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-19T06:41:06.915 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-19T06:41:06.915 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-19T06:41:06.915 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr to start... 2024-09-19T06:41:06.915 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr... 2024-09-19T06:41:07.018 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:06 smithi114 podman[31029]: 2024-09-19 06:41:06.763318839 +0000 UTC m=+0.462056992 container init f9558e3ea80223d48438d79f91de74da7e64298d5a68ad7a0d72ad4aeee237be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-2f599a4) 2024-09-19T06:41:07.018 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:06 smithi114 podman[31029]: 2024-09-19 06:41:06.76959701 +0000 UTC m=+0.468335143 container start f9558e3ea80223d48438d79f91de74da7e64298d5a68ad7a0d72ad4aeee237be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-19T06:41:07.018 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:06 smithi114 bash[31029]: f9558e3ea80223d48438d79f91de74da7e64298d5a68ad7a0d72ad4aeee237be 2024-09-19T06:41:07.018 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:06 smithi114 systemd[1]: Started Ceph mgr.x for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:41:07.018 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:06.941+0000 7f10f1fd8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T06:41:07.018 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:07.015+0000 7f10f1fd8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T06:41:07.302 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:07.215+0000 7f10f1fd8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T06:41:07.302 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:07.301+0000 7f10f1fd8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T06:41:07.559 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:07.484+0000 7f10f1fd8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:41:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:07 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3526629908' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-19T06:41:07.946 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:07.668+0000 7f10f1fd8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T06:41:08.229 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:08.045+0000 7f10f1fd8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T06:41:08.229 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:08.116+0000 7f10f1fd8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T06:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-19T06:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-19T06:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-19T06:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-19T06:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-19T06:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-19T06:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-19T06:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "btime": "2024-09-19T06:40:58:566495+0000", 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-19T06:40:58.567302+0000", 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T06:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.459+0000 7f978ba00640 1 Processor -- start 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.459+0000 7f978ba00640 1 -- start start 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.459+0000 7f978ba00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f9784109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.459+0000 7f978ba00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f978410a290 con 0x7f97841098c0 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.460+0000 7f9789000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f9784109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.460+0000 7f9789000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f9784109cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50930/0 (socket says 172.21.15.114:50930) 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.460+0000 7f9789000640 1 -- 172.21.15.114:0/3556120235 learned_addr learned my addr 172.21.15.114:0/3556120235 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.460+0000 7f9789000640 1 -- 172.21.15.114:0/3556120235 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978410aa70 con 0x7f97841098c0 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.461+0000 7f9789000640 1 --2- 172.21.15.114:0/3556120235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f9784109cc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9778009820 tx=0x7f977802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd0c25ca5d14c5d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.462+0000 7f9783400640 1 -- 172.21.15.114:0/3556120235 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9778037070 con 0x7f97841098c0 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.462+0000 7f9783400640 1 -- 172.21.15.114:0/3556120235 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9778032900 con 0x7f97841098c0 2024-09-19T06:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.462+0000 7f9783400640 1 -- 172.21.15.114:0/3556120235 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9778030850 con 0x7f97841098c0 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.462+0000 7f978ba00640 1 -- 172.21.15.114:0/3556120235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 msgr2=0x7f9784109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.463+0000 7f978ba00640 1 --2- 172.21.15.114:0/3556120235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f9784109cc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9778009820 tx=0x7f977802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.463+0000 7f978ba00640 1 -- 172.21.15.114:0/3556120235 shutdown_connections 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.463+0000 7f978ba00640 1 --2- 172.21.15.114:0/3556120235 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f9784109cc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.463+0000 7f978ba00640 1 -- 172.21.15.114:0/3556120235 >> 172.21.15.114:0/3556120235 conn(0x7f978407cf40 msgr2=0x7f978407d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.463+0000 7f978ba00640 1 -- 172.21.15.114:0/3556120235 shutdown_connections 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.463+0000 7f978ba00640 1 -- 172.21.15.114:0/3556120235 wait complete. 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.464+0000 7f978ba00640 1 Processor -- start 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.464+0000 7f978ba00640 1 -- start start 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.464+0000 7f978ba00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f978419d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.464+0000 7f978ba00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f978410de20 con 0x7f97841098c0 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.465+0000 7f9789000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f978419d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.465+0000 7f9789000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f978419d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50936/0 (socket says 172.21.15.114:50936) 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.465+0000 7f9789000640 1 -- 172.21.15.114:0/3526629908 learned_addr learned my addr 172.21.15.114:0/3526629908 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.465+0000 7f9789000640 1 -- 172.21.15.114:0/3526629908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978419ddd0 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.465+0000 7f9789000640 1 --2- 172.21.15.114:0/3526629908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f978419d890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f97780097f0 tx=0x7f9778030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.466+0000 7f9781600640 1 -- 172.21.15.114:0/3526629908 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9778037040 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.466+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978419e060 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.466+0000 7f9781600640 1 -- 172.21.15.114:0/3526629908 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9778040070 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.466+0000 7f9781600640 1 -- 172.21.15.114:0/3526629908 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9778045590 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.466+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97841a0d50 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.466+0000 7f9781600640 1 -- 172.21.15.114:0/3526629908 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9778045730 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.466+0000 7f9781600640 1 -- 172.21.15.114:0/3526629908 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f97780449b0 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.467+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9784109d40 con 0x7f97841098c0 2024-09-19T06:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.472+0000 7f9781600640 1 -- 172.21.15.114:0/3526629908 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f97780443c0 con 0x7f97841098c0 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.537+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f97841a0f20 con 0x7f97841098c0 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.537+0000 7f9781600640 1 -- 172.21.15.114:0/3526629908 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f977803b430 con 0x7f97841098c0 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.538+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 msgr2=0x7f978419d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.538+0000 7f978ba00640 1 --2- 172.21.15.114:0/3526629908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f978419d890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f97780097f0 tx=0x7f9778030d80 comp rx=0 tx=0).stop 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.538+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 shutdown_connections 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.538+0000 7f978ba00640 1 --2- 172.21.15.114:0/3526629908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97841098c0 0x7f978419d890 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.539+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 >> 172.21.15.114:0/3526629908 conn(0x7f978407cf40 msgr2=0x7f9784107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.539+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 shutdown_connections 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:07.539+0000 7f978ba00640 1 -- 172.21.15.114:0/3526629908 wait complete. 2024-09-19T06:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (1/15)... 2024-09-19T06:41:08.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:08.330+0000 7f10f1fd8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T06:41:09.493 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:09.414+0000 7f10f1fd8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T06:41:09.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:09.492+0000 7f10f1fd8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T06:41:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:10.415+0000 7f10f1fd8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T06:41:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:10.553+0000 7f10f1fd8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T06:41:10.980 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:10 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1341833104' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-19T06:41:10.981 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:10.717+0000 7f10f1fd8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:41:10.981 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:10.794+0000 7f10f1fd8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T06:41:10.981 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:10.979+0000 7f10f1fd8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T06:41:11.327 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:11.063+0000 7f10f1fd8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T06:41:11.327 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:11.137+0000 7f10f1fd8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T06:41:11.327 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:11.325+0000 7f10f1fd8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T06:41:11.582 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:11.407+0000 7f10f1fd8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T06:41:11.582 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:11.484+0000 7f10f1fd8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T06:41:11.582 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:11.580+0000 7f10f1fd8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:11.757 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:11.758 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-19T06:41:11.759 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-19T06:41:11.769 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "btime": "2024-09-19T06:40:58:566495+0000", 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:11.770 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-19T06:40:58.567302+0000", 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T06:41:11.771 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.779+0000 7fdfa2200640 1 Processor -- start 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.779+0000 7fdfa2200640 1 -- start start 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.780+0000 7fdfa2200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.780+0000 7fdfa2200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c07c800 con 0x7fdf9c07de60 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.780+0000 7fdfa0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.780+0000 7fdfa0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50946/0 (socket says 172.21.15.114:50946) 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.780+0000 7fdfa0e00640 1 -- 172.21.15.114:0/4022708670 learned_addr learned my addr 172.21.15.114:0/4022708670 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.780+0000 7fdfa0e00640 1 -- 172.21.15.114:0/4022708670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c07c980 con 0x7fdf9c07de60 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.781+0000 7fdfa0e00640 1 --2- 172.21.15.114:0/4022708670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c07c2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009820 tx=0x7fdf9002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6a24ac7d76b363 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.782+0000 7fdf9b400640 1 -- 172.21.15.114:0/4022708670 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf90037070 con 0x7fdf9c07de60 2024-09-19T06:41:11.772 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.782+0000 7fdf9b400640 1 -- 172.21.15.114:0/4022708670 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdf90032900 con 0x7fdf9c07de60 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.782+0000 7fdf9b400640 1 -- 172.21.15.114:0/4022708670 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf90030850 con 0x7fdf9c07de60 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.782+0000 7fdfa2200640 1 -- 172.21.15.114:0/4022708670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 msgr2=0x7fdf9c07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.782+0000 7fdfa2200640 1 --2- 172.21.15.114:0/4022708670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c07c2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009820 tx=0x7fdf9002a270 comp rx=0 tx=0).stop 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.782+0000 7fdfa2200640 1 -- 172.21.15.114:0/4022708670 shutdown_connections 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.782+0000 7fdfa2200640 1 --2- 172.21.15.114:0/4022708670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c07c2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.782+0000 7fdfa2200640 1 -- 172.21.15.114:0/4022708670 >> 172.21.15.114:0/4022708670 conn(0x7fdf9c102c10 msgr2=0x7fdf9c105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.783+0000 7fdfa2200640 1 -- 172.21.15.114:0/4022708670 shutdown_connections 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.783+0000 7fdfa2200640 1 -- 172.21.15.114:0/4022708670 wait complete. 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.783+0000 7fdfa2200640 1 Processor -- start 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.784+0000 7fdfa2200640 1 -- start start 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.784+0000 7fdfa2200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c10a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.784+0000 7fdfa2200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c10da90 con 0x7fdf9c07de60 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.784+0000 7fdfa0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c10a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:11.773 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.784+0000 7fdfa0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c10a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50952/0 (socket says 172.21.15.114:50952) 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.784+0000 7fdfa0e00640 1 -- 172.21.15.114:0/1341833104 learned_addr learned my addr 172.21.15.114:0/1341833104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.785+0000 7fdfa0e00640 1 -- 172.21.15.114:0/1341833104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c10ac50 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.785+0000 7fdfa0e00640 1 --2- 172.21.15.114:0/1341833104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c10a710 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdf900097f0 tx=0x7fdf90030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.785+0000 7fdf99600640 1 -- 172.21.15.114:0/1341833104 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf90042070 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.785+0000 7fdf99600640 1 -- 172.21.15.114:0/1341833104 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdf90032ca0 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.785+0000 7fdf99600640 1 -- 172.21.15.114:0/1341833104 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf90037040 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.785+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9c10aee0 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.785+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9c1074a0 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.786+0000 7fdf99600640 1 -- 172.21.15.114:0/1341833104 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf90045070 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.786+0000 7fdf99600640 1 -- 172.21.15.114:0/1341833104 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdf90047a90 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.786+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf9c10b0b0 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.789+0000 7fdf99600640 1 -- 172.21.15.114:0/1341833104 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fdf9c10b0b0 con 0x7fdf9c07de60 2024-09-19T06:41:11.774 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.853+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdf9c107b10 con 0x7fdf9c07de60 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.853+0000 7fdf99600640 1 -- 172.21.15.114:0/1341833104 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7fdf9003db10 con 0x7fdf9c07de60 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.854+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 msgr2=0x7fdf9c10a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.854+0000 7fdfa2200640 1 --2- 172.21.15.114:0/1341833104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c10a710 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdf900097f0 tx=0x7fdf90030ec0 comp rx=0 tx=0).stop 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.854+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 shutdown_connections 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.854+0000 7fdfa2200640 1 --2- 172.21.15.114:0/1341833104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf9c07de60 0x7fdf9c10a710 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.854+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 >> 172.21.15.114:0/1341833104 conn(0x7fdf9c102c10 msgr2=0x7fdf9c1037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.854+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 shutdown_connections 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:10.855+0000 7fdfa2200640 1 -- 172.21.15.114:0/1341833104 wait complete. 2024-09-19T06:41:11.775 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (2/15)... 2024-09-19T06:41:11.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:11.653+0000 7f10f1fd8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T06:41:11.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:11.721+0000 7f10f1fd8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T06:41:12.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:12.620+0000 7f10f1fd8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T06:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: Activating manager daemon x 2024-09-19T06:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: mgrmap e2: x(active, starting, since 0.00410382s) 2024-09-19T06:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T06:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T06:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T06:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T06:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: Manager daemon x is now available 2024-09-19T06:41:12.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T06:41:12.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T06:41:12.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' 2024-09-19T06:41:12.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' 2024-09-19T06:41:12.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:12 smithi114 ceph-mon[30820]: from='mgr.14100 172.21.15.114:0/2867885445' entity='mgr.x' 2024-09-19T06:41:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:14 smithi114 ceph-mon[30820]: mgrmap e3: x(active, since 1.00892s) 2024-09-19T06:41:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:14 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2954300532' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-19T06:41:15.375 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:15.375 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-19T06:41:15.375 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-19T06:41:15.376 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-09-19T06:41:15.377 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:15.377 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-09-19T06:41:15.377 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-19T06:41:15.377 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:15.377 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-19T06:41:15.377 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-19T06:41:15.377 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:15.377 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-19T06:41:15.378 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "btime": "2024-09-19T06:40:58:566495+0000", 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-19T06:41:15.379 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T06:41:15.380 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-09-19T06:40:58.567302+0000", 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.244+0000 7f94ef200640 1 Processor -- start 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.244+0000 7f94ef200640 1 -- start start 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.244+0000 7f94ef200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e8109d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:15.381 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.244+0000 7f94ef200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f94e810a340 con 0x7f94e8109970 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.245+0000 7f94ede00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e8109d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.245+0000 7f94ede00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e8109d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51038/0 (socket says 172.21.15.114:51038) 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.245+0000 7f94ede00640 1 -- 172.21.15.114:0/3275365460 learned_addr learned my addr 172.21.15.114:0/3275365460 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.245+0000 7f94ede00640 1 -- 172.21.15.114:0/3275365460 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e810a840 con 0x7f94e8109970 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.246+0000 7f94ede00640 1 --2- 172.21.15.114:0/3275365460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e8109d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009820 tx=0x7f94dc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1638d89b843dbd server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.246+0000 7f94eca00640 1 -- 172.21.15.114:0/3275365460 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94dc037070 con 0x7f94e8109970 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.246+0000 7f94eca00640 1 -- 172.21.15.114:0/3275365460 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f94dc032900 con 0x7f94e8109970 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.246+0000 7f94eca00640 1 -- 172.21.15.114:0/3275365460 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94dc030850 con 0x7f94e8109970 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.247+0000 7f94ef200640 1 -- 172.21.15.114:0/3275365460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 msgr2=0x7f94e8109d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.247+0000 7f94ef200640 1 --2- 172.21.15.114:0/3275365460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e8109d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009820 tx=0x7f94dc02a270 comp rx=0 tx=0).stop 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.247+0000 7f94ef200640 1 -- 172.21.15.114:0/3275365460 shutdown_connections 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.247+0000 7f94ef200640 1 --2- 172.21.15.114:0/3275365460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e8109d70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.247+0000 7f94ef200640 1 -- 172.21.15.114:0/3275365460 >> 172.21.15.114:0/3275365460 conn(0x7f94e807cca0 msgr2=0x7f94e807d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:15.382 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.247+0000 7f94ef200640 1 -- 172.21.15.114:0/3275365460 shutdown_connections 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.247+0000 7f94ef200640 1 -- 172.21.15.114:0/3275365460 wait complete. 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.248+0000 7f94ef200640 1 Processor -- start 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.248+0000 7f94ef200640 1 -- start start 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.248+0000 7f94ef200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e819d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.248+0000 7f94ef200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f94e810dbf0 con 0x7f94e8109970 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.248+0000 7f94ede00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e819d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.248+0000 7f94ede00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e819d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51042/0 (socket says 172.21.15.114:51042) 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.248+0000 7f94ede00640 1 -- 172.21.15.114:0/2954300532 learned_addr learned my addr 172.21.15.114:0/2954300532 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.249+0000 7f94ede00640 1 -- 172.21.15.114:0/2954300532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e819dc70 con 0x7f94e8109970 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.249+0000 7f94ede00640 1 --2- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e819d730 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0097f0 tx=0x7f94dc030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.249+0000 7f94e2a00640 1 -- 172.21.15.114:0/2954300532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94dc042070 con 0x7f94e8109970 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.249+0000 7f94e2a00640 1 -- 172.21.15.114:0/2954300532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f94dc032be0 con 0x7f94e8109970 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.249+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e819df00 con 0x7f94e8109970 2024-09-19T06:41:15.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.249+0000 7f94e2a00640 1 -- 172.21.15.114:0/2954300532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94dc037040 con 0x7f94e8109970 2024-09-19T06:41:15.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.249+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e81a0bf0 con 0x7f94e8109970 2024-09-19T06:41:15.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.250+0000 7f94e2a00640 1 -- 172.21.15.114:0/2954300532 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 3) ==== 46344+0+0 (secure 0 0 0) 0x7f94dc045070 con 0x7f94e8109970 2024-09-19T06:41:15.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.250+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e8109df0 con 0x7f94e8109970 2024-09-19T06:41:15.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.251+0000 7f94e2a00640 1 --2- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f94d0039050 0x7f94d003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:15.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.251+0000 7f94e2a00640 1 -- 172.21.15.114:0/2954300532 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f94dc06bf10 con 0x7f94e8109970 2024-09-19T06:41:15.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.251+0000 7f94ed400640 1 --2- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f94d0039050 0x7f94d003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:15.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.252+0000 7f94ed400640 1 --2- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f94d0039050 0x7f94d003b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f94d80098c0 tx=0x7f94d8006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:15.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.257+0000 7f94e2a00640 1 -- 172.21.15.114:0/2954300532 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f94dc03d430 con 0x7f94e8109970 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.500+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f94e81a1270 con 0x7f94e8109970 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.501+0000 7f94e2a00640 1 -- 172.21.15.114:0/2954300532 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f94dc045360 con 0x7f94e8109970 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.503+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f94d0039050 msgr2=0x7f94d003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.503+0000 7f94ef200640 1 --2- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f94d0039050 0x7f94d003b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f94d80098c0 tx=0x7f94d8006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.503+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 msgr2=0x7f94e819d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.503+0000 7f94ef200640 1 --2- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e819d730 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0097f0 tx=0x7f94dc030d80 comp rx=0 tx=0).stop 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.504+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 shutdown_connections 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.504+0000 7f94ef200640 1 --2- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f94d0039050 0x7f94d003b510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.504+0000 7f94ef200640 1 --2- 172.21.15.114:0/2954300532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94e8109970 0x7f94e819d730 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.504+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 >> 172.21.15.114:0/2954300532 conn(0x7f94e807cca0 msgr2=0x7f94e8107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.504+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 shutdown_connections 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:14.504+0000 7f94ef200640 1 -- 172.21.15.114:0/2954300532 wait complete. 2024-09-19T06:41:15.385 INFO:teuthology.orchestra.run.smithi114.stdout:mgr is available 2024-09-19T06:41:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:15 smithi114 ceph-mon[30820]: mgrmap e4: x(active, since 2s) 2024-09-19T06:41:16.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:16.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-09-19T06:41:16.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.840+0000 7f5bb6c00640 1 Processor -- start 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.840+0000 7f5bb6c00640 1 -- start start 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.841+0000 7f5bb6c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb0107950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.841+0000 7f5bb6c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0107f20 con 0x7f5bb0107550 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.841+0000 7f5bb5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb0107950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.841+0000 7f5bb5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb0107950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51052/0 (socket says 172.21.15.114:51052) 2024-09-19T06:41:16.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.841+0000 7f5bb5800640 1 -- 172.21.15.114:0/809077361 learned_addr learned my addr 172.21.15.114:0/809077361 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.841+0000 7f5bb5800640 1 -- 172.21.15.114:0/809077361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb01086b0 con 0x7f5bb0107550 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.842+0000 7f5bb5800640 1 --2- 172.21.15.114:0/809077361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb0107950 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009820 tx=0x7f5ba402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555837ec501f8e01 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bafe00640 1 -- 172.21.15.114:0/809077361 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ba4037070 con 0x7f5bb0107550 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bafe00640 1 -- 172.21.15.114:0/809077361 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5ba4032900 con 0x7f5bb0107550 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bafe00640 1 -- 172.21.15.114:0/809077361 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ba4030850 con 0x7f5bb0107550 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bb6c00640 1 -- 172.21.15.114:0/809077361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 msgr2=0x7f5bb0107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bb6c00640 1 --2- 172.21.15.114:0/809077361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb0107950 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009820 tx=0x7f5ba402a270 comp rx=0 tx=0).stop 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bb6c00640 1 -- 172.21.15.114:0/809077361 shutdown_connections 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bb6c00640 1 --2- 172.21.15.114:0/809077361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb0107950 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:16.647 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bb6c00640 1 -- 172.21.15.114:0/809077361 >> 172.21.15.114:0/809077361 conn(0x7f5bb0102d00 msgr2=0x7f5bb0105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bb6c00640 1 -- 172.21.15.114:0/809077361 shutdown_connections 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.843+0000 7f5bb6c00640 1 -- 172.21.15.114:0/809077361 wait complete. 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.844+0000 7f5bb6c00640 1 Processor -- start 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.844+0000 7f5bb6c00640 1 -- start start 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.845+0000 7f5bb6c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb007db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.845+0000 7f5bb6c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb010ba60 con 0x7f5bb0107550 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.845+0000 7f5bb5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb007db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.845+0000 7f5bb5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb007db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51060/0 (socket says 172.21.15.114:51060) 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.845+0000 7f5bb5800640 1 -- 172.21.15.114:0/1610344210 learned_addr learned my addr 172.21.15.114:0/1610344210 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.845+0000 7f5bb5800640 1 -- 172.21.15.114:0/1610344210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb007e0b0 con 0x7f5bb0107550 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.846+0000 7f5bb5800640 1 --2- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb007db70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5ba40097f0 tx=0x7f5ba4030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:16.648 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.846+0000 7f5bae000640 1 -- 172.21.15.114:0/1610344210 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ba4042070 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.846+0000 7f5bae000640 1 -- 172.21.15.114:0/1610344210 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5ba4032ca0 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.846+0000 7f5bae000640 1 -- 172.21.15.114:0/1610344210 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ba4037040 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.846+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb007c2a0 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.846+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb007c7e0 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.847+0000 7f5bae000640 1 -- 172.21.15.114:0/1610344210 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f5ba4045070 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.847+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb01079d0 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.847+0000 7f5bae000640 1 --2- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f5b980391b0 0x7f5b9803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.848+0000 7f5bae000640 1 -- 172.21.15.114:0/1610344210 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5ba406c140 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.849+0000 7f5bb4e00640 1 --2- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f5b980391b0 0x7f5b9803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.849+0000 7f5bb4e00640 1 --2- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f5b980391b0 0x7f5b9803b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0098c0 tx=0x7f5b9c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:15.853+0000 7f5bae000640 1 -- 172.21.15.114:0/1610344210 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5ba404b170 con 0x7f5bb0107550 2024-09-19T06:41:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.028+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f5bb007d020 con 0x7f5bb0107550 2024-09-19T06:41:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.030+0000 7f5bae000640 1 -- 172.21.15.114:0/1610344210 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+431 (secure 0 0 0) 0x7f5ba406bb60 con 0x7f5bb0107550 2024-09-19T06:41:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f5b980391b0 msgr2=0x7f5b9803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 --2- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f5b980391b0 0x7f5b9803b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0098c0 tx=0x7f5b9c006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 msgr2=0x7f5bb007db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 --2- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb007db70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5ba40097f0 tx=0x7f5ba4030ec0 comp rx=0 tx=0).stop 2024-09-19T06:41:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 shutdown_connections 2024-09-19T06:41:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 --2- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f5b980391b0 0x7f5b9803b670 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:16.650 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 --2- 172.21.15.114:0/1610344210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bb0107550 0x7f5bb007db70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:16.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 >> 172.21.15.114:0/1610344210 conn(0x7f5bb0102d00 msgr2=0x7f5bb0103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:16.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 shutdown_connections 2024-09-19T06:41:16.651 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:16.032+0000 7f5bb6c00640 1 -- 172.21.15.114:0/1610344210 wait complete. 2024-09-19T06:41:16.651 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling cephadm module... 2024-09-19T06:41:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:16 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1610344210' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-19T06:41:17.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:17 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3448944255' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-19T06:41:17.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: ignoring --setuser ceph since I am not root 2024-09-19T06:41:17.994 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: ignoring --setgroup ceph since I am not root 2024-09-19T06:41:17.995 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:17.853+0000 7f48185da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T06:41:17.995 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:17.925+0000 7f48185da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T06:41:18.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:18.110+0000 7f48185da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T06:41:18.296 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:18.190+0000 7f48185da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.198+0000 7f6e07e00640 1 Processor -- start 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.198+0000 7f6e07e00640 1 -- start start 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.198+0000 7f6e07e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.198+0000 7f6e07e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0807c8b0 con 0x7f6e0807de80 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.198+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.198+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57790/0 (socket says 172.21.15.114:57790) 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.198+0000 7f6e06a00640 1 -- 172.21.15.114:0/2617834557 learned_addr learned my addr 172.21.15.114:0/2617834557 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.199+0000 7f6e06a00640 1 -- 172.21.15.114:0/2617834557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0807ca30 con 0x7f6e0807de80 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.200+0000 7f6e06a00640 1 --2- 172.21.15.114:0/2617834557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0807c2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009820 tx=0x7f6df802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a1a875a1f076a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.200+0000 7f6e05600640 1 -- 172.21.15.114:0/2617834557 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6df8037070 con 0x7f6e0807de80 2024-09-19T06:41:18.309 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.200+0000 7f6e05600640 1 -- 172.21.15.114:0/2617834557 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6df8032900 con 0x7f6e0807de80 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.200+0000 7f6e05600640 1 -- 172.21.15.114:0/2617834557 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6df8030850 con 0x7f6e0807de80 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.200+0000 7f6e07e00640 1 -- 172.21.15.114:0/2617834557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 msgr2=0x7f6e0807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.200+0000 7f6e07e00640 1 --2- 172.21.15.114:0/2617834557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0807c2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009820 tx=0x7f6df802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.201+0000 7f6e07e00640 1 -- 172.21.15.114:0/2617834557 shutdown_connections 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.201+0000 7f6e07e00640 1 --2- 172.21.15.114:0/2617834557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0807c2e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.201+0000 7f6e07e00640 1 -- 172.21.15.114:0/2617834557 >> 172.21.15.114:0/2617834557 conn(0x7f6e08102c10 msgr2=0x7f6e08105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.201+0000 7f6e07e00640 1 -- 172.21.15.114:0/2617834557 shutdown_connections 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.201+0000 7f6e07e00640 1 -- 172.21.15.114:0/2617834557 wait complete. 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.202+0000 7f6e07e00640 1 Processor -- start 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.202+0000 7f6e07e00640 1 -- start start 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.202+0000 7f6e07e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0819d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.202+0000 7f6e07e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0810da70 con 0x7f6e0807de80 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.202+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0819d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:18.310 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.202+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0819d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57794/0 (socket says 172.21.15.114:57794) 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.202+0000 7f6e06a00640 1 -- 172.21.15.114:0/3448944255 learned_addr learned my addr 172.21.15.114:0/3448944255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.203+0000 7f6e06a00640 1 -- 172.21.15.114:0/3448944255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e0819dbf0 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.203+0000 7f6e06a00640 1 --2- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0819d6b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6df80097f0 tx=0x7f6df8030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.203+0000 7f6dff400640 1 -- 172.21.15.114:0/3448944255 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6df8042070 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.203+0000 7f6dff400640 1 -- 172.21.15.114:0/3448944255 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6df8032ca0 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.203+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e0819de80 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.203+0000 7f6dff400640 1 -- 172.21.15.114:0/3448944255 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6df8037040 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.203+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e081a0b70 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.204+0000 7f6dff400640 1 -- 172.21.15.114:0/3448944255 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f6df8045070 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.204+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e0807c360 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.205+0000 7f6dff400640 1 --2- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f6dec0391b0 0x7f6dec03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.205+0000 7f6dff400640 1 -- 172.21.15.114:0/3448944255 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6df806c150 con 0x7f6e0807de80 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.205+0000 7f6e06000640 1 --2- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f6dec0391b0 0x7f6dec03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:18.311 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.206+0000 7f6e06000640 1 --2- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f6dec0391b0 0x7f6dec03b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6df40098c0 tx=0x7f6df4006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.210+0000 7f6dff400640 1 -- 172.21.15.114:0/3448944255 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f6df8045360 con 0x7f6e0807de80 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.429+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f6e081a11f0 con 0x7f6e0807de80 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.636+0000 7f6dff400640 1 -- 172.21.15.114:0/3448944255 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f6df8043260 con 0x7f6e0807de80 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.638+0000 7f6dff400640 1 -- 172.21.15.114:0/3448944255 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f6df803d430 con 0x7f6e0807de80 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f6dec0391b0 msgr2=0x7f6dec03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 --2- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f6dec0391b0 0x7f6dec03b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6df40098c0 tx=0x7f6df4006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 msgr2=0x7f6e0819d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 --2- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0819d6b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6df80097f0 tx=0x7f6df8030ec0 comp rx=0 tx=0).stop 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 shutdown_connections 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 --2- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7f6dec0391b0 0x7f6dec03b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 --2- 172.21.15.114:0/3448944255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e0807de80 0x7f6e0819d6b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:18.312 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 >> 172.21.15.114:0/3448944255 conn(0x7f6e08102c10 msgr2=0x7f6e08103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:18.313 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 shutdown_connections 2024-09-19T06:41:18.313 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:17.640+0000 7f6e07e00640 1 -- 172.21.15.114:0/3448944255 wait complete. 2024-09-19T06:41:18.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:18.369+0000 7f48185da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:41:18.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:18.534+0000 7f48185da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T06:41:18.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:18 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3448944255' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-19T06:41:18.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:18 smithi114 ceph-mon[30820]: mgrmap e5: x(active, since 5s) 2024-09-19T06:41:19.112 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:18.913+0000 7f48185da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T06:41:19.112 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:18.987+0000 7f48185da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T06:41:19.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:19.207+0000 7f48185da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T06:41:19.878 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:19 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1802318427' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-19T06:41:19.892 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.854+0000 7fc9fb000640 1 Processor -- start 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.854+0000 7fc9fb000640 1 -- start start 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.855+0000 7fc9fb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f4109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.855+0000 7fc9fb000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f410a290 con 0x7fc9f41098c0 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.855+0000 7fc9f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f4109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.855+0000 7fc9f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f4109cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57816/0 (socket says 172.21.15.114:57816) 2024-09-19T06:41:19.893 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.855+0000 7fc9f3e00640 1 -- 172.21.15.114:0/2832084061 learned_addr learned my addr 172.21.15.114:0/2832084061 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.856+0000 7fc9f3e00640 1 -- 172.21.15.114:0/2832084061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f410aa70 con 0x7fc9f41098c0 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.857+0000 7fc9f3e00640 1 --2- 172.21.15.114:0/2832084061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f4109cc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009820 tx=0x7fc9e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9e4d05912cb5c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.857+0000 7fc9f2a00640 1 -- 172.21.15.114:0/2832084061 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9e4037070 con 0x7fc9f41098c0 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.857+0000 7fc9f2a00640 1 -- 172.21.15.114:0/2832084061 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc9e4032900 con 0x7fc9f41098c0 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.857+0000 7fc9f2a00640 1 -- 172.21.15.114:0/2832084061 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9e4030850 con 0x7fc9f41098c0 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.858+0000 7fc9fb000640 1 -- 172.21.15.114:0/2832084061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 msgr2=0x7fc9f4109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.858+0000 7fc9fb000640 1 --2- 172.21.15.114:0/2832084061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f4109cc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009820 tx=0x7fc9e402a270 comp rx=0 tx=0).stop 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.858+0000 7fc9fb000640 1 -- 172.21.15.114:0/2832084061 shutdown_connections 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.858+0000 7fc9fb000640 1 --2- 172.21.15.114:0/2832084061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f4109cc0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.858+0000 7fc9fb000640 1 -- 172.21.15.114:0/2832084061 >> 172.21.15.114:0/2832084061 conn(0x7fc9f407cf40 msgr2=0x7fc9f407d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.858+0000 7fc9fb000640 1 -- 172.21.15.114:0/2832084061 shutdown_connections 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.858+0000 7fc9fb000640 1 -- 172.21.15.114:0/2832084061 wait complete. 2024-09-19T06:41:19.894 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.859+0000 7fc9fb000640 1 Processor -- start 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.859+0000 7fc9fb000640 1 -- start start 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.859+0000 7fc9fb000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f419d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.859+0000 7fc9fb000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f410de20 con 0x7fc9f41098c0 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.859+0000 7fc9f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f419d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.859+0000 7fc9f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f419d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57826/0 (socket says 172.21.15.114:57826) 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.859+0000 7fc9f3e00640 1 -- 172.21.15.114:0/1802318427 learned_addr learned my addr 172.21.15.114:0/1802318427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.860+0000 7fc9f3e00640 1 -- 172.21.15.114:0/1802318427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f419ddd0 con 0x7fc9f41098c0 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.860+0000 7fc9f3e00640 1 --2- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f419d890 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40097f0 tx=0x7fc9e4030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.860+0000 7fc9f0c00640 1 -- 172.21.15.114:0/1802318427 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9e4037040 con 0x7fc9f41098c0 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.860+0000 7fc9f0c00640 1 -- 172.21.15.114:0/1802318427 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc9e4040070 con 0x7fc9f41098c0 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.860+0000 7fc9f0c00640 1 -- 172.21.15.114:0/1802318427 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9e403a390 con 0x7fc9f41098c0 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.860+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f419e060 con 0x7fc9f41098c0 2024-09-19T06:41:19.895 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.861+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f41a0d50 con 0x7fc9f41098c0 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.862+0000 7fc9f0c00640 1 -- 172.21.15.114:0/1802318427 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7fc9e4032900 con 0x7fc9f41098c0 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.862+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f4109d40 con 0x7fc9f41098c0 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.862+0000 7fc9f0c00640 1 --2- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fc9d8039200 0x7fc9d803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.862+0000 7fc9f0c00640 1 -- 172.21.15.114:0/1802318427 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc9e406c3e0 con 0x7fc9f41098c0 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.862+0000 7fc9f3400640 1 -- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fc9d8039200 msgr2=0x7fc9d803b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2384341327 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.862+0000 7fc9f3400640 1 --2- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fc9d8039200 0x7fc9d803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:18.867+0000 7fc9f0c00640 1 -- 172.21.15.114:0/1802318427 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc9e406c140 con 0x7fc9f41098c0 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.063+0000 7fc9f3400640 1 -- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fc9d8039200 msgr2=0x7fc9d803b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2384341327 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.063+0000 7fc9f3400640 1 --2- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fc9d8039200 0x7fc9d803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.095+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc9f41a0f20 con 0x7fc9f41098c0 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.095+0000 7fc9f0c00640 1 -- 172.21.15.114:0/1802318427 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc9f41a0f20 con 0x7fc9f41098c0 2024-09-19T06:41:19.896 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fc9d8039200 msgr2=0x7fc9d803b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 --2- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fc9d8039200 0x7fc9d803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 msgr2=0x7fc9f419d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 --2- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f419d890 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40097f0 tx=0x7fc9e4030d80 comp rx=0 tx=0).stop 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 shutdown_connections 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 --2- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fc9d8039200 0x7fc9d803b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 --2- 172.21.15.114:0/1802318427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9f41098c0 0x7fc9f419d890 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 >> 172.21.15.114:0/1802318427 conn(0x7fc9f407cf40 msgr2=0x7fc9f4107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 shutdown_connections 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:19.098+0000 7fc9fb000640 1 -- 172.21.15.114:0/1802318427 wait complete. 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-09-19T06:41:19.897 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 5... 2024-09-19T06:41:20.616 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:20.335+0000 7f48185da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T06:41:20.617 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:20.420+0000 7f48185da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T06:41:21.617 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:21.319+0000 7f48185da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T06:41:21.617 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:21.453+0000 7f48185da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T06:41:21.617 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:21.615+0000 7f48185da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:41:21.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:21.685+0000 7f48185da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T06:41:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:21.854+0000 7f48185da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T06:41:22.260 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:21.934+0000 7f48185da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T06:41:22.260 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:22.005+0000 7f48185da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T06:41:22.260 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:22.183+0000 7f48185da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T06:41:22.260 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:22.258+0000 7f48185da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T06:41:22.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:22.336+0000 7f48185da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T06:41:22.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:22.431+0000 7f48185da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T06:41:22.571 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:22.500+0000 7f48185da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T06:41:22.571 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:22.569+0000 7f48185da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T06:41:23.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:23.466+0000 7f48185da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T06:41:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: Active manager daemon x restarted 2024-09-19T06:41:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: Activating manager daemon x 2024-09-19T06:41:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: osdmap e2: 0 total, 0 up, 0 in 2024-09-19T06:41:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: mgrmap e6: x(active, starting, since 0.00373546s) 2024-09-19T06:41:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:41:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T06:41:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T06:41:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T06:41:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T06:41:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: Manager daemon x is now available 2024-09-19T06:41:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:41:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:23 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:41:24.844 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:24 smithi114 ceph-mon[30820]: Found migration_current of "None". Setting to last migration. 2024-09-19T06:41:24.845 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:24 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T06:41:24.845 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:24 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T06:41:24.845 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:24 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:24.845 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:24 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:24.845 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:24 smithi114 ceph-mon[30820]: mgrmap e7: x(active, since 1.0067s) 2024-09-19T06:41:25.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-19T06:41:25.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-19T06:41:25.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-19T06:41:25.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-19T06:41:25.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.406+0000 7fe02a400640 1 Processor -- start 2024-09-19T06:41:25.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.406+0000 7fe02a400640 1 -- start start 2024-09-19T06:41:25.171 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.406+0000 7fe02a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02407c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.406+0000 7fe02a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe02407c8b0 con 0x7fe02407de80 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.406+0000 7fe029000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02407c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.406+0000 7fe029000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02407c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57836/0 (socket says 172.21.15.114:57836) 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.406+0000 7fe029000640 1 -- 172.21.15.114:0/3858502068 learned_addr learned my addr 172.21.15.114:0/3858502068 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.407+0000 7fe029000640 1 -- 172.21.15.114:0/3858502068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02407ca30 con 0x7fe02407de80 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.408+0000 7fe029000640 1 --2- 172.21.15.114:0/3858502068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02407c2e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe018009820 tx=0x7fe01802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3e497b959d256c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.408+0000 7fe023400640 1 -- 172.21.15.114:0/3858502068 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe018037070 con 0x7fe02407de80 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.408+0000 7fe023400640 1 -- 172.21.15.114:0/3858502068 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe018032900 con 0x7fe02407de80 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.408+0000 7fe023400640 1 -- 172.21.15.114:0/3858502068 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe018030850 con 0x7fe02407de80 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.409+0000 7fe02a400640 1 -- 172.21.15.114:0/3858502068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 msgr2=0x7fe02407c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.409+0000 7fe02a400640 1 --2- 172.21.15.114:0/3858502068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02407c2e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe018009820 tx=0x7fe01802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:25.172 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.409+0000 7fe02a400640 1 -- 172.21.15.114:0/3858502068 shutdown_connections 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.409+0000 7fe02a400640 1 --2- 172.21.15.114:0/3858502068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02407c2e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.409+0000 7fe02a400640 1 -- 172.21.15.114:0/3858502068 >> 172.21.15.114:0/3858502068 conn(0x7fe024102c10 msgr2=0x7fe024105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.409+0000 7fe02a400640 1 -- 172.21.15.114:0/3858502068 shutdown_connections 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.409+0000 7fe02a400640 1 -- 172.21.15.114:0/3858502068 wait complete. 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.410+0000 7fe02a400640 1 Processor -- start 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.410+0000 7fe02a400640 1 -- start start 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.410+0000 7fe02a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.410+0000 7fe02a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe02410da70 con 0x7fe02407de80 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.411+0000 7fe029000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.411+0000 7fe029000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02419d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57850/0 (socket says 172.21.15.114:57850) 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.411+0000 7fe029000640 1 -- 172.21.15.114:0/2655362318 learned_addr learned my addr 172.21.15.114:0/2655362318 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.411+0000 7fe029000640 1 -- 172.21.15.114:0/2655362318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02419dbf0 con 0x7fe02407de80 2024-09-19T06:41:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.411+0000 7fe029000640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02419d6b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe0180097f0 tx=0x7fe018030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.412+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe018042070 con 0x7fe02407de80 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.412+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe018032ca0 con 0x7fe02407de80 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.412+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe02419de80 con 0x7fe02407de80 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.412+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe018037040 con 0x7fe02407de80 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.412+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0241a0b70 con 0x7fe02407de80 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.413+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7fe018045070 con 0x7fe02407de80 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.413+0000 7fe021600640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 0x7fe00c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.413+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 --> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe00c03bdd0 con 0x7fe00c039200 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.413+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe01806c180 con 0x7fe02407de80 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.414+0000 7fe023e00640 1 -- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 msgr2=0x7fe00c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2384341327 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.414+0000 7fe023e00640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 0x7fe00c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.615+0000 7fe023e00640 1 -- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 msgr2=0x7fe00c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2384341327 2024-09-19T06:41:25.174 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:20.615+0000 7fe023e00640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 0x7fe00c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:21.016+0000 7fe023e00640 1 -- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 msgr2=0x7fe00c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2384341327 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:21.016+0000 7fe023e00640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 0x7fe00c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:21.817+0000 7fe023e00640 1 -- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 msgr2=0x7fe00c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2384341327 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:21.817+0000 7fe023e00640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 0x7fe00c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:23.419+0000 7fe023e00640 1 -- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 msgr2=0x7fe00c03b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2384341327 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:23.419+0000 7fe023e00640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 0x7fe00c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:23.470+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mgrmap(e 6) ==== 46228+0+0 (secure 0 0 0) 0x7fe01803cc40 con 0x7fe02407de80 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:23.470+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 msgr2=0x7fe00c03b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:23.470+0000 7fe021600640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 0x7fe00c03b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.471+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7fe01803c720 con 0x7fe02407de80 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.472+0000 7fe021600640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe00c03cb30 0x7fe00c03ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.472+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe00c03bdd0 con 0x7fe00c03cb30 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.472+0000 7fe023e00640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe00c03cb30 0x7fe00c03ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:25.175 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.474+0000 7fe023e00640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe00c03cb30 0x7fe00c03ef20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe010003980 tx=0x7fe0100092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.475+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mgr.14118 v2:172.21.15.114:6800/3886387391 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fe00c03bdd0 con 0x7fe00c03cb30 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.482+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe02407c360 con 0x7fe00c03cb30 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.482+0000 7fe021600640 1 -- 172.21.15.114:0/2655362318 <== mgr.14118 v2:172.21.15.114:6800/3886387391 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fe02407c360 con 0x7fe00c03cb30 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe00c03cb30 msgr2=0x7fe00c03ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe00c03cb30 0x7fe00c03ef20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe010003980 tx=0x7fe0100092b0 comp rx=0 tx=0).stop 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 msgr2=0x7fe02419d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02419d6b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe0180097f0 tx=0x7fe018030ec0 comp rx=0 tx=0).stop 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 shutdown_connections 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe00c03cb30 0x7fe00c03ef20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:6800/2384341327,v1:172.21.15.114:6801/2384341327] conn(0x7fe00c039200 0x7fe00c03b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 --2- 172.21.15.114:0/2655362318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe02407de80 0x7fe02419d6b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 >> 172.21.15.114:0/2655362318 conn(0x7fe024102c10 msgr2=0x7fe024103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:25.176 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 shutdown_connections 2024-09-19T06:41:25.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:24.483+0000 7fe02a400640 1 -- 172.21.15.114:0/2655362318 wait complete. 2024-09-19T06:41:25.177 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 5 is available 2024-09-19T06:41:25.177 INFO:teuthology.orchestra.run.smithi114.stdout:Setting orchestrator backend to cephadm... 2024-09-19T06:41:25.678 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:25 smithi114 ceph-mon[30820]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-19T06:41:25.679 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:25 smithi114 ceph-mon[30820]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.702+0000 7f5c5ca00640 1 Processor -- start 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.702+0000 7f5c5ca00640 1 -- start start 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.702+0000 7f5c5ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c58107840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.702+0000 7f5c5ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5c58107e10 con 0x7f5c58107440 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.703+0000 7f5c57400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c58107840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.703+0000 7f5c57400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c58107840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57946/0 (socket says 172.21.15.114:57946) 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.703+0000 7f5c57400640 1 -- 172.21.15.114:0/3158050304 learned_addr learned my addr 172.21.15.114:0/3158050304 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.703+0000 7f5c57400640 1 -- 172.21.15.114:0/3158050304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c58108640 con 0x7f5c58107440 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.704+0000 7f5c57400640 1 --2- 172.21.15.114:0/3158050304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c58107840 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009820 tx=0x7f5c4402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5242c15bf077be4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:26.536 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.704+0000 7f5c56000640 1 -- 172.21.15.114:0/3158050304 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c44037070 con 0x7f5c58107440 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.704+0000 7f5c56000640 1 -- 172.21.15.114:0/3158050304 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5c44032900 con 0x7f5c58107440 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.704+0000 7f5c56000640 1 -- 172.21.15.114:0/3158050304 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c44030850 con 0x7f5c58107440 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.705+0000 7f5c5ca00640 1 -- 172.21.15.114:0/3158050304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 msgr2=0x7f5c58107840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.705+0000 7f5c5ca00640 1 --2- 172.21.15.114:0/3158050304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c58107840 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009820 tx=0x7f5c4402a270 comp rx=0 tx=0).stop 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.705+0000 7f5c5ca00640 1 -- 172.21.15.114:0/3158050304 shutdown_connections 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.705+0000 7f5c5ca00640 1 --2- 172.21.15.114:0/3158050304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c58107840 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.705+0000 7f5c5ca00640 1 -- 172.21.15.114:0/3158050304 >> 172.21.15.114:0/3158050304 conn(0x7f5c58102c10 msgr2=0x7f5c58105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.705+0000 7f5c5ca00640 1 -- 172.21.15.114:0/3158050304 shutdown_connections 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.705+0000 7f5c5ca00640 1 -- 172.21.15.114:0/3158050304 wait complete. 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.706+0000 7f5c5ca00640 1 Processor -- start 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.707+0000 7f5c5ca00640 1 -- start start 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.707+0000 7f5c5ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c5819d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:26.537 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.707+0000 7f5c5ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5810b9f0 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.707+0000 7f5c57400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c5819d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.707+0000 7f5c57400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c5819d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57954/0 (socket says 172.21.15.114:57954) 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.707+0000 7f5c57400640 1 -- 172.21.15.114:0/419246070 learned_addr learned my addr 172.21.15.114:0/419246070 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.708+0000 7f5c57400640 1 -- 172.21.15.114:0/419246070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5819dc50 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.708+0000 7f5c57400640 1 --2- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c5819d710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5c440097f0 tx=0x7f5c44030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.708+0000 7f5c4fe00640 1 -- 172.21.15.114:0/419246070 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c44042070 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.708+0000 7f5c4fe00640 1 -- 172.21.15.114:0/419246070 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5c44032ca0 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.708+0000 7f5c5ca00640 1 -- 172.21.15.114:0/419246070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5819dee0 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.708+0000 7f5c4fe00640 1 -- 172.21.15.114:0/419246070 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c44037040 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.708+0000 7f5c5ca00640 1 -- 172.21.15.114:0/419246070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c581a0bd0 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.709+0000 7f5c4fe00640 1 -- 172.21.15.114:0/419246070 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f5c44045070 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.710+0000 7f5c4d600640 1 -- 172.21.15.114:0/419246070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c5810bb70 con 0x7f5c58107440 2024-09-19T06:41:26.538 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.710+0000 7f5c4fe00640 1 --2- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f5c3c0391b0 0x7f5c3c03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.710+0000 7f5c4fe00640 1 -- 172.21.15.114:0/419246070 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5c4406c660 con 0x7f5c58107440 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.710+0000 7f5c56a00640 1 --2- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f5c3c0391b0 0x7f5c3c03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.711+0000 7f5c56a00640 1 --2- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f5c3c0391b0 0x7f5c3c03b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5c480098c0 tx=0x7f5c48006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.716+0000 7f5c4fe00640 1 -- 172.21.15.114:0/419246070 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5c4404b170 con 0x7f5c58107440 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.926+0000 7f5c4d600640 1 -- 172.21.15.114:0/419246070 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f5c581078c0 con 0x7f5c3c0391b0 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.932+0000 7f5c4fe00640 1 -- 172.21.15.114:0/419246070 <== mgr.14118 v2:172.21.15.114:6800/3886387391 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5c581078c0 con 0x7f5c3c0391b0 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.934+0000 7f5c4d600640 1 -- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f5c3c0391b0 msgr2=0x7f5c3c03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.934+0000 7f5c4d600640 1 --2- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f5c3c0391b0 0x7f5c3c03b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5c480098c0 tx=0x7f5c48006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.934+0000 7f5c4d600640 1 -- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 msgr2=0x7f5c5819d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.934+0000 7f5c4d600640 1 --2- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c5819d710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5c440097f0 tx=0x7f5c44030ec0 comp rx=0 tx=0).stop 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.935+0000 7f5c4d600640 1 -- 172.21.15.114:0/419246070 shutdown_connections 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.935+0000 7f5c4d600640 1 --2- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f5c3c0391b0 0x7f5c3c03b670 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:26.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.935+0000 7f5c4d600640 1 --2- 172.21.15.114:0/419246070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5c58107440 0x7f5c5819d710 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:26.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.935+0000 7f5c4d600640 1 -- 172.21.15.114:0/419246070 >> 172.21.15.114:0/419246070 conn(0x7f5c58102c10 msgr2=0x7f5c581037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:26.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.935+0000 7f5c4d600640 1 -- 172.21.15.114:0/419246070 shutdown_connections 2024-09-19T06:41:26.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:25.935+0000 7f5c4d600640 1 -- 172.21.15.114:0/419246070 wait complete. 2024-09-19T06:41:26.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:26 smithi114 ceph-mon[30820]: mgrmap e8: x(active, since 2s) 2024-09-19T06:41:26.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:26 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:26.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:26 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:27.045+0000 7f4776c00640 -1 log_channel(cephadm) log [ERR] : [19/Sep/2024:06:41:27] ENGINE Error in HTTPServer.serve 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: self._connections.run(self.expiration_interval) 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: self._run(expiration_interval) 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T06:41:27.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: s = self.context.wrap_socket( 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return self.sslsocket_class._create( 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: self.do_handshake() 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: self._sslobj.do_handshake() 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T06:41:27.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:27.862 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-mon[30820]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:27.863 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:27 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:41:27.982 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-19T06:41:27.982 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.124+0000 7fdc9fe00640 1 Processor -- start 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.125+0000 7fdc9fe00640 1 -- start start 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.125+0000 7fdc9fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc98109c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.125+0000 7fdc9fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9810a200 con 0x7fdc98109830 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.125+0000 7fdc9d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc98109c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.125+0000 7fdc9d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc98109c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33260/0 (socket says 172.21.15.114:33260) 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.125+0000 7fdc9d400640 1 -- 172.21.15.114:0/1566510834 learned_addr learned my addr 172.21.15.114:0/1566510834 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.126+0000 7fdc9d400640 1 -- 172.21.15.114:0/1566510834 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9810aa40 con 0x7fdc98109830 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.127+0000 7fdc9d400640 1 --2- 172.21.15.114:0/1566510834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc98109c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009820 tx=0x7fdc8c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ee65b328f70a3f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.127+0000 7fdc97e00640 1 -- 172.21.15.114:0/1566510834 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c037070 con 0x7fdc98109830 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.127+0000 7fdc97e00640 1 -- 172.21.15.114:0/1566510834 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdc8c032900 con 0x7fdc98109830 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.127+0000 7fdc97e00640 1 -- 172.21.15.114:0/1566510834 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c030850 con 0x7fdc98109830 2024-09-19T06:41:27.983 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.127+0000 7fdc9fe00640 1 -- 172.21.15.114:0/1566510834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 msgr2=0x7fdc98109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.127+0000 7fdc9fe00640 1 --2- 172.21.15.114:0/1566510834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc98109c30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009820 tx=0x7fdc8c02a270 comp rx=0 tx=0).stop 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.128+0000 7fdc9fe00640 1 -- 172.21.15.114:0/1566510834 shutdown_connections 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.128+0000 7fdc9fe00640 1 --2- 172.21.15.114:0/1566510834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc98109c30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.128+0000 7fdc9fe00640 1 -- 172.21.15.114:0/1566510834 >> 172.21.15.114:0/1566510834 conn(0x7fdc9807ce40 msgr2=0x7fdc9807d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.128+0000 7fdc9fe00640 1 -- 172.21.15.114:0/1566510834 shutdown_connections 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.128+0000 7fdc9fe00640 1 -- 172.21.15.114:0/1566510834 wait complete. 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.129+0000 7fdc9fe00640 1 Processor -- start 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.129+0000 7fdc9fe00640 1 -- start start 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.129+0000 7fdc9fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc9819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.129+0000 7fdc9fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9810ddf0 con 0x7fdc98109830 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.129+0000 7fdc9d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc9819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.129+0000 7fdc9d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc9819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33276/0 (socket says 172.21.15.114:33276) 2024-09-19T06:41:27.984 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.129+0000 7fdc9d400640 1 -- 172.21.15.114:0/2485375080 learned_addr learned my addr 172.21.15.114:0/2485375080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.130+0000 7fdc9d400640 1 -- 172.21.15.114:0/2485375080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9819de40 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.130+0000 7fdc9d400640 1 --2- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc9819d900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009940 tx=0x7fdc8c0326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.130+0000 7fdc96000640 1 -- 172.21.15.114:0/2485375080 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c042070 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.130+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9819e0d0 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.130+0000 7fdc96000640 1 -- 172.21.15.114:0/2485375080 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdc8c032be0 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.130+0000 7fdc96000640 1 -- 172.21.15.114:0/2485375080 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c037040 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.130+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc981a0dc0 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.131+0000 7fdc96000640 1 -- 172.21.15.114:0/2485375080 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fdc8c045070 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.132+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc98109c30 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.132+0000 7fdc96000640 1 --2- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fdc80039200 0x7fdc8003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.132+0000 7fdc96000640 1 -- 172.21.15.114:0/2485375080 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdc8c06c470 con 0x7fdc98109830 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.132+0000 7fdc9ca00640 1 --2- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fdc80039200 0x7fdc8003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:27.985 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.133+0000 7fdc9ca00640 1 --2- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fdc80039200 0x7fdc8003b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc840098c0 tx=0x7fdc84006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.137+0000 7fdc96000640 1 -- 172.21.15.114:0/2485375080 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdc8c032e00 con 0x7fdc98109830 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.314+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fdc98107cc0 con 0x7fdc80039200 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.314+0000 7fdc96000640 1 -- 172.21.15.114:0/2485375080 <== mgr.14118 v2:172.21.15.114:6800/3886387391 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fdc98107cc0 con 0x7fdc80039200 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fdc80039200 msgr2=0x7fdc8003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 --2- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fdc80039200 0x7fdc8003b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdc840098c0 tx=0x7fdc84006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 msgr2=0x7fdc9819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 --2- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc9819d900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009940 tx=0x7fdc8c0326e0 comp rx=0 tx=0).stop 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 shutdown_connections 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 --2- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fdc80039200 0x7fdc8003b6c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 --2- 172.21.15.114:0/2485375080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc98109830 0x7fdc9819d900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 >> 172.21.15.114:0/2485375080 conn(0x7fdc9807ce40 msgr2=0x7fdc981075b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 shutdown_connections 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:27.317+0000 7fdc9fe00640 1 -- 172.21.15.114:0/2485375080 wait complete. 2024-09-19T06:41:27.986 INFO:teuthology.orchestra.run.smithi114.stdout:Generating ssh key... 2024-09-19T06:41:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:28 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:26] ENGINE Bus STARTING 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:28 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:27] ENGINE Serving on https://172.21.15.114:7150 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:28 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:27] ENGINE Error in HTTPServer.serve 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T06:41:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:28 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:27] ENGINE Serving on http://172.21.15.114:8765 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:28 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:27] ENGINE Bus STARTED 2024-09-19T06:41:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:28 smithi114 ceph-mon[30820]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Generating public/private rsa key pair. 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Your identification has been saved in /tmp/tmpdwpdbdpb/key 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Your public key has been saved in /tmp/tmpdwpdbdpb/key.pub 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: The key fingerprint is: 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: SHA256:RcF22Q4dOtUglglZhEVyh3VyWl4Fl1X4eGr9vMuiVoE ceph-fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: The key's randomart image is: 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: +---[RSA 3072]----+ 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | .=XOOO*#| 2024-09-19T06:41:29.379 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | .=+B=+Xo| 2024-09-19T06:41:29.380 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | ...+o.o.| 2024-09-19T06:41:29.380 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | . E oo o| 2024-09-19T06:41:29.380 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | S .+ | 2024-09-19T06:41:29.380 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | .o .| 2024-09-19T06:41:29.380 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | .. ..| 2024-09-19T06:41:29.380 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | . .. o| 2024-09-19T06:41:29.380 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: | ... .+o| 2024-09-19T06:41:29.380 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: +----[SHA256]-----+ 2024-09-19T06:41:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-mon[30820]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-mon[30820]: Generating ssh key... 2024-09-19T06:41:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:29 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:29.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.483+0000 7fbcf9000640 1 Processor -- start 2024-09-19T06:41:29.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.483+0000 7fbcf9000640 1 -- start start 2024-09-19T06:41:29.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.483+0000 7fbcf9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf4109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:29.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.483+0000 7fbcf9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf410a290 con 0x7fbcf41098c0 2024-09-19T06:41:29.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.484+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf4109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:29.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.484+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf4109cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33282/0 (socket says 172.21.15.114:33282) 2024-09-19T06:41:29.732 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.484+0000 7fbcf1e00640 1 -- 172.21.15.114:0/2978065931 learned_addr learned my addr 172.21.15.114:0/2978065931 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.484+0000 7fbcf1e00640 1 -- 172.21.15.114:0/2978065931 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf410aa70 con 0x7fbcf41098c0 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.485+0000 7fbcf1e00640 1 --2- 172.21.15.114:0/2978065931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf4109cc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009820 tx=0x7fbce402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339aac2610cded37 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.485+0000 7fbcf0a00640 1 -- 172.21.15.114:0/2978065931 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbce4037070 con 0x7fbcf41098c0 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.485+0000 7fbcf0a00640 1 -- 172.21.15.114:0/2978065931 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbce4032900 con 0x7fbcf41098c0 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.485+0000 7fbcf0a00640 1 -- 172.21.15.114:0/2978065931 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbce4030850 con 0x7fbcf41098c0 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.486+0000 7fbcf9000640 1 -- 172.21.15.114:0/2978065931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 msgr2=0x7fbcf4109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.486+0000 7fbcf9000640 1 --2- 172.21.15.114:0/2978065931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf4109cc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbce4009820 tx=0x7fbce402a270 comp rx=0 tx=0).stop 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.486+0000 7fbcf9000640 1 -- 172.21.15.114:0/2978065931 shutdown_connections 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.486+0000 7fbcf9000640 1 --2- 172.21.15.114:0/2978065931 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf4109cc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.486+0000 7fbcf9000640 1 -- 172.21.15.114:0/2978065931 >> 172.21.15.114:0/2978065931 conn(0x7fbcf407cf40 msgr2=0x7fbcf407d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:29.733 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.486+0000 7fbcf9000640 1 -- 172.21.15.114:0/2978065931 shutdown_connections 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.486+0000 7fbcf9000640 1 -- 172.21.15.114:0/2978065931 wait complete. 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.487+0000 7fbcf9000640 1 Processor -- start 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.487+0000 7fbcf9000640 1 -- start start 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.487+0000 7fbcf9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf419d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.487+0000 7fbcf9000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf410de20 con 0x7fbcf41098c0 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.487+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf419d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.487+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf419d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33286/0 (socket says 172.21.15.114:33286) 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.487+0000 7fbcf1e00640 1 -- 172.21.15.114:0/626781862 learned_addr learned my addr 172.21.15.114:0/626781862 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.488+0000 7fbcf1e00640 1 -- 172.21.15.114:0/626781862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf419ddd0 con 0x7fbcf41098c0 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.488+0000 7fbcf1e00640 1 --2- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf419d890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbce40097f0 tx=0x7fbce4030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:29.734 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.488+0000 7fbceaa00640 1 -- 172.21.15.114:0/626781862 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbce4037040 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.488+0000 7fbceaa00640 1 -- 172.21.15.114:0/626781862 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbce4040070 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.488+0000 7fbceaa00640 1 -- 172.21.15.114:0/626781862 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbce403a390 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.488+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf419e060 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.489+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf41a0d50 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.490+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf4109d40 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.490+0000 7fbceaa00640 1 -- 172.21.15.114:0/626781862 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fbce403b430 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.490+0000 7fbceaa00640 1 --2- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fbcd8039200 0x7fbcd803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.490+0000 7fbceaa00640 1 -- 172.21.15.114:0/626781862 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbce406c7d0 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.491+0000 7fbcf1400640 1 --2- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fbcd8039200 0x7fbcd803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.491+0000 7fbcf1400640 1 --2- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fbcd8039200 0x7fbcd803b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbce00098c0 tx=0x7fbce0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.495+0000 7fbceaa00640 1 -- 172.21.15.114:0/626781862 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbce406c120 con 0x7fbcf41098c0 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:28.671+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fbcf4107c40 con 0x7fbcd8039200 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.012+0000 7fbceaa00640 1 -- 172.21.15.114:0/626781862 <== mgr.14118 v2:172.21.15.114:6800/3886387391 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbcf4107c40 con 0x7fbcd8039200 2024-09-19T06:41:29.735 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fbcd8039200 msgr2=0x7fbcd803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 --2- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fbcd8039200 0x7fbcd803b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbce00098c0 tx=0x7fbce0006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 msgr2=0x7fbcf419d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 --2- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf419d890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbce40097f0 tx=0x7fbce4030d80 comp rx=0 tx=0).stop 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 shutdown_connections 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 --2- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fbcd8039200 0x7fbcd803b6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 --2- 172.21.15.114:0/626781862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf41098c0 0x7fbcf419d890 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 >> 172.21.15.114:0/626781862 conn(0x7fbcf407cf40 msgr2=0x7fbcf4107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 shutdown_connections 2024-09-19T06:41:29.736 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:29.015+0000 7fbcf9000640 1 -- 172.21.15.114:0/626781862 wait complete. 2024-09-19T06:41:31.142 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:30 smithi114 ceph-mon[30820]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:31.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDev5FMHV0qE+tC2R7ZXOwW5nSTR09slp886CpbnSr+sFzI8/sh5Fv1qCFoR0HwrBG07kc+su9t60r1nLtTkgbpkje3H217SXfu4+K9nPoPlfTzwOIn69xHNsI3/nD1OEQNcOQw99ewX2VSUgjsZert25Istz8dMLXODYXjkmQxTxJhUus4Z7XyJ5r14rgPjmyv52mKb4kA3Dlf+M8zPpXdHpRLk9XMgZ4Eic3Ho5nWtKk1yIPIQXaOK85HRFqLQ/0DkAx4YE3zatLbtv5XgDEXg0TbVAe9jWwrmVZ2PsjBRBrQuoKm0Z4lym2Ke7fUnWTPtqDbqk9jJpxO0Ojy24dfIACy/Lgo9XZV7hT3MWJPdwWr+ZQhaZnuBzn2lpcj8+a1RhffAGOY3pB+j6ygB8VD/u8MVrqob/biXb5UfG3zPFDLtbAkpKbFPusJ7cjX1b1Ncx7mV0170+jlmGGYhM2uDC1pgoAoqdQUN1vHJ6yomYGYqruU13Jxgits3mSgwHc= ceph-fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:41:31.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.233+0000 7f86ab200640 1 Processor -- start 2024-09-19T06:41:31.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.234+0000 7f86ab200640 1 -- start start 2024-09-19T06:41:31.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.234+0000 7f86ab200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac07c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:31.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.234+0000 7f86ab200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f86ac07c820 con 0x7f86ac07dde0 2024-09-19T06:41:31.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.234+0000 7f86a9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac07c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.234+0000 7f86a9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac07c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33302/0 (socket says 172.21.15.114:33302) 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.234+0000 7f86a9e00640 1 -- 172.21.15.114:0/2376306117 learned_addr learned my addr 172.21.15.114:0/2376306117 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.235+0000 7f86a9e00640 1 -- 172.21.15.114:0/2376306117 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86ac07c9a0 con 0x7f86ac07dde0 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.236+0000 7f86a9e00640 1 --2- 172.21.15.114:0/2376306117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac07c2e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f869c009820 tx=0x7f869c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d5b5d70dd8ea5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.236+0000 7f86a9400640 1 -- 172.21.15.114:0/2376306117 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f869c037070 con 0x7f86ac07dde0 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.236+0000 7f86a9400640 1 -- 172.21.15.114:0/2376306117 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f869c032900 con 0x7f86ac07dde0 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.236+0000 7f86a9400640 1 -- 172.21.15.114:0/2376306117 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f869c030850 con 0x7f86ac07dde0 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.237+0000 7f86ab200640 1 -- 172.21.15.114:0/2376306117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 msgr2=0x7f86ac07c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.237+0000 7f86ab200640 1 --2- 172.21.15.114:0/2376306117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac07c2e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f869c009820 tx=0x7f869c02a270 comp rx=0 tx=0).stop 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.237+0000 7f86ab200640 1 -- 172.21.15.114:0/2376306117 shutdown_connections 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.237+0000 7f86ab200640 1 --2- 172.21.15.114:0/2376306117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac07c2e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.237+0000 7f86ab200640 1 -- 172.21.15.114:0/2376306117 >> 172.21.15.114:0/2376306117 conn(0x7f86ac102d00 msgr2=0x7f86ac105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:31.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.237+0000 7f86ab200640 1 -- 172.21.15.114:0/2376306117 shutdown_connections 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.237+0000 7f86ab200640 1 -- 172.21.15.114:0/2376306117 wait complete. 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.238+0000 7f86ab200640 1 Processor -- start 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.238+0000 7f86ab200640 1 -- start start 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.238+0000 7f86ab200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.238+0000 7f86ab200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f86ac10dac0 con 0x7f86ac07dde0 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.238+0000 7f86a9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.239+0000 7f86a9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac19d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33314/0 (socket says 172.21.15.114:33314) 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.239+0000 7f86a9e00640 1 -- 172.21.15.114:0/1366189149 learned_addr learned my addr 172.21.15.114:0/1366189149 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.239+0000 7f86a9e00640 1 -- 172.21.15.114:0/1366189149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86ac19dbc0 con 0x7f86ac07dde0 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.239+0000 7f86a9e00640 1 --2- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac19d680 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f869c031000 tx=0x7f869c0326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.239+0000 7f86a3400640 1 -- 172.21.15.114:0/1366189149 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f869c042070 con 0x7f86ac07dde0 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.239+0000 7f86a3400640 1 -- 172.21.15.114:0/1366189149 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f869c032be0 con 0x7f86ac07dde0 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.239+0000 7f86a3400640 1 -- 172.21.15.114:0/1366189149 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f869c037040 con 0x7f86ac07dde0 2024-09-19T06:41:31.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.239+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86ac19de50 con 0x7f86ac07dde0 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.240+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86ac1a0b40 con 0x7f86ac07dde0 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.241+0000 7f86a3400640 1 -- 172.21.15.114:0/1366189149 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f869c045070 con 0x7f86ac07dde0 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.241+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86ac07c360 con 0x7f86ac07dde0 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.241+0000 7f86a3400640 1 --2- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8690039200 0x7f869003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.241+0000 7f86a3400640 1 -- 172.21.15.114:0/1366189149 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f869c06c300 con 0x7f86ac07dde0 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.241+0000 7f86a1400640 1 --2- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8690039200 0x7f869003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.242+0000 7f86a1400640 1 --2- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8690039200 0x7f869003b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86940098c0 tx=0x7f8694006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.246+0000 7f86a3400640 1 -- 172.21.15.114:0/1366189149 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f869c02f2a0 con 0x7f86ac07dde0 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.421+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f86ac103fc0 con 0x7f8690039200 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.422+0000 7f86a3400640 1 -- 172.21.15.114:0/1366189149 <== mgr.14118 v2:172.21.15.114:6800/3886387391 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f86ac103fc0 con 0x7f8690039200 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8690039200 msgr2=0x7f869003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 --2- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8690039200 0x7f869003b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86940098c0 tx=0x7f8694006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:31.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 msgr2=0x7f86ac19d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 --2- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac19d680 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f869c031000 tx=0x7f869c0326e0 comp rx=0 tx=0).stop 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 shutdown_connections 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 --2- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8690039200 0x7f869003b6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 --2- 172.21.15.114:0/1366189149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86ac07dde0 0x7f86ac19d680 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 >> 172.21.15.114:0/1366189149 conn(0x7f86ac102d00 msgr2=0x7f86ac1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 shutdown_connections 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:30.424+0000 7f86ab200640 1 -- 172.21.15.114:0/1366189149 wait complete. 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-19T06:41:31.157 INFO:teuthology.orchestra.run.smithi114.stdout:Adding key to root@localhost authorized_keys... 2024-09-19T06:41:31.169 INFO:teuthology.orchestra.run.smithi114.stdout:Adding host smithi114... 2024-09-19T06:41:33.129 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:32 smithi114 ceph-mon[30820]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:33.984 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:33 smithi114 ceph-mon[30820]: Deploying cephadm binary to smithi114 2024-09-19T06:41:36.588 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Added host 'smithi114' with addr '172.21.15.114' 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.677+0000 7f15b5c00640 1 Processor -- start 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.677+0000 7f15b5c00640 1 -- start start 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.678+0000 7f15b5c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b0109c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.678+0000 7f15b5c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f15b010a240 con 0x7f15b0109870 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.678+0000 7f15aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b0109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.678+0000 7f15aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b0109c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33330/0 (socket says 172.21.15.114:33330) 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.678+0000 7f15aea00640 1 -- 172.21.15.114:0/2637478571 learned_addr learned my addr 172.21.15.114:0/2637478571 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.678+0000 7f15aea00640 1 -- 172.21.15.114:0/2637478571 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b010aa10 con 0x7f15b0109870 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.679+0000 7f15aea00640 1 --2- 172.21.15.114:0/2637478571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b0109c70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009820 tx=0x7f15a002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20b3527f3743d81 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:36.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15ad600640 1 -- 172.21.15.114:0/2637478571 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a0037070 con 0x7f15b0109870 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15ad600640 1 -- 172.21.15.114:0/2637478571 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f15a0032900 con 0x7f15b0109870 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15ad600640 1 -- 172.21.15.114:0/2637478571 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a0030850 con 0x7f15b0109870 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15b5c00640 1 -- 172.21.15.114:0/2637478571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 msgr2=0x7f15b0109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15b5c00640 1 --2- 172.21.15.114:0/2637478571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b0109c70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009820 tx=0x7f15a002a270 comp rx=0 tx=0).stop 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15b5c00640 1 -- 172.21.15.114:0/2637478571 shutdown_connections 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15b5c00640 1 --2- 172.21.15.114:0/2637478571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b0109c70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15b5c00640 1 -- 172.21.15.114:0/2637478571 >> 172.21.15.114:0/2637478571 conn(0x7f15b007cf40 msgr2=0x7f15b007d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.680+0000 7f15b5c00640 1 -- 172.21.15.114:0/2637478571 shutdown_connections 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.681+0000 7f15b5c00640 1 -- 172.21.15.114:0/2637478571 wait complete. 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.681+0000 7f15b5c00640 1 Processor -- start 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.681+0000 7f15b5c00640 1 -- start start 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.682+0000 7f15b5c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.682+0000 7f15b5c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f15b010ddc0 con 0x7f15b0109870 2024-09-19T06:41:36.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.682+0000 7f15aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.682+0000 7f15aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b019d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33338/0 (socket says 172.21.15.114:33338) 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.682+0000 7f15aea00640 1 -- 172.21.15.114:0/2863166219 learned_addr learned my addr 172.21.15.114:0/2863166219 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.682+0000 7f15aea00640 1 -- 172.21.15.114:0/2863166219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b019dd20 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.683+0000 7f15aea00640 1 --2- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b019d7e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f15a0032ce0 tx=0x7f15a0032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.683+0000 7f15a7400640 1 -- 172.21.15.114:0/2863166219 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a0042070 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.683+0000 7f15a7400640 1 -- 172.21.15.114:0/2863166219 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f15a00308c0 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.683+0000 7f15a7400640 1 -- 172.21.15.114:0/2863166219 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15a0037040 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.683+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b019dfb0 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.683+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b01a0ca0 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.684+0000 7f15a7400640 1 -- 172.21.15.114:0/2863166219 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f15a0045070 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.684+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b0109c70 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.684+0000 7f15a7400640 1 --2- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f1594039200 0x7f159403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.685+0000 7f15a7400640 1 -- 172.21.15.114:0/2863166219 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f15a006c320 con 0x7f15b0109870 2024-09-19T06:41:36.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.685+0000 7f15ae000640 1 --2- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f1594039200 0x7f159403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.685+0000 7f15ae000640 1 --2- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f1594039200 0x7f159403b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f159c0098c0 tx=0x7f159c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.690+0000 7f15a7400640 1 -- 172.21.15.114:0/2863166219 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f15a006bc70 con 0x7f15b0109870 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:31.868+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}) -- 0x7f15b0107b50 con 0x7f1594039200 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.684+0000 7f15a7400640 1 -- 172.21.15.114:0/2863166219 <== mgr.14118 v2:172.21.15.114:6800/3886387391 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f15b0107b50 con 0x7f1594039200 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f1594039200 msgr2=0x7f159403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 --2- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f1594039200 0x7f159403b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f159c0098c0 tx=0x7f159c006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 msgr2=0x7f15b019d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 --2- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b019d7e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f15a0032ce0 tx=0x7f15a0032d10 comp rx=0 tx=0).stop 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 shutdown_connections 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 --2- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f1594039200 0x7f159403b6c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 --2- 172.21.15.114:0/2863166219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15b0109870 0x7f15b019d7e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 >> 172.21.15.114:0/2863166219 conn(0x7f15b007cf40 msgr2=0x7f15b0107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 shutdown_connections 2024-09-19T06:41:36.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:35.688+0000 7f15b5c00640 1 -- 172.21.15.114:0/2863166219 wait complete. 2024-09-19T06:41:36.593 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mon service... 2024-09-19T06:41:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:36 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:36 smithi114 ceph-mon[30820]: Added host smithi114 2024-09-19T06:41:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:36 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:41:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:38 smithi114 ceph-mon[30820]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:38 smithi114 ceph-mon[30820]: Saving service mon spec with placement count:5 2024-09-19T06:41:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:38 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:39.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-19T06:41:39.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.231+0000 7f7b67200640 1 Processor -- start 2024-09-19T06:41:39.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.232+0000 7f7b67200640 1 -- start start 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.232+0000 7f7b67200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b68109d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.232+0000 7f7b67200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6810a340 con 0x7f7b68109970 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.232+0000 7f7b65e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b68109d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.233+0000 7f7b65e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b68109d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60226/0 (socket says 172.21.15.114:60226) 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.233+0000 7f7b65e00640 1 -- 172.21.15.114:0/1844303270 learned_addr learned my addr 172.21.15.114:0/1844303270 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.233+0000 7f7b65e00640 1 -- 172.21.15.114:0/1844303270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6810a840 con 0x7f7b68109970 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.234+0000 7f7b65e00640 1 --2- 172.21.15.114:0/1844303270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b68109d70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009820 tx=0x7f7b5802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8fc1eb67cbeb46 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.234+0000 7f7b65400640 1 -- 172.21.15.114:0/1844303270 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b58037070 con 0x7f7b68109970 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.234+0000 7f7b65400640 1 -- 172.21.15.114:0/1844303270 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7b58032900 con 0x7f7b68109970 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.234+0000 7f7b65400640 1 -- 172.21.15.114:0/1844303270 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b58030850 con 0x7f7b68109970 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.235+0000 7f7b67200640 1 -- 172.21.15.114:0/1844303270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 msgr2=0x7f7b68109d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.235+0000 7f7b67200640 1 --2- 172.21.15.114:0/1844303270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b68109d70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009820 tx=0x7f7b5802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:39.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.235+0000 7f7b67200640 1 -- 172.21.15.114:0/1844303270 shutdown_connections 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.235+0000 7f7b67200640 1 --2- 172.21.15.114:0/1844303270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b68109d70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.235+0000 7f7b67200640 1 -- 172.21.15.114:0/1844303270 >> 172.21.15.114:0/1844303270 conn(0x7f7b6807cca0 msgr2=0x7f7b6807d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.235+0000 7f7b67200640 1 -- 172.21.15.114:0/1844303270 shutdown_connections 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.235+0000 7f7b67200640 1 -- 172.21.15.114:0/1844303270 wait complete. 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.236+0000 7f7b67200640 1 Processor -- start 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.236+0000 7f7b67200640 1 -- start start 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.236+0000 7f7b67200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b6819d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.236+0000 7f7b67200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6810dbf0 con 0x7f7b68109970 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.237+0000 7f7b65e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b6819d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.237+0000 7f7b65e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b6819d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60232/0 (socket says 172.21.15.114:60232) 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.237+0000 7f7b65e00640 1 -- 172.21.15.114:0/616085997 learned_addr learned my addr 172.21.15.114:0/616085997 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.237+0000 7f7b65e00640 1 -- 172.21.15.114:0/616085997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6819dc70 con 0x7f7b68109970 2024-09-19T06:41:39.155 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.237+0000 7f7b65e00640 1 --2- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b6819d730 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7b580097f0 tx=0x7f7b58030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.238+0000 7f7b5f400640 1 -- 172.21.15.114:0/616085997 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b58042070 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.238+0000 7f7b5f400640 1 -- 172.21.15.114:0/616085997 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7b58032be0 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.238+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6819df00 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.238+0000 7f7b5f400640 1 -- 172.21.15.114:0/616085997 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b58037040 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.238+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b681a0bf0 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.239+0000 7f7b5f400640 1 -- 172.21.15.114:0/616085997 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f7b58045070 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.239+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b68109df0 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.239+0000 7f7b5f400640 1 --2- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f7b4c0391b0 0x7f7b4c03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.239+0000 7f7b5f400640 1 -- 172.21.15.114:0/616085997 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7b5806c200 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.239+0000 7f7b5d400640 1 --2- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f7b4c0391b0 0x7f7b4c03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.240+0000 7f7b5d400640 1 --2- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f7b4c0391b0 0x7f7b4c03b670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7b500098c0 tx=0x7f7b50006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.244+0000 7f7b5f400640 1 -- 172.21.15.114:0/616085997 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7b5804b170 con 0x7f7b68109970 2024-09-19T06:41:39.156 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.440+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f7b68107d10 con 0x7f7b4c0391b0 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.445+0000 7f7b5f400640 1 -- 172.21.15.114:0/616085997 <== mgr.14118 v2:172.21.15.114:6800/3886387391 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7b68107d10 con 0x7f7b4c0391b0 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f7b4c0391b0 msgr2=0x7f7b4c03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 --2- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f7b4c0391b0 0x7f7b4c03b670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7b500098c0 tx=0x7f7b50006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 msgr2=0x7f7b6819d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 --2- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b6819d730 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7b580097f0 tx=0x7f7b58030d80 comp rx=0 tx=0).stop 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 shutdown_connections 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 --2- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f7b4c0391b0 0x7f7b4c03b670 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 --2- 172.21.15.114:0/616085997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7b68109970 0x7f7b6819d730 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 >> 172.21.15.114:0/616085997 conn(0x7f7b6807cca0 msgr2=0x7f7b68107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 shutdown_connections 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:37.448+0000 7f7b67200640 1 -- 172.21.15.114:0/616085997 wait complete. 2024-09-19T06:41:39.157 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mgr service... 2024-09-19T06:41:39.944 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:39 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:39.945 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:39 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:40.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-19T06:41:40.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.696+0000 7fc56f200640 1 Processor -- start 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.696+0000 7fc56f200640 1 -- start start 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.696+0000 7fc56f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.696+0000 7fc56f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc56807c8b0 con 0x7fc56807de80 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.697+0000 7fc56de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.697+0000 7fc56de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60246/0 (socket says 172.21.15.114:60246) 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.697+0000 7fc56de00640 1 -- 172.21.15.114:0/2956154077 learned_addr learned my addr 172.21.15.114:0/2956154077 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.697+0000 7fc56de00640 1 -- 172.21.15.114:0/2956154077 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56807ca30 con 0x7fc56807de80 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.698+0000 7fc56de00640 1 --2- 172.21.15.114:0/2956154077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56807c2e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009820 tx=0x7fc55c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be0a1ca21b9d2b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.699+0000 7fc56ca00640 1 -- 172.21.15.114:0/2956154077 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc55c037070 con 0x7fc56807de80 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.699+0000 7fc56ca00640 1 -- 172.21.15.114:0/2956154077 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc55c032920 con 0x7fc56807de80 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.699+0000 7fc56ca00640 1 -- 172.21.15.114:0/2956154077 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc55c030810 con 0x7fc56807de80 2024-09-19T06:41:40.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.699+0000 7fc56f200640 1 -- 172.21.15.114:0/2956154077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 msgr2=0x7fc56807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.699+0000 7fc56f200640 1 --2- 172.21.15.114:0/2956154077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56807c2e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc55c009820 tx=0x7fc55c02a270 comp rx=0 tx=0).stop 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.700+0000 7fc56f200640 1 -- 172.21.15.114:0/2956154077 shutdown_connections 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.700+0000 7fc56f200640 1 --2- 172.21.15.114:0/2956154077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56807c2e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.700+0000 7fc56f200640 1 -- 172.21.15.114:0/2956154077 >> 172.21.15.114:0/2956154077 conn(0x7fc568102c10 msgr2=0x7fc568105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.700+0000 7fc56f200640 1 -- 172.21.15.114:0/2956154077 shutdown_connections 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.700+0000 7fc56f200640 1 -- 172.21.15.114:0/2956154077 wait complete. 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56f200640 1 Processor -- start 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56f200640 1 -- start start 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56819d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc56810dad0 con 0x7fc56807de80 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56819d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56819d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60252/0 (socket says 172.21.15.114:60252) 2024-09-19T06:41:40.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56de00640 1 -- 172.21.15.114:0/1021485463 learned_addr learned my addr 172.21.15.114:0/1021485463 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56de00640 1 -- 172.21.15.114:0/1021485463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56819dc20 con 0x7fc56807de80 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56de00640 1 --2- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56819d6e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0097f0 tx=0x7fc55c0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc562a00640 1 -- 172.21.15.114:0/1021485463 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc55c037040 con 0x7fc56807de80 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc562a00640 1 -- 172.21.15.114:0/1021485463 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc55c030c40 con 0x7fc56807de80 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc562a00640 1 -- 172.21.15.114:0/1021485463 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc55c03a5d0 con 0x7fc56807de80 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.702+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56819deb0 con 0x7fc56807de80 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.703+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5681a0ba0 con 0x7fc56807de80 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.705+0000 7fc562a00640 1 -- 172.21.15.114:0/1021485463 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fc55c03b3f0 con 0x7fc56807de80 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.705+0000 7fc562a00640 1 --2- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fc550039260 0x7fc55003b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.705+0000 7fc562a00640 1 -- 172.21.15.114:0/1021485463 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc55c06c7e0 con 0x7fc56807de80 2024-09-19T06:41:40.545 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.705+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56807c360 con 0x7fc56807de80 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.705+0000 7fc56d400640 1 --2- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fc550039260 0x7fc55003b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.706+0000 7fc56d400640 1 --2- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fc550039260 0x7fc55003b720 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc5580098c0 tx=0x7fc558006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.710+0000 7fc562a00640 1 -- 172.21.15.114:0/1021485463 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc55c0371e0 con 0x7fc56807de80 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.922+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fc568103f20 con 0x7fc550039260 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.928+0000 7fc562a00640 1 -- 172.21.15.114:0/1021485463 <== mgr.14118 v2:172.21.15.114:6800/3886387391 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc568103f20 con 0x7fc550039260 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fc550039260 msgr2=0x7fc55003b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 --2- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fc550039260 0x7fc55003b720 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc5580098c0 tx=0x7fc558006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 msgr2=0x7fc56819d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 --2- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56819d6e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0097f0 tx=0x7fc55c0327b0 comp rx=0 tx=0).stop 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 shutdown_connections 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 --2- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fc550039260 0x7fc55003b720 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 --2- 172.21.15.114:0/1021485463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc56807de80 0x7fc56819d6e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:40.547 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 >> 172.21.15.114:0/1021485463 conn(0x7fc568102c10 msgr2=0x7fc568103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:40.547 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 shutdown_connections 2024-09-19T06:41:40.547 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:39.930+0000 7fc56f200640 1 -- 172.21.15.114:0/1021485463 wait complete. 2024-09-19T06:41:40.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:40 smithi114 ceph-mon[30820]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:40.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:40 smithi114 ceph-mon[30820]: Saving service mgr spec with placement count:2 2024-09-19T06:41:40.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:40 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:42.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:42 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1758955406' entity='client.admin' 2024-09-19T06:41:42.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:42 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:42.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.079+0000 7f2342c00640 1 Processor -- start 2024-09-19T06:41:42.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.079+0000 7f2342c00640 1 -- start start 2024-09-19T06:41:42.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.079+0000 7f2342c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c07c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:42.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.079+0000 7f2342c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f233c07c8d0 con 0x7f233c07dea0 2024-09-19T06:41:42.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.080+0000 7f2341800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c07c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:42.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.080+0000 7f2341800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c07c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60268/0 (socket says 172.21.15.114:60268) 2024-09-19T06:41:42.341 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.080+0000 7f2341800640 1 -- 172.21.15.114:0/3548103625 learned_addr learned my addr 172.21.15.114:0/3548103625 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.080+0000 7f2341800640 1 -- 172.21.15.114:0/3548103625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f233c07ca50 con 0x7f233c07dea0 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.081+0000 7f2341800640 1 --2- 172.21.15.114:0/3548103625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c07c300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2330009820 tx=0x7f233002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e407a0b9d50267 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.081+0000 7f233be00640 1 -- 172.21.15.114:0/3548103625 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2330037070 con 0x7f233c07dea0 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.081+0000 7f233be00640 1 -- 172.21.15.114:0/3548103625 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2330032920 con 0x7f233c07dea0 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.081+0000 7f233be00640 1 -- 172.21.15.114:0/3548103625 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2330030810 con 0x7f233c07dea0 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.082+0000 7f2342c00640 1 -- 172.21.15.114:0/3548103625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 msgr2=0x7f233c07c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.082+0000 7f2342c00640 1 --2- 172.21.15.114:0/3548103625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c07c300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2330009820 tx=0x7f233002a270 comp rx=0 tx=0).stop 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.082+0000 7f2342c00640 1 -- 172.21.15.114:0/3548103625 shutdown_connections 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.082+0000 7f2342c00640 1 --2- 172.21.15.114:0/3548103625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c07c300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.082+0000 7f2342c00640 1 -- 172.21.15.114:0/3548103625 >> 172.21.15.114:0/3548103625 conn(0x7f233c102c10 msgr2=0x7f233c105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:42.342 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.082+0000 7f2342c00640 1 -- 172.21.15.114:0/3548103625 shutdown_connections 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.082+0000 7f2342c00640 1 -- 172.21.15.114:0/3548103625 wait complete. 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.083+0000 7f2342c00640 1 Processor -- start 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.084+0000 7f2342c00640 1 -- start start 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.084+0000 7f2342c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c19d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.084+0000 7f2342c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f233c10db10 con 0x7f233c07dea0 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.084+0000 7f2341800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c19d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.084+0000 7f2341800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c19d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60276/0 (socket says 172.21.15.114:60276) 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.084+0000 7f2341800640 1 -- 172.21.15.114:0/1758955406 learned_addr learned my addr 172.21.15.114:0/1758955406 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.085+0000 7f2341800640 1 -- 172.21.15.114:0/1758955406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f233c19dc00 con 0x7f233c07dea0 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.085+0000 7f2341800640 1 --2- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c19d6c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f23300097f0 tx=0x7f23300327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.085+0000 7f233a000640 1 -- 172.21.15.114:0/1758955406 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2330037040 con 0x7f233c07dea0 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.085+0000 7f233a000640 1 -- 172.21.15.114:0/1758955406 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2330030c40 con 0x7f233c07dea0 2024-09-19T06:41:42.343 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.085+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f233c19de90 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.085+0000 7f233a000640 1 -- 172.21.15.114:0/1758955406 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f233003a5d0 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.085+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f233c1a0b80 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.086+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f233c07c380 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.087+0000 7f233a000640 1 -- 172.21.15.114:0/1758955406 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f233003b3f0 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.087+0000 7f233a000640 1 --2- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f23240391b0 0x7f232403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.087+0000 7f233a000640 1 -- 172.21.15.114:0/1758955406 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f233006c830 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.087+0000 7f2340e00640 1 --2- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f23240391b0 0x7f232403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.088+0000 7f2340e00640 1 --2- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f23240391b0 0x7f232403b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f23280098c0 tx=0x7f2328006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.096+0000 7f233a000640 1 -- 172.21.15.114:0/1758955406 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f23300313a0 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.275+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f233c1a1170 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.279+0000 7f233a000640 1 -- 172.21.15.114:0/1758955406 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f2330038250 con 0x7f233c07dea0 2024-09-19T06:41:42.344 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.282+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f23240391b0 msgr2=0x7f232403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.282+0000 7f2342c00640 1 --2- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f23240391b0 0x7f232403b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f23280098c0 tx=0x7f2328006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.282+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 msgr2=0x7f233c19d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.282+0000 7f2342c00640 1 --2- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c19d6c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f23300097f0 tx=0x7f23300327b0 comp rx=0 tx=0).stop 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.282+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 shutdown_connections 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.282+0000 7f2342c00640 1 --2- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f23240391b0 0x7f232403b670 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.282+0000 7f2342c00640 1 --2- 172.21.15.114:0/1758955406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f233c07dea0 0x7f233c19d6c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.282+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 >> 172.21.15.114:0/1758955406 conn(0x7f233c102c10 msgr2=0x7f233c1037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.283+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 shutdown_connections 2024-09-19T06:41:42.345 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:41.283+0000 7f2342c00640 1 -- 172.21.15.114:0/1758955406 wait complete. 2024-09-19T06:41:43.777 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.892+0000 7f8a84200640 1 Processor -- start 2024-09-19T06:41:43.777 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.892+0000 7f8a84200640 1 -- start start 2024-09-19T06:41:43.777 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.893+0000 7f8a84200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c109c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:43.777 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.893+0000 7f8a84200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c10a200 con 0x7f8a7c109830 2024-09-19T06:41:43.777 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.893+0000 7f8a81800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c109c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:43.777 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.893+0000 7f8a81800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c109c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60288/0 (socket says 172.21.15.114:60288) 2024-09-19T06:41:43.777 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.893+0000 7f8a81800640 1 -- 172.21.15.114:0/1992900143 learned_addr learned my addr 172.21.15.114:0/1992900143 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.894+0000 7f8a81800640 1 -- 172.21.15.114:0/1992900143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7c10a9e0 con 0x7f8a7c109830 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.895+0000 7f8a81800640 1 --2- 172.21.15.114:0/1992900143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c109c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009e80 tx=0x7f8a7002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c945f37af9e93c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.895+0000 7f8a7be00640 1 -- 172.21.15.114:0/1992900143 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a70036070 con 0x7f8a7c109830 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.895+0000 7f8a7be00640 1 -- 172.21.15.114:0/1992900143 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a700318e0 con 0x7f8a7c109830 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.895+0000 7f8a7be00640 1 -- 172.21.15.114:0/1992900143 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a70007a00 con 0x7f8a7c109830 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.896+0000 7f8a84200640 1 -- 172.21.15.114:0/1992900143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 msgr2=0x7f8a7c109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.896+0000 7f8a84200640 1 --2- 172.21.15.114:0/1992900143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c109c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009e80 tx=0x7f8a7002a7a0 comp rx=0 tx=0).stop 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.896+0000 7f8a84200640 1 -- 172.21.15.114:0/1992900143 shutdown_connections 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.896+0000 7f8a84200640 1 --2- 172.21.15.114:0/1992900143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c109c30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.896+0000 7f8a84200640 1 -- 172.21.15.114:0/1992900143 >> 172.21.15.114:0/1992900143 conn(0x7f8a7c07ced0 msgr2=0x7f8a7c07d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.896+0000 7f8a84200640 1 -- 172.21.15.114:0/1992900143 shutdown_connections 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.896+0000 7f8a84200640 1 -- 172.21.15.114:0/1992900143 wait complete. 2024-09-19T06:41:43.778 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.897+0000 7f8a84200640 1 Processor -- start 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.897+0000 7f8a84200640 1 -- start start 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.897+0000 7f8a84200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.897+0000 7f8a84200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7c10dd90 con 0x7f8a7c109830 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.898+0000 7f8a81800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.898+0000 7f8a81800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c19d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60296/0 (socket says 172.21.15.114:60296) 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.898+0000 7f8a81800640 1 -- 172.21.15.114:0/2454213355 learned_addr learned my addr 172.21.15.114:0/2454213355 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.898+0000 7f8a81800640 1 -- 172.21.15.114:0/2454213355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7c19dd50 con 0x7f8a7c109830 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.898+0000 7f8a81800640 1 --2- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c19d810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a70031cc0 tx=0x7f8a70031cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.899+0000 7f8a7a000640 1 -- 172.21.15.114:0/2454213355 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a70036050 con 0x7f8a7c109830 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.899+0000 7f8a7a000640 1 -- 172.21.15.114:0/2454213355 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a70042070 con 0x7f8a7c109830 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.899+0000 7f8a7a000640 1 -- 172.21.15.114:0/2454213355 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a7003d8e0 con 0x7f8a7c109830 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.899+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7c19dfe0 con 0x7f8a7c109830 2024-09-19T06:41:43.779 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.899+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7c1a0cd0 con 0x7f8a7c109830 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.900+0000 7f8a7a000640 1 -- 172.21.15.114:0/2454213355 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f8a7003db90 con 0x7f8a7c109830 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.900+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a7c109c30 con 0x7f8a7c109830 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.901+0000 7f8a7a000640 1 --2- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8a64039200 0x7f8a6403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.901+0000 7f8a7a000640 1 -- 172.21.15.114:0/2454213355 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8a7006d740 con 0x7f8a7c109830 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.901+0000 7f8a80e00640 1 --2- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8a64039200 0x7f8a6403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.902+0000 7f8a80e00640 1 --2- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8a64039200 0x7f8a6403b6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8a680098c0 tx=0x7f8a68006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:42.905+0000 7f8a7a000640 1 -- 172.21.15.114:0/2454213355 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8a7006d090 con 0x7f8a7c109830 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.090+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f8a7c1a1330 con 0x7f8a7c109830 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.092+0000 7f8a7a000640 1 -- 172.21.15.114:0/2454213355 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f8a7003bd30 con 0x7f8a7c109830 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8a64039200 msgr2=0x7f8a6403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 --2- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8a64039200 0x7f8a6403b6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8a680098c0 tx=0x7f8a68006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 msgr2=0x7f8a7c19d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:43.780 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 --2- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c19d810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a70031cc0 tx=0x7f8a70031cf0 comp rx=0 tx=0).stop 2024-09-19T06:41:43.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 shutdown_connections 2024-09-19T06:41:43.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 --2- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8a64039200 0x7f8a6403b6c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:43.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 --2- 172.21.15.114:0/2454213355 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7c109830 0x7f8a7c19d810 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:43.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 >> 172.21.15.114:0/2454213355 conn(0x7f8a7c07ced0 msgr2=0x7f8a7c1074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:43.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 shutdown_connections 2024-09-19T06:41:43.781 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:43.095+0000 7f8a84200640 1 -- 172.21.15.114:0/2454213355 wait complete. 2024-09-19T06:41:43.781 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling the dashboard module... 2024-09-19T06:41:44.320 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:44 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2454213355' entity='client.admin' 2024-09-19T06:41:44.320 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:44 smithi114 ceph-mon[30820]: from='mgr.14118 172.21.15.114:0/154626490' entity='mgr.x' 2024-09-19T06:41:44.320 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:44 smithi114 ceph-mon[30820]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:41:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:45 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/781278800' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-19T06:41:45.462 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: ignoring --setuser ceph since I am not root 2024-09-19T06:41:45.462 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: ignoring --setgroup ceph since I am not root 2024-09-19T06:41:45.462 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:45.338+0000 7fe986a36040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T06:41:45.462 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:45.410+0000 7fe986a36040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T06:41:45.736 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:45.599+0000 7fe986a36040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T06:41:45.736 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:45.680+0000 7fe986a36040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T06:41:46.022 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:45.855+0000 7fe986a36040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:41:46.022 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:46.020+0000 7fe986a36040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T06:41:46.227 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.329+0000 7f8ff7e00640 1 Processor -- start 2024-09-19T06:41:46.227 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.329+0000 7f8ff7e00640 1 -- start start 2024-09-19T06:41:46.227 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.329+0000 7f8ff7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:46.227 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.329+0000 7f8ff7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff807c8b0 con 0x7f8ff807de80 2024-09-19T06:41:46.227 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.330+0000 7f8ff6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:46.227 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.330+0000 7f8ff6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60302/0 (socket says 172.21.15.114:60302) 2024-09-19T06:41:46.227 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.330+0000 7f8ff6a00640 1 -- 172.21.15.114:0/3485368908 learned_addr learned my addr 172.21.15.114:0/3485368908 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.330+0000 7f8ff6a00640 1 -- 172.21.15.114:0/3485368908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff807ca30 con 0x7f8ff807de80 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.331+0000 7f8ff6a00640 1 --2- 172.21.15.114:0/3485368908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff807c2e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009820 tx=0x7f8fe802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a315851927701a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.332+0000 7f8ff5600640 1 -- 172.21.15.114:0/3485368908 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe8037070 con 0x7f8ff807de80 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.332+0000 7f8ff5600640 1 -- 172.21.15.114:0/3485368908 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe8032920 con 0x7f8ff807de80 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.332+0000 7f8ff5600640 1 -- 172.21.15.114:0/3485368908 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe8030810 con 0x7f8ff807de80 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.332+0000 7f8ff7e00640 1 -- 172.21.15.114:0/3485368908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 msgr2=0x7f8ff807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.332+0000 7f8ff7e00640 1 --2- 172.21.15.114:0/3485368908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff807c2e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009820 tx=0x7f8fe802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.332+0000 7f8ff7e00640 1 -- 172.21.15.114:0/3485368908 shutdown_connections 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.332+0000 7f8ff7e00640 1 --2- 172.21.15.114:0/3485368908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff807c2e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:46.228 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.332+0000 7f8ff7e00640 1 -- 172.21.15.114:0/3485368908 >> 172.21.15.114:0/3485368908 conn(0x7f8ff8102c10 msgr2=0x7f8ff8105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.333+0000 7f8ff7e00640 1 -- 172.21.15.114:0/3485368908 shutdown_connections 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.333+0000 7f8ff7e00640 1 -- 172.21.15.114:0/3485368908 wait complete. 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.334+0000 7f8ff7e00640 1 Processor -- start 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.334+0000 7f8ff7e00640 1 -- start start 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.334+0000 7f8ff7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff819d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.334+0000 7f8ff7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff810dad0 con 0x7f8ff807de80 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.334+0000 7f8ff6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff819d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.334+0000 7f8ff6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff819d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60312/0 (socket says 172.21.15.114:60312) 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.334+0000 7f8ff6a00640 1 -- 172.21.15.114:0/781278800 learned_addr learned my addr 172.21.15.114:0/781278800 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.335+0000 7f8ff6a00640 1 -- 172.21.15.114:0/781278800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff819dc20 con 0x7f8ff807de80 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.335+0000 7f8ff6a00640 1 --2- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff819d6e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8030420 tx=0x7f8fe8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.335+0000 7f8fef400640 1 -- 172.21.15.114:0/781278800 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe8037040 con 0x7f8ff807de80 2024-09-19T06:41:46.229 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.335+0000 7f8fef400640 1 -- 172.21.15.114:0/781278800 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe8040070 con 0x7f8ff807de80 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.335+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff819deb0 con 0x7f8ff807de80 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.335+0000 7f8fef400640 1 -- 172.21.15.114:0/781278800 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe803a6e0 con 0x7f8ff807de80 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.335+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a0ba0 con 0x7f8ff807de80 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.336+0000 7f8fef400640 1 -- 172.21.15.114:0/781278800 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f8fe803a990 con 0x7f8ff807de80 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.336+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff807c360 con 0x7f8ff807de80 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.337+0000 7f8fef400640 1 --2- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8fdc039200 0x7f8fdc03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.337+0000 7f8fef400640 1 -- 172.21.15.114:0/781278800 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8fe806c960 con 0x7f8ff807de80 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.337+0000 7f8ff6000640 1 --2- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8fdc039200 0x7f8fdc03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.338+0000 7f8ff6000640 1 --2- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8fdc039200 0x7f8fdc03b6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40098c0 tx=0x7f8fe4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.342+0000 7f8fef400640 1 -- 172.21.15.114:0/781278800 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8fe8044c70 con 0x7f8ff807de80 2024-09-19T06:41:46.230 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:44.586+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f8ff81a1190 con 0x7f8ff807de80 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.100+0000 7f8fef400640 1 -- 172.21.15.114:0/781278800 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f8fe8042e30 con 0x7f8ff807de80 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.102+0000 7f8fef400640 1 -- 172.21.15.114:0/781278800 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f8fe803ad80 con 0x7f8ff807de80 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.103+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8fdc039200 msgr2=0x7f8fdc03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.103+0000 7f8ff7e00640 1 --2- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8fdc039200 0x7f8fdc03b6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8fe40098c0 tx=0x7f8fe4006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.104+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 msgr2=0x7f8ff819d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.104+0000 7f8ff7e00640 1 --2- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff819d6e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8030420 tx=0x7f8fe8032670 comp rx=0 tx=0).stop 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.104+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 shutdown_connections 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.104+0000 7f8ff7e00640 1 --2- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f8fdc039200 0x7f8fdc03b6c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.104+0000 7f8ff7e00640 1 --2- 172.21.15.114:0/781278800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8ff807de80 0x7f8ff819d6e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.104+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 >> 172.21.15.114:0/781278800 conn(0x7f8ff8102c10 msgr2=0x7f8ff8103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.104+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 shutdown_connections 2024-09-19T06:41:46.231 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:45.104+0000 7f8ff7e00640 1 -- 172.21.15.114:0/781278800 wait complete. 2024-09-19T06:41:46.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:46 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/781278800' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-19T06:41:46.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:46 smithi114 ceph-mon[30820]: mgrmap e9: x(active, since 21s) 2024-09-19T06:41:46.644 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:46.399+0000 7fe986a36040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T06:41:46.644 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:46.470+0000 7fe986a36040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T06:41:46.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:46.691+0000 7fe986a36040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T06:41:47.335 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:47 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2911936341' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.750+0000 7fe1ce400640 1 Processor -- start 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.750+0000 7fe1ce400640 1 -- start start 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.750+0000 7fe1ce400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.750+0000 7fe1ce400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c807c800 con 0x7fe1c807de60 2024-09-19T06:41:47.656 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.751+0000 7fe1cd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.751+0000 7fe1cd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60340/0 (socket says 172.21.15.114:60340) 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.751+0000 7fe1cd000640 1 -- 172.21.15.114:0/3057153281 learned_addr learned my addr 172.21.15.114:0/3057153281 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.751+0000 7fe1cd000640 1 -- 172.21.15.114:0/3057153281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c807c980 con 0x7fe1c807de60 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.752+0000 7fe1cd000640 1 --2- 172.21.15.114:0/3057153281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c807c2c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009820 tx=0x7fe1bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be19b54a9b3698de server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.753+0000 7fe1c7400640 1 -- 172.21.15.114:0/3057153281 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc037070 con 0x7fe1c807de60 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.753+0000 7fe1c7400640 1 -- 172.21.15.114:0/3057153281 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1bc032920 con 0x7fe1c807de60 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.753+0000 7fe1c7400640 1 -- 172.21.15.114:0/3057153281 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc030810 con 0x7fe1c807de60 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.754+0000 7fe1ce400640 1 -- 172.21.15.114:0/3057153281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 msgr2=0x7fe1c807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.754+0000 7fe1ce400640 1 --2- 172.21.15.114:0/3057153281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c807c2c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009820 tx=0x7fe1bc02a270 comp rx=0 tx=0).stop 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.754+0000 7fe1ce400640 1 -- 172.21.15.114:0/3057153281 shutdown_connections 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.754+0000 7fe1ce400640 1 --2- 172.21.15.114:0/3057153281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c807c2c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.754+0000 7fe1ce400640 1 -- 172.21.15.114:0/3057153281 >> 172.21.15.114:0/3057153281 conn(0x7fe1c8102c10 msgr2=0x7fe1c8105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:47.657 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.754+0000 7fe1ce400640 1 -- 172.21.15.114:0/3057153281 shutdown_connections 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.754+0000 7fe1ce400640 1 -- 172.21.15.114:0/3057153281 wait complete. 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.755+0000 7fe1ce400640 1 Processor -- start 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.755+0000 7fe1ce400640 1 -- start start 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.755+0000 7fe1ce400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c819d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.755+0000 7fe1ce400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c810da90 con 0x7fe1c807de60 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.755+0000 7fe1cd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c819d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.756+0000 7fe1cd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c819d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60342/0 (socket says 172.21.15.114:60342) 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.756+0000 7fe1cd000640 1 -- 172.21.15.114:0/2911936341 learned_addr learned my addr 172.21.15.114:0/2911936341 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.756+0000 7fe1cd000640 1 -- 172.21.15.114:0/2911936341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c819dc00 con 0x7fe1c807de60 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.756+0000 7fe1cd000640 1 --2- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c819d6c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0097f0 tx=0x7fe1bc0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.756+0000 7fe1c5600640 1 -- 172.21.15.114:0/2911936341 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc037040 con 0x7fe1c807de60 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.756+0000 7fe1c5600640 1 -- 172.21.15.114:0/2911936341 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1bc030c40 con 0x7fe1c807de60 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.756+0000 7fe1c5600640 1 -- 172.21.15.114:0/2911936341 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1bc03a5d0 con 0x7fe1c807de60 2024-09-19T06:41:47.658 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.756+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c819de90 con 0x7fe1c807de60 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.757+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c81a0b80 con 0x7fe1c807de60 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.758+0000 7fe1c5600640 1 -- 172.21.15.114:0/2911936341 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fe1bc03a7c0 con 0x7fe1c807de60 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.758+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1c807c340 con 0x7fe1c807de60 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.758+0000 7fe1c5600640 1 --2- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe1b0039260 0x7fe1b003b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.758+0000 7fe1c5600640 1 -- 172.21.15.114:0/2911936341 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe1bc06c7a0 con 0x7fe1c807de60 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.758+0000 7fe1c7e00640 1 -- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe1b0039260 msgr2=0x7fe1b003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/3886387391 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.758+0000 7fe1c7e00640 1 --2- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe1b0039260 0x7fe1b003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.763+0000 7fe1c5600640 1 -- 172.21.15.114:0/2911936341 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe1bc03b3f0 con 0x7fe1c807de60 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.959+0000 7fe1c7e00640 1 -- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe1b0039260 msgr2=0x7fe1b003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/3886387391 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:46.959+0000 7fe1c7e00640 1 --2- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe1b0039260 0x7fe1b003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.003+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe1c81a1170 con 0x7fe1c807de60 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.004+0000 7fe1c5600640 1 -- 172.21.15.114:0/2911936341 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fe1bc03aab0 con 0x7fe1c807de60 2024-09-19T06:41:47.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe1b0039260 msgr2=0x7fe1b003b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 --2- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe1b0039260 0x7fe1b003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 msgr2=0x7fe1c819d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 --2- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c819d6c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0097f0 tx=0x7fe1bc0327b0 comp rx=0 tx=0).stop 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 shutdown_connections 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 --2- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7fe1b0039260 0x7fe1b003b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 --2- 172.21.15.114:0/2911936341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1c807de60 0x7fe1c819d6c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 >> 172.21.15.114:0/2911936341 conn(0x7fe1c8102c10 msgr2=0x7fe1c81037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 shutdown_connections 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:47.006+0000 7fe1ce400640 1 -- 172.21.15.114:0/2911936341 wait complete. 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-09-19T06:41:47.660 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 9... 2024-09-19T06:41:47.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:47.859+0000 7fe986a36040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T06:41:48.375 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:47.939+0000 7fe986a36040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T06:41:49.147 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:48.851+0000 7fe986a36040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T06:41:49.147 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:48.984+0000 7fe986a36040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T06:41:49.147 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.145+0000 7fe986a36040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:41:49.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.217+0000 7fe986a36040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T06:41:49.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.384+0000 7fe986a36040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T06:41:49.792 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.465+0000 7fe986a36040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T06:41:49.792 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.534+0000 7fe986a36040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T06:41:49.792 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.712+0000 7fe986a36040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T06:41:49.792 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.787+0000 7fe986a36040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T06:41:50.099 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.866+0000 7fe986a36040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T06:41:50.099 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:49.960+0000 7fe986a36040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T06:41:50.099 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:50.029+0000 7fe986a36040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T06:41:50.099 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:50.097+0000 7fe986a36040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T06:41:51.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:50.991+0000 7fe986a36040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: Active manager daemon x restarted 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: Activating manager daemon x 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: osdmap e3: 0 total, 0 up, 0 in 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: mgrmap e10: x(active, starting, since 0.0038629s) 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: Manager daemon x is now available 2024-09-19T06:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:41:52.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-19T06:41:52.292 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-19T06:41:52.293 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-mon[30820]: mgrmap e11: x(active, since 1.00835s) 2024-09-19T06:41:52.548 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:52.290+0000 7fe8e3e00640 -1 log_channel(cephadm) log [ERR] : [19/Sep/2024:06:41:52] ENGINE Error in HTTPServer.serve 2024-09-19T06:41:52.548 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T06:41:52.548 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T06:41:52.548 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: self._connections.run(self.expiration_interval) 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: self._run(expiration_interval) 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: s = self.context.wrap_socket( 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return self.sslsocket_class._create( 2024-09-19T06:41:52.549 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T06:41:52.550 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: self.do_handshake() 2024-09-19T06:41:52.550 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T06:41:52.550 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: self._sslobj.do_handshake() 2024-09-19T06:41:52.550 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T06:41:52.550 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:41:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:41:53.164 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:53 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:52] ENGINE Bus STARTING 2024-09-19T06:41:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-09-19T06:41:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-19T06:41:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-19T06:41:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-09-19T06:41:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.164+0000 7f3118600640 1 Processor -- start 2024-09-19T06:41:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.164+0000 7f3118600640 1 -- start start 2024-09-19T06:41:53.177 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.165+0000 7f3118600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f3110109c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.165+0000 7f3118600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f311010a200 con 0x7f3110109830 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.165+0000 7f3115c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f3110109c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.165+0000 7f3115c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f3110109c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46866/0 (socket says 172.21.15.114:46866) 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.165+0000 7f3115c00640 1 -- 172.21.15.114:0/3136558013 learned_addr learned my addr 172.21.15.114:0/3136558013 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.166+0000 7f3115c00640 1 -- 172.21.15.114:0/3136558013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311010aa40 con 0x7f3110109830 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.167+0000 7f3115c00640 1 --2- 172.21.15.114:0/3136558013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f3110109c30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3104009820 tx=0x7f310402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d51ba19ebf4ece server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.167+0000 7f310fe00640 1 -- 172.21.15.114:0/3136558013 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3104037070 con 0x7f3110109830 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.167+0000 7f310fe00640 1 -- 172.21.15.114:0/3136558013 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3104032920 con 0x7f3110109830 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.167+0000 7f310fe00640 1 -- 172.21.15.114:0/3136558013 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3104030810 con 0x7f3110109830 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.168+0000 7f3118600640 1 -- 172.21.15.114:0/3136558013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 msgr2=0x7f3110109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.168+0000 7f3118600640 1 --2- 172.21.15.114:0/3136558013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f3110109c30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3104009820 tx=0x7f310402a270 comp rx=0 tx=0).stop 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.168+0000 7f3118600640 1 -- 172.21.15.114:0/3136558013 shutdown_connections 2024-09-19T06:41:53.178 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.168+0000 7f3118600640 1 --2- 172.21.15.114:0/3136558013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f3110109c30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.168+0000 7f3118600640 1 -- 172.21.15.114:0/3136558013 >> 172.21.15.114:0/3136558013 conn(0x7f311007ce40 msgr2=0x7f311007d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.168+0000 7f3118600640 1 -- 172.21.15.114:0/3136558013 shutdown_connections 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.168+0000 7f3118600640 1 -- 172.21.15.114:0/3136558013 wait complete. 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.169+0000 7f3118600640 1 Processor -- start 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.169+0000 7f3118600640 1 -- start start 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.169+0000 7f3118600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f311019d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.169+0000 7f3118600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f311010ddf0 con 0x7f3110109830 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.169+0000 7f3115c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f311019d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.170+0000 7f3115c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f311019d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46870/0 (socket says 172.21.15.114:46870) 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.170+0000 7f3115c00640 1 -- 172.21.15.114:0/4190676141 learned_addr learned my addr 172.21.15.114:0/4190676141 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.170+0000 7f3115c00640 1 -- 172.21.15.114:0/4190676141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311019de20 con 0x7f3110109830 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.170+0000 7f3115c00640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f311019d8e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f31040097f0 tx=0x7f31040327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:53.179 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.170+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3104037040 con 0x7f3110109830 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.171+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3104030c40 con 0x7f3110109830 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.171+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311019e0b0 con 0x7f3110109830 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.171+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f310403a5b0 con 0x7f3110109830 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.171+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31101a0da0 con 0x7f3110109830 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.172+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f310403b3f0 con 0x7f3110109830 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.173+0000 7f310e000640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 0x7f30f803b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.173+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 --> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f30f803be30 con 0x7f30f8039260 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.173+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f310406c750 con 0x7f3110109830 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.173+0000 7f3115200640 1 -- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 msgr2=0x7f30f803b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/3886387391 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.173+0000 7f3115200640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 0x7f30f803b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.374+0000 7f3115200640 1 -- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 msgr2=0x7f30f803b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/3886387391 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.374+0000 7f3115200640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 0x7f30f803b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-19T06:41:53.180 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.775+0000 7f3115200640 1 -- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 msgr2=0x7f30f803b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/3886387391 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:48.775+0000 7f3115200640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 0x7f30f803b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:49.576+0000 7f3115200640 1 -- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 msgr2=0x7f30f803b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/3886387391 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:49.576+0000 7f3115200640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 0x7f30f803b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:50.996+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mgrmap(e 10) ==== 46241+0+0 (secure 0 0 0) 0x7f3104041d30 con 0x7f3110109830 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:50.996+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 msgr2=0x7f30f803b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:50.996+0000 7f310e000640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 0x7f30f803b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:51.999+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7f310403ab20 con 0x7f3110109830 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:51.999+0000 7f310e000640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f30f803cd20 0x7f30f803f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:51.999+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f30f803be30 con 0x7f30f803cd20 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.000+0000 7f3115200640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f30f803cd20 0x7f30f803f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.001+0000 7f3115200640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f30f803cd20 0x7f30f803f110 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f30fc003980 tx=0x7f30fc0092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.002+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f30f803be30 con 0x7f30f803cd20 2024-09-19T06:41:53.181 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.015+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3110109c30 con 0x7f30f803cd20 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.015+0000 7f310e000640 1 -- 172.21.15.114:0/4190676141 <== mgr.14150 v2:172.21.15.114:6800/3586930729 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f3110109c30 con 0x7f30f803cd20 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f30f803cd20 msgr2=0x7f30f803f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f30f803cd20 0x7f30f803f110 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f30fc003980 tx=0x7f30fc0092b0 comp rx=0 tx=0).stop 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 msgr2=0x7f311019d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f311019d8e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f31040097f0 tx=0x7f31040327b0 comp rx=0 tx=0).stop 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 shutdown_connections 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f30f803cd20 0x7f30f803f110 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:6800/3886387391,v1:172.21.15.114:6801/3886387391] conn(0x7f30f8039260 0x7f30f803b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 --2- 172.21.15.114:0/4190676141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3110109830 0x7f311019d8e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 >> 172.21.15.114:0/4190676141 conn(0x7f311007ce40 msgr2=0x7f31101075b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 shutdown_connections 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:52.016+0000 7f3118600640 1 -- 172.21.15.114:0/4190676141 wait complete. 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 9 is available 2024-09-19T06:41:53.182 INFO:teuthology.orchestra.run.smithi114.stdout:Generating a dashboard self-signed certificate... 2024-09-19T06:41:54.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:54 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:52] ENGINE Serving on https://172.21.15.114:7150 2024-09-19T06:41:54.376 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:54 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:52] ENGINE Error in HTTPServer.serve 2024-09-19T06:41:54.376 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T06:41:54.376 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-19T06:41:54.376 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._connections.run(self.expiration_interval) 2024-09-19T06:41:54.376 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-19T06:41:54.376 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._run(expiration_interval) 2024-09-19T06:41:54.376 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: s = self.context.wrap_socket( 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: return self.sslsocket_class._create( 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-19T06:41:54.377 INFO:journalctl@ceph.mon.a.smithi114.stdout: self.do_handshake() 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout: self._sslobj.do_handshake() 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:54 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:52] ENGINE Serving on http://172.21.15.114:8765 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:54 smithi114 ceph-mon[30820]: [19/Sep/2024:06:41:52] ENGINE Bus STARTED 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:54 smithi114 ceph-mon[30820]: mgrmap e12: x(active, since 2s) 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:41:54.378 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:41:54.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-19T06:41:54.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.756+0000 7efd0ee00640 1 Processor -- start 2024-09-19T06:41:54.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.756+0000 7efd0ee00640 1 -- start start 2024-09-19T06:41:54.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.756+0000 7efd0ee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd08105630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:54.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.756+0000 7efd0ee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd08105c00 con 0x7efd08105230 2024-09-19T06:41:54.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.757+0000 7efd07e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd08105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:54.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.757+0000 7efd07e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd08105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46934/0 (socket says 172.21.15.114:46934) 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.757+0000 7efd07e00640 1 -- 172.21.15.114:0/2697564791 learned_addr learned my addr 172.21.15.114:0/2697564791 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.757+0000 7efd07e00640 1 -- 172.21.15.114:0/2697564791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd08105d80 con 0x7efd08105230 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.758+0000 7efd07e00640 1 --2- 172.21.15.114:0/2697564791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd08105630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009820 tx=0x7efcf802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18adf8c428bf55c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.758+0000 7efd06a00640 1 -- 172.21.15.114:0/2697564791 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf8037070 con 0x7efd08105230 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.758+0000 7efd06a00640 1 -- 172.21.15.114:0/2697564791 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcf8032920 con 0x7efd08105230 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.758+0000 7efd06a00640 1 -- 172.21.15.114:0/2697564791 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf8030810 con 0x7efd08105230 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.759+0000 7efd0ee00640 1 -- 172.21.15.114:0/2697564791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 msgr2=0x7efd08105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.759+0000 7efd0ee00640 1 --2- 172.21.15.114:0/2697564791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd08105630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009820 tx=0x7efcf802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.759+0000 7efd0ee00640 1 -- 172.21.15.114:0/2697564791 shutdown_connections 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.759+0000 7efd0ee00640 1 --2- 172.21.15.114:0/2697564791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd08105630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.759+0000 7efd0ee00640 1 -- 172.21.15.114:0/2697564791 >> 172.21.15.114:0/2697564791 conn(0x7efd08100fa0 msgr2=0x7efd08103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.759+0000 7efd0ee00640 1 -- 172.21.15.114:0/2697564791 shutdown_connections 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.759+0000 7efd0ee00640 1 -- 172.21.15.114:0/2697564791 wait complete. 2024-09-19T06:41:54.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.760+0000 7efd0ee00640 1 Processor -- start 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.760+0000 7efd0ee00640 1 -- start start 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.761+0000 7efd0ee00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd0819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.761+0000 7efd0ee00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd0810b990 con 0x7efd08105230 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.761+0000 7efd07e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd0819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.761+0000 7efd07e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd0819d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46942/0 (socket says 172.21.15.114:46942) 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.761+0000 7efd07e00640 1 -- 172.21.15.114:0/966146631 learned_addr learned my addr 172.21.15.114:0/966146631 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.761+0000 7efd07e00640 1 -- 172.21.15.114:0/966146631 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0819de70 con 0x7efd08105230 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.761+0000 7efd07e00640 1 --2- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd0819d930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efcf80097f0 tx=0x7efcf80327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.762+0000 7efd04c00640 1 -- 172.21.15.114:0/966146631 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf8037040 con 0x7efd08105230 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.762+0000 7efd04c00640 1 -- 172.21.15.114:0/966146631 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcf8030c40 con 0x7efd08105230 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.762+0000 7efd04c00640 1 -- 172.21.15.114:0/966146631 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcf803a5b0 con 0x7efd08105230 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.762+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0819e100 con 0x7efd08105230 2024-09-19T06:41:54.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.762+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd081a0df0 con 0x7efd08105230 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.763+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd081056b0 con 0x7efd08105230 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.764+0000 7efd04c00640 1 -- 172.21.15.114:0/966146631 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7efcf803a7a0 con 0x7efd08105230 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.764+0000 7efd04c00640 1 --2- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7efcec039260 0x7efcec03b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.764+0000 7efd04c00640 1 -- 172.21.15.114:0/966146631 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efcf806c890 con 0x7efd08105230 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.764+0000 7efd07400640 1 --2- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7efcec039260 0x7efcec03b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.765+0000 7efd07400640 1 --2- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7efcec039260 0x7efcec03b720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efcf40098c0 tx=0x7efcf4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.770+0000 7efd04c00640 1 -- 172.21.15.114:0/966146631 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7efcf80371e0 con 0x7efd08105230 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:53.956+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7efd081022e0 con 0x7efcec039260 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.001+0000 7efd04c00640 1 -- 172.21.15.114:0/966146631 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7efd081022e0 con 0x7efcec039260 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7efcec039260 msgr2=0x7efcec03b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:54.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 --2- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7efcec039260 0x7efcec03b720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efcf40098c0 tx=0x7efcf4006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 msgr2=0x7efd0819d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 --2- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd0819d930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efcf80097f0 tx=0x7efcf80327b0 comp rx=0 tx=0).stop 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 shutdown_connections 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 --2- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7efcec039260 0x7efcec03b720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 --2- 172.21.15.114:0/966146631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd08105230 0x7efd0819d930 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 >> 172.21.15.114:0/966146631 conn(0x7efd08100fa0 msgr2=0x7efd08101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.003+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 shutdown_connections 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:54.004+0000 7efd0ee00640 1 -- 172.21.15.114:0/966146631 wait complete. 2024-09-19T06:41:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial admin user... 2024-09-19T06:41:55.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:55 smithi114 ceph-mon[30820]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:55.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:41:56.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$U.3vDrQmxpFGn4nnDFvb2eLV0bc3KiG76fdaFv56IaZoqnVhjx7r2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726728115, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-19T06:41:56.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.207+0000 7fe947200640 1 Processor -- start 2024-09-19T06:41:56.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.207+0000 7fe947200640 1 -- start start 2024-09-19T06:41:56.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.208+0000 7fe947200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe948107840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:56.808 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.208+0000 7fe947200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe948107e10 con 0x7fe948107440 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.208+0000 7fe945e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe948107840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.208+0000 7fe945e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe948107840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46944/0 (socket says 172.21.15.114:46944) 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.208+0000 7fe945e00640 1 -- 172.21.15.114:0/1419088828 learned_addr learned my addr 172.21.15.114:0/1419088828 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.209+0000 7fe945e00640 1 -- 172.21.15.114:0/1419088828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe948108640 con 0x7fe948107440 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.210+0000 7fe945e00640 1 --2- 172.21.15.114:0/1419088828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe948107840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe938009820 tx=0x7fe93802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2985af5fe9637fa4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.210+0000 7fe944a00640 1 -- 172.21.15.114:0/1419088828 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe938037070 con 0x7fe948107440 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.210+0000 7fe944a00640 1 -- 172.21.15.114:0/1419088828 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe938032920 con 0x7fe948107440 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.210+0000 7fe944a00640 1 -- 172.21.15.114:0/1419088828 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe938030810 con 0x7fe948107440 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.211+0000 7fe947200640 1 -- 172.21.15.114:0/1419088828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 msgr2=0x7fe948107840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.211+0000 7fe947200640 1 --2- 172.21.15.114:0/1419088828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe948107840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe938009820 tx=0x7fe93802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.211+0000 7fe947200640 1 -- 172.21.15.114:0/1419088828 shutdown_connections 2024-09-19T06:41:56.809 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.211+0000 7fe947200640 1 --2- 172.21.15.114:0/1419088828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe948107840 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.211+0000 7fe947200640 1 -- 172.21.15.114:0/1419088828 >> 172.21.15.114:0/1419088828 conn(0x7fe948102c10 msgr2=0x7fe948105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.211+0000 7fe947200640 1 -- 172.21.15.114:0/1419088828 shutdown_connections 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.211+0000 7fe947200640 1 -- 172.21.15.114:0/1419088828 wait complete. 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.212+0000 7fe947200640 1 Processor -- start 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.212+0000 7fe947200640 1 -- start start 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.212+0000 7fe947200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe94819b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.212+0000 7fe947200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe94810b9f0 con 0x7fe948107440 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.213+0000 7fe945e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe94819b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.213+0000 7fe945e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe94819b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46948/0 (socket says 172.21.15.114:46948) 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.213+0000 7fe945e00640 1 -- 172.21.15.114:0/1273675669 learned_addr learned my addr 172.21.15.114:0/1273675669 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.213+0000 7fe945e00640 1 -- 172.21.15.114:0/1273675669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94819bab0 con 0x7fe948107440 2024-09-19T06:41:56.810 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.214+0000 7fe945e00640 1 --2- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe94819b570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe938030560 tx=0x7fe938032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.214+0000 7fe93ea00640 1 -- 172.21.15.114:0/1273675669 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe938037040 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.214+0000 7fe93ea00640 1 -- 172.21.15.114:0/1273675669 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe938040070 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.214+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe94819bd40 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.214+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe94819ea30 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.214+0000 7fe93ea00640 1 -- 172.21.15.114:0/1273675669 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe93803a6d0 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.215+0000 7fe93ea00640 1 -- 172.21.15.114:0/1273675669 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fe93803a940 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.216+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe94810bb70 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.216+0000 7fe93ea00640 1 --2- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe92c039200 0x7fe92c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.216+0000 7fe93ea00640 1 -- 172.21.15.114:0/1273675669 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe93806ca00 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.217+0000 7fe945400640 1 --2- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe92c039200 0x7fe92c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.217+0000 7fe945400640 1 --2- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe92c039200 0x7fe92c03b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe9340098c0 tx=0x7fe934006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.221+0000 7fe93ea00640 1 -- 172.21.15.114:0/1273675669 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe938041270 con 0x7fe948107440 2024-09-19T06:41:56.811 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.433+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fe9481078c0 con 0x7fe92c039200 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.659+0000 7fe93ea00640 1 -- 172.21.15.114:0/1273675669 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fe9481078c0 con 0x7fe92c039200 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe92c039200 msgr2=0x7fe92c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 --2- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe92c039200 0x7fe92c03b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe9340098c0 tx=0x7fe934006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 msgr2=0x7fe94819b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 --2- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe94819b570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe938030560 tx=0x7fe938032670 comp rx=0 tx=0).stop 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 shutdown_connections 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 --2- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe92c039200 0x7fe92c03b6c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 --2- 172.21.15.114:0/1273675669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe948107440 0x7fe94819b570 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 >> 172.21.15.114:0/1273675669 conn(0x7fe948102c10 msgr2=0x7fe948103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 shutdown_connections 2024-09-19T06:41:56.812 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:55.662+0000 7fe947200640 1 -- 172.21.15.114:0/1273675669 wait complete. 2024-09-19T06:41:56.813 INFO:teuthology.orchestra.run.smithi114.stdout:Fetching dashboard port number... 2024-09-19T06:41:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:56 smithi114 ceph-mon[30820]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:41:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:41:58.508 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 8443 2024-09-19T06:41:58.508 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.676+0000 7f7dfd400640 1 Processor -- start 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.676+0000 7f7dfd400640 1 -- start start 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.676+0000 7f7dfd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.676+0000 7f7dfd400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7df807c800 con 0x7f7df807de60 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.676+0000 7f7df7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.676+0000 7f7df7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36624/0 (socket says 172.21.15.114:36624) 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.676+0000 7f7df7e00640 1 -- 172.21.15.114:0/2372631293 learned_addr learned my addr 172.21.15.114:0/2372631293 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.677+0000 7f7df7e00640 1 -- 172.21.15.114:0/2372631293 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df807c980 con 0x7f7df807de60 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.678+0000 7f7df7e00640 1 --2- 172.21.15.114:0/2372631293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df807c2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009820 tx=0x7f7de802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=717f71b68608feeb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.678+0000 7f7df7400640 1 -- 172.21.15.114:0/2372631293 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7de8037070 con 0x7f7df807de60 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.678+0000 7f7df7400640 1 -- 172.21.15.114:0/2372631293 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de8032920 con 0x7f7df807de60 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.678+0000 7f7df7400640 1 -- 172.21.15.114:0/2372631293 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7de8030810 con 0x7f7df807de60 2024-09-19T06:41:58.509 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.679+0000 7f7dfd400640 1 -- 172.21.15.114:0/2372631293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 msgr2=0x7f7df807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.679+0000 7f7dfd400640 1 --2- 172.21.15.114:0/2372631293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df807c2c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7de8009820 tx=0x7f7de802a270 comp rx=0 tx=0).stop 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.679+0000 7f7dfd400640 1 -- 172.21.15.114:0/2372631293 shutdown_connections 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.679+0000 7f7dfd400640 1 --2- 172.21.15.114:0/2372631293 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df807c2c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.679+0000 7f7dfd400640 1 -- 172.21.15.114:0/2372631293 >> 172.21.15.114:0/2372631293 conn(0x7f7df8102c10 msgr2=0x7f7df8105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.679+0000 7f7dfd400640 1 -- 172.21.15.114:0/2372631293 shutdown_connections 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.679+0000 7f7dfd400640 1 -- 172.21.15.114:0/2372631293 wait complete. 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.680+0000 7f7dfd400640 1 Processor -- start 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.680+0000 7f7dfd400640 1 -- start start 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.680+0000 7f7dfd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df819d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.680+0000 7f7dfd400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7df810da90 con 0x7f7df807de60 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.680+0000 7f7df7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df819d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:58.510 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.680+0000 7f7df7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df819d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36638/0 (socket says 172.21.15.114:36638) 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.680+0000 7f7df7e00640 1 -- 172.21.15.114:0/1374179890 learned_addr learned my addr 172.21.15.114:0/1374179890 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.681+0000 7f7df7e00640 1 -- 172.21.15.114:0/1374179890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df819dc00 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.681+0000 7f7df7e00640 1 --2- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df819d6c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7de80097f0 tx=0x7f7de80327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.681+0000 7f7df5600640 1 -- 172.21.15.114:0/1374179890 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7de8037040 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.681+0000 7f7df5600640 1 -- 172.21.15.114:0/1374179890 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de8030c40 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.681+0000 7f7df5600640 1 -- 172.21.15.114:0/1374179890 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7de803a5d0 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.681+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df819de90 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.682+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df81a0b80 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.682+0000 7f7df5600640 1 -- 172.21.15.114:0/1374179890 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f7de803b3f0 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.682+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df807c340 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.683+0000 7f7df5600640 1 --2- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7ddc039300 0x7f7ddc03b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.683+0000 7f7df5600640 1 -- 172.21.15.114:0/1374179890 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7de806c9d0 con 0x7f7df807de60 2024-09-19T06:41:58.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.683+0000 7f7defe00640 1 --2- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7ddc039300 0x7f7ddc03b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.684+0000 7f7defe00640 1 --2- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7ddc039300 0x7f7ddc03b7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7de00098c0 tx=0x7f7de0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.688+0000 7f7df5600640 1 -- 172.21.15.114:0/1374179890 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7de8041510 con 0x7f7df807de60 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.861+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f7df81a1170 con 0x7f7df807de60 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.862+0000 7f7df5600640 1 -- 172.21.15.114:0/1374179890 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f7de80412b0 con 0x7f7df807de60 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7ddc039300 msgr2=0x7f7ddc03b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 --2- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7ddc039300 0x7f7ddc03b7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7de00098c0 tx=0x7f7de0006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 msgr2=0x7f7df819d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 --2- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df819d6c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7de80097f0 tx=0x7f7de80327b0 comp rx=0 tx=0).stop 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 shutdown_connections 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 --2- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7ddc039300 0x7f7ddc03b7c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 --2- 172.21.15.114:0/1374179890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df807de60 0x7f7df819d6c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 >> 172.21.15.114:0/1374179890 conn(0x7f7df8102c10 msgr2=0x7f7df81037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:58.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 shutdown_connections 2024-09-19T06:41:58.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:57.864+0000 7f7dfd400640 1 -- 172.21.15.114:0/1374179890 wait complete. 2024-09-19T06:41:58.523 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-19T06:41:58.524 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-09-19T06:41:58.537 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-19T06:41:58.537 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-09-19T06:41:58.537 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-09-19T06:41:58.537 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-19T06:41:58.539 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph Dashboard is now available at: 2024-09-19T06:41:58.539 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:58.539 INFO:teuthology.orchestra.run.smithi114.stdout: URL: https://smithi114.front.sepia.ceph.com:8443/ 2024-09-19T06:41:58.539 INFO:teuthology.orchestra.run.smithi114.stdout: User: admin 2024-09-19T06:41:58.540 INFO:teuthology.orchestra.run.smithi114.stdout: Password: 9jpcubec53 2024-09-19T06:41:58.540 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:58.540 INFO:teuthology.orchestra.run.smithi114.stdout:Saving cluster configuration to /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config directory 2024-09-19T06:41:58.772 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:58 smithi114 ceph-mon[30820]: mgrmap e13: x(active, since 6s) 2024-09-19T06:41:58.772 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:41:58 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1374179890' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.997+0000 7f79faa00640 1 Processor -- start 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.997+0000 7f79faa00640 1 -- start start 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.998+0000 7f79faa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f407c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.998+0000 7f79faa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79f407c840 con 0x7f79f407dd90 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.998+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f407c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.998+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f407c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36648/0 (socket says 172.21.15.114:36648) 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.998+0000 7f79f3e00640 1 -- 172.21.15.114:0/494259170 learned_addr learned my addr 172.21.15.114:0/494259170 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.998+0000 7f79f3e00640 1 -- 172.21.15.114:0/494259170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f407c9c0 con 0x7f79f407dd90 2024-09-19T06:41:59.964 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:58.999+0000 7f79f3e00640 1 --2- 172.21.15.114:0/494259170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f407c300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009820 tx=0x7f79e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3503545a0755fd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79f2a00640 1 -- 172.21.15.114:0/494259170 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79e4037070 con 0x7f79f407dd90 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79f2a00640 1 -- 172.21.15.114:0/494259170 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e4032920 con 0x7f79f407dd90 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79f2a00640 1 -- 172.21.15.114:0/494259170 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79e4030810 con 0x7f79f407dd90 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79faa00640 1 -- 172.21.15.114:0/494259170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 msgr2=0x7f79f407c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79faa00640 1 --2- 172.21.15.114:0/494259170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f407c300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009820 tx=0x7f79e402a270 comp rx=0 tx=0).stop 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79faa00640 1 -- 172.21.15.114:0/494259170 shutdown_connections 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79faa00640 1 --2- 172.21.15.114:0/494259170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f407c300 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79faa00640 1 -- 172.21.15.114:0/494259170 >> 172.21.15.114:0/494259170 conn(0x7f79f4102eb0 msgr2=0x7f79f41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79faa00640 1 -- 172.21.15.114:0/494259170 shutdown_connections 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.000+0000 7f79faa00640 1 -- 172.21.15.114:0/494259170 wait complete. 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.001+0000 7f79faa00640 1 Processor -- start 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.001+0000 7f79faa00640 1 -- start start 2024-09-19T06:41:59.965 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.002+0000 7f79faa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f419d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.002+0000 7f79faa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79f410dc00 con 0x7f79f407dd90 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.002+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f419d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.002+0000 7f79f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f419d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36664/0 (socket says 172.21.15.114:36664) 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.002+0000 7f79f3e00640 1 -- 172.21.15.114:0/1230144165 learned_addr learned my addr 172.21.15.114:0/1230144165 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.002+0000 7f79f3e00640 1 -- 172.21.15.114:0/1230144165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f419ddd0 con 0x7f79f407dd90 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.002+0000 7f79f3e00640 1 --2- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f419d890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79e40097f0 tx=0x7f79e40327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.003+0000 7f79f0c00640 1 -- 172.21.15.114:0/1230144165 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79e4037040 con 0x7f79f407dd90 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.003+0000 7f79f0c00640 1 -- 172.21.15.114:0/1230144165 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e4030c40 con 0x7f79f407dd90 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.003+0000 7f79f0c00640 1 -- 172.21.15.114:0/1230144165 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79e403a5d0 con 0x7f79f407dd90 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.003+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f419e060 con 0x7f79f407dd90 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.003+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f41a0d50 con 0x7f79f407dd90 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.004+0000 7f79f0c00640 1 -- 172.21.15.114:0/1230144165 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f79e403b3f0 con 0x7f79f407dd90 2024-09-19T06:41:59.966 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.004+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f407c380 con 0x7f79f407dd90 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.005+0000 7f79f0c00640 1 --2- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f79d8039300 0x7f79d803b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.005+0000 7f79f0c00640 1 -- 172.21.15.114:0/1230144165 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f79e406c9f0 con 0x7f79f407dd90 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.005+0000 7f79f3400640 1 --2- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f79d8039300 0x7f79d803b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.005+0000 7f79f3400640 1 --2- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f79d8039300 0x7f79d803b7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f79e00098c0 tx=0x7f79e0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.009+0000 7f79f0c00640 1 -- 172.21.15.114:0/1230144165 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f79e40412b0 con 0x7f79f407dd90 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.270+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f79f41a1340 con 0x7f79f407dd90 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.274+0000 7f79f0c00640 1 -- 172.21.15.114:0/1230144165 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f79e403b6e0 con 0x7f79f407dd90 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.276+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f79d8039300 msgr2=0x7f79d803b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.276+0000 7f79faa00640 1 --2- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f79d8039300 0x7f79d803b7c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f79e00098c0 tx=0x7f79e0006db0 comp rx=0 tx=0).stop 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.276+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 msgr2=0x7f79f419d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.276+0000 7f79faa00640 1 --2- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f419d890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79e40097f0 tx=0x7f79e40327b0 comp rx=0 tx=0).stop 2024-09-19T06:41:59.967 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.276+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 shutdown_connections 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.276+0000 7f79faa00640 1 --2- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f79d8039300 0x7f79d803b7c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.276+0000 7f79faa00640 1 --2- 172.21.15.114:0/1230144165 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79f407dd90 0x7f79f419d890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.276+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 >> 172.21.15.114:0/1230144165 conn(0x7f79f4102eb0 msgr2=0x7f79f4103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.277+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 shutdown_connections 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-09-19T06:41:59.277+0000 7f79faa00640 1 -- 172.21.15.114:0/1230144165 wait complete. 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid fface640-7651-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout:Or, if you are only running a single cluster on this host: 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-19T06:41:59.968 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout: ceph telemetry on 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout:For more information see: 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:41:59.969 INFO:teuthology.orchestra.run.smithi114.stdout:Bootstrap complete. 2024-09-19T06:42:00.019 INFO:tasks.cephadm:Fetching config... 2024-09-19T06:42:00.019 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:42:00.019 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-19T06:42:00.054 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-19T06:42:00.055 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:42:00.055 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-19T06:42:00.131 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-19T06:42:00.131 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:42:00.131 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-19T06:42:00.227 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-19T06:42:00.227 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:42:00.228 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-19T06:42:00.304 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-19T06:42:00.304 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDev5FMHV0qE+tC2R7ZXOwW5nSTR09slp886CpbnSr+sFzI8/sh5Fv1qCFoR0HwrBG07kc+su9t60r1nLtTkgbpkje3H217SXfu4+K9nPoPlfTzwOIn69xHNsI3/nD1OEQNcOQw99ewX2VSUgjsZert25Istz8dMLXODYXjkmQxTxJhUus4Z7XyJ5r14rgPjmyv52mKb4kA3Dlf+M8zPpXdHpRLk9XMgZ4Eic3Ho5nWtKk1yIPIQXaOK85HRFqLQ/0DkAx4YE3zatLbtv5XgDEXg0TbVAe9jWwrmVZ2PsjBRBrQuoKm0Z4lym2Ke7fUnWTPtqDbqk9jJpxO0Ojy24dfIACy/Lgo9XZV7hT3MWJPdwWr+ZQhaZnuBzn2lpcj8+a1RhffAGOY3pB+j6ygB8VD/u8MVrqob/biXb5UfG3zPFDLtbAkpKbFPusJ7cjX1b1Ncx7mV0170+jlmGGYhM2uDC1pgoAoqdQUN1vHJ6yomYGYqruU13Jxgits3mSgwHc= ceph-fface640-7651-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-19T06:42:00.427 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDev5FMHV0qE+tC2R7ZXOwW5nSTR09slp886CpbnSr+sFzI8/sh5Fv1qCFoR0HwrBG07kc+su9t60r1nLtTkgbpkje3H217SXfu4+K9nPoPlfTzwOIn69xHNsI3/nD1OEQNcOQw99ewX2VSUgjsZert25Istz8dMLXODYXjkmQxTxJhUus4Z7XyJ5r14rgPjmyv52mKb4kA3Dlf+M8zPpXdHpRLk9XMgZ4Eic3Ho5nWtKk1yIPIQXaOK85HRFqLQ/0DkAx4YE3zatLbtv5XgDEXg0TbVAe9jWwrmVZ2PsjBRBrQuoKm0Z4lym2Ke7fUnWTPtqDbqk9jJpxO0Ojy24dfIACy/Lgo9XZV7hT3MWJPdwWr+ZQhaZnuBzn2lpcj8+a1RhffAGOY3pB+j6ygB8VD/u8MVrqob/biXb5UfG3zPFDLtbAkpKbFPusJ7cjX1b1Ncx7mV0170+jlmGGYhM2uDC1pgoAoqdQUN1vHJ6yomYGYqruU13Jxgits3mSgwHc= ceph-fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:00.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDev5FMHV0qE+tC2R7ZXOwW5nSTR09slp886CpbnSr+sFzI8/sh5Fv1qCFoR0HwrBG07kc+su9t60r1nLtTkgbpkje3H217SXfu4+K9nPoPlfTzwOIn69xHNsI3/nD1OEQNcOQw99ewX2VSUgjsZert25Istz8dMLXODYXjkmQxTxJhUus4Z7XyJ5r14rgPjmyv52mKb4kA3Dlf+M8zPpXdHpRLk9XMgZ4Eic3Ho5nWtKk1yIPIQXaOK85HRFqLQ/0DkAx4YE3zatLbtv5XgDEXg0TbVAe9jWwrmVZ2PsjBRBrQuoKm0Z4lym2Ke7fUnWTPtqDbqk9jJpxO0Ojy24dfIACy/Lgo9XZV7hT3MWJPdwWr+ZQhaZnuBzn2lpcj8+a1RhffAGOY3pB+j6ygB8VD/u8MVrqob/biXb5UfG3zPFDLtbAkpKbFPusJ7cjX1b1Ncx7mV0170+jlmGGYhM2uDC1pgoAoqdQUN1vHJ6yomYGYqruU13Jxgits3mSgwHc= ceph-fface640-7651-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-19T06:42:00.494 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDev5FMHV0qE+tC2R7ZXOwW5nSTR09slp886CpbnSr+sFzI8/sh5Fv1qCFoR0HwrBG07kc+su9t60r1nLtTkgbpkje3H217SXfu4+K9nPoPlfTzwOIn69xHNsI3/nD1OEQNcOQw99ewX2VSUgjsZert25Istz8dMLXODYXjkmQxTxJhUus4Z7XyJ5r14rgPjmyv52mKb4kA3Dlf+M8zPpXdHpRLk9XMgZ4Eic3Ho5nWtKk1yIPIQXaOK85HRFqLQ/0DkAx4YE3zatLbtv5XgDEXg0TbVAe9jWwrmVZ2PsjBRBrQuoKm0Z4lym2Ke7fUnWTPtqDbqk9jJpxO0Ojy24dfIACy/Lgo9XZV7hT3MWJPdwWr+ZQhaZnuBzn2lpcj8+a1RhffAGOY3pB+j6ygB8VD/u8MVrqob/biXb5UfG3zPFDLtbAkpKbFPusJ7cjX1b1Ncx7mV0170+jlmGGYhM2uDC1pgoAoqdQUN1vHJ6yomYGYqruU13Jxgits3mSgwHc= ceph-fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:00.507 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDev5FMHV0qE+tC2R7ZXOwW5nSTR09slp886CpbnSr+sFzI8/sh5Fv1qCFoR0HwrBG07kc+su9t60r1nLtTkgbpkje3H217SXfu4+K9nPoPlfTzwOIn69xHNsI3/nD1OEQNcOQw99ewX2VSUgjsZert25Istz8dMLXODYXjkmQxTxJhUus4Z7XyJ5r14rgPjmyv52mKb4kA3Dlf+M8zPpXdHpRLk9XMgZ4Eic3Ho5nWtKk1yIPIQXaOK85HRFqLQ/0DkAx4YE3zatLbtv5XgDEXg0TbVAe9jWwrmVZ2PsjBRBrQuoKm0Z4lym2Ke7fUnWTPtqDbqk9jJpxO0Ojy24dfIACy/Lgo9XZV7hT3MWJPdwWr+ZQhaZnuBzn2lpcj8+a1RhffAGOY3pB+j6ygB8VD/u8MVrqob/biXb5UfG3zPFDLtbAkpKbFPusJ7cjX1b1Ncx7mV0170+jlmGGYhM2uDC1pgoAoqdQUN1vHJ6yomYGYqruU13Jxgits3mSgwHc= ceph-fface640-7651-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-19T06:42:00.548 INFO:teuthology.orchestra.run.smithi118.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDev5FMHV0qE+tC2R7ZXOwW5nSTR09slp886CpbnSr+sFzI8/sh5Fv1qCFoR0HwrBG07kc+su9t60r1nLtTkgbpkje3H217SXfu4+K9nPoPlfTzwOIn69xHNsI3/nD1OEQNcOQw99ewX2VSUgjsZert25Istz8dMLXODYXjkmQxTxJhUus4Z7XyJ5r14rgPjmyv52mKb4kA3Dlf+M8zPpXdHpRLk9XMgZ4Eic3Ho5nWtKk1yIPIQXaOK85HRFqLQ/0DkAx4YE3zatLbtv5XgDEXg0TbVAe9jWwrmVZ2PsjBRBrQuoKm0Z4lym2Ke7fUnWTPtqDbqk9jJpxO0Ojy24dfIACy/Lgo9XZV7hT3MWJPdwWr+ZQhaZnuBzn2lpcj8+a1RhffAGOY3pB+j6ygB8VD/u8MVrqob/biXb5UfG3zPFDLtbAkpKbFPusJ7cjX1b1Ncx7mV0170+jlmGGYhM2uDC1pgoAoqdQUN1vHJ6yomYGYqruU13Jxgits3mSgwHc= ceph-fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:00.557 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:00 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1230144165' entity='client.admin' 2024-09-19T06:42:00.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-19T06:42:00.927 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:01.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.543+0000 7fd094a00640 1 -- 172.21.15.114:0/3504849644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 msgr2=0x7fd090074600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:01.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.543+0000 7fd094a00640 1 --2- 172.21.15.114:0/3504849644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 0x7fd090074600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd080009880 tx=0x7fd08002a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:01.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.543+0000 7fd094a00640 1 -- 172.21.15.114:0/3504849644 shutdown_connections 2024-09-19T06:42:01.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.543+0000 7fd094a00640 1 --2- 172.21.15.114:0/3504849644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 0x7fd090074600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:01.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.543+0000 7fd094a00640 1 -- 172.21.15.114:0/3504849644 >> 172.21.15.114:0/3504849644 conn(0x7fd0900fcbd0 msgr2=0x7fd0900feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:01.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.543+0000 7fd094a00640 1 -- 172.21.15.114:0/3504849644 shutdown_connections 2024-09-19T06:42:01.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.543+0000 7fd094a00640 1 -- 172.21.15.114:0/3504849644 wait complete. 2024-09-19T06:42:01.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.544+0000 7fd094a00640 1 Processor -- start 2024-09-19T06:42:01.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.544+0000 7fd094a00640 1 -- start start 2024-09-19T06:42:01.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.544+0000 7fd094a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 0x7fd090198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:01.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.544+0000 7fd094a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd090109720 con 0x7fd0900761a0 2024-09-19T06:42:01.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.545+0000 7fd08f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 0x7fd090198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:01.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.545+0000 7fd08f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 0x7fd090198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36676/0 (socket says 172.21.15.114:36676) 2024-09-19T06:42:01.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.545+0000 7fd08f400640 1 -- 172.21.15.114:0/3076278153 learned_addr learned my addr 172.21.15.114:0/3076278153 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:01.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.545+0000 7fd08f400640 1 -- 172.21.15.114:0/3076278153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0901992d0 con 0x7fd0900761a0 2024-09-19T06:42:01.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.545+0000 7fd08f400640 1 --2- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 0x7fd090198d90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd080004f60 tx=0x7fd0800048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:01.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.546+0000 7fd087e00640 1 -- 172.21.15.114:0/3076278153 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd08003f070 con 0x7fd0900761a0 2024-09-19T06:42:01.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.546+0000 7fd087e00640 1 -- 172.21.15.114:0/3076278153 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd080003ad0 con 0x7fd0900761a0 2024-09-19T06:42:01.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.546+0000 7fd087e00640 1 -- 172.21.15.114:0/3076278153 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd080035030 con 0x7fd0900761a0 2024-09-19T06:42:01.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.546+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd090199560 con 0x7fd0900761a0 2024-09-19T06:42:01.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.546+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09019c210 con 0x7fd0900761a0 2024-09-19T06:42:01.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.547+0000 7fd087e00640 1 -- 172.21.15.114:0/3076278153 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7fd080042070 con 0x7fd0900761a0 2024-09-19T06:42:01.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.547+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd090109ea0 con 0x7fd0900761a0 2024-09-19T06:42:01.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.547+0000 7fd087e00640 1 --2- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd074039300 0x7fd07403b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:01.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.548+0000 7fd087e00640 1 -- 172.21.15.114:0/3076278153 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd08006d150 con 0x7fd0900761a0 2024-09-19T06:42:01.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.548+0000 7fd08ea00640 1 --2- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd074039300 0x7fd07403b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:01.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.549+0000 7fd08ea00640 1 --2- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd074039300 0x7fd07403b7c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0098c0 tx=0x7fd07c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:01.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.553+0000 7fd087e00640 1 -- 172.21.15.114:0/3076278153 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd08003a3c0 con 0x7fd0900761a0 2024-09-19T06:42:01.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.740+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fd09002d940 con 0x7fd0900761a0 2024-09-19T06:42:01.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.743+0000 7fd087e00640 1 -- 172.21.15.114:0/3076278153 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fd080030d90 con 0x7fd0900761a0 2024-09-19T06:42:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.745+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd074039300 msgr2=0x7fd07403b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.745+0000 7fd094a00640 1 --2- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd074039300 0x7fd07403b7c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0098c0 tx=0x7fd07c006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.745+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 msgr2=0x7fd090198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.745+0000 7fd094a00640 1 --2- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 0x7fd090198d90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd080004f60 tx=0x7fd0800048f0 comp rx=0 tx=0).stop 2024-09-19T06:42:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.745+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 shutdown_connections 2024-09-19T06:42:01.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.745+0000 7fd094a00640 1 --2- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd074039300 0x7fd07403b7c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.745+0000 7fd094a00640 1 --2- 172.21.15.114:0/3076278153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd0900761a0 0x7fd090198d90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.745+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 >> 172.21.15.114:0/3076278153 conn(0x7fd0900fcbd0 msgr2=0x7fd0900fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.746+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 shutdown_connections 2024-09-19T06:42:01.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:01.746+0000 7fd094a00640 1 -- 172.21.15.114:0/3076278153 wait complete. 2024-09-19T06:42:02.418 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-19T06:42:02.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-19T06:42:03.032 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:03.123 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:02 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3076278153' entity='client.admin' 2024-09-19T06:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.931+0000 7f488a200640 1 -- 172.21.15.114:0/3165209873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 msgr2=0x7f4884103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.931+0000 7f488a200640 1 --2- 172.21.15.114:0/3165209873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 0x7f4884103a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4878009c20 tx=0x7f487802aa90 comp rx=0 tx=0).stop 2024-09-19T06:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.931+0000 7f488a200640 1 -- 172.21.15.114:0/3165209873 shutdown_connections 2024-09-19T06:42:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.931+0000 7f488a200640 1 --2- 172.21.15.114:0/3165209873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 0x7f4884103a20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.931+0000 7f488a200640 1 -- 172.21.15.114:0/3165209873 >> 172.21.15.114:0/3165209873 conn(0x7f48840fedd0 msgr2=0x7f48841011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.931+0000 7f488a200640 1 -- 172.21.15.114:0/3165209873 shutdown_connections 2024-09-19T06:42:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.931+0000 7f488a200640 1 -- 172.21.15.114:0/3165209873 wait complete. 2024-09-19T06:42:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.932+0000 7f488a200640 1 Processor -- start 2024-09-19T06:42:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.932+0000 7f488a200640 1 -- start start 2024-09-19T06:42:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.932+0000 7f488a200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 0x7f4884198de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.932+0000 7f488a200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4884109730 con 0x7f4884103620 2024-09-19T06:42:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.932+0000 7f4888e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 0x7f4884198de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:03.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.932+0000 7f4888e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 0x7f4884198de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36690/0 (socket says 172.21.15.114:36690) 2024-09-19T06:42:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.932+0000 7f4888e00640 1 -- 172.21.15.114:0/793351910 learned_addr learned my addr 172.21.15.114:0/793351910 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.933+0000 7f4888e00640 1 -- 172.21.15.114:0/793351910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4884199320 con 0x7f4884103620 2024-09-19T06:42:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.933+0000 7f4888e00640 1 --2- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 0x7f4884198de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4878005cc0 tx=0x7f487802fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.933+0000 7f4881600640 1 -- 172.21.15.114:0/793351910 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4878047070 con 0x7f4884103620 2024-09-19T06:42:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.933+0000 7f4881600640 1 -- 172.21.15.114:0/793351910 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4878003e50 con 0x7f4884103620 2024-09-19T06:42:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.933+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48841995b0 con 0x7f4884103620 2024-09-19T06:42:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.933+0000 7f4881600640 1 -- 172.21.15.114:0/793351910 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4878033050 con 0x7f4884103620 2024-09-19T06:42:03.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.933+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488419c2a0 con 0x7f4884103620 2024-09-19T06:42:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.934+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4884109eb0 con 0x7f4884103620 2024-09-19T06:42:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.935+0000 7f4881600640 1 -- 172.21.15.114:0/793351910 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f487804a070 con 0x7f4884103620 2024-09-19T06:42:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.935+0000 7f4881600640 1 --2- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f486c0392a0 0x7f486c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.935+0000 7f4881600640 1 -- 172.21.15.114:0/793351910 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f487803e030 con 0x7f4884103620 2024-09-19T06:42:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.935+0000 7f4883e00640 1 --2- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f486c0392a0 0x7f486c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:03.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.936+0000 7f4883e00640 1 --2- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f486c0392a0 0x7f486c03b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f48700098c0 tx=0x7f4870006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:03.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:03.940+0000 7f4881600640 1 -- 172.21.15.114:0/793351910 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f48780373b0 con 0x7f4884103620 2024-09-19T06:42:04.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.129+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f488402d830 con 0x7f486c0392a0 2024-09-19T06:42:04.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:04.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:04.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:42:04.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:04.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:04.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:04.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.134+0000 7f4881600640 1 -- 172.21.15.114:0/793351910 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f488402d830 con 0x7f486c0392a0 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f486c0392a0 msgr2=0x7f486c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 --2- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f486c0392a0 0x7f486c03b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f48700098c0 tx=0x7f4870006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 msgr2=0x7f4884198de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 --2- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 0x7f4884198de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4878005cc0 tx=0x7f487802fc10 comp rx=0 tx=0).stop 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 shutdown_connections 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 --2- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f486c0392a0 0x7f486c03b760 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 --2- 172.21.15.114:0/793351910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4884103620 0x7f4884198de0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 >> 172.21.15.114:0/793351910 conn(0x7f48840fedd0 msgr2=0x7f48840ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.136+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 shutdown_connections 2024-09-19T06:42:04.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:04.137+0000 7f488a200640 1 -- 172.21.15.114:0/793351910 wait complete. 2024-09-19T06:42:04.961 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi116 2024-09-19T06:42:04.961 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:42:04.961 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.conf 2024-09-19T06:42:04.982 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:42:04.982 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:05.040 INFO:tasks.cephadm:Adding host smithi116 to orchestrator... 2024-09-19T06:42:05.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch host add smithi116 2024-09-19T06:42:05.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:05 smithi114 ceph-mon[30820]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:05.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:05.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:05.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:05.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:05.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:05 smithi114 ceph-mon[30820]: Updating smithi114:/etc/ceph/ceph.conf 2024-09-19T06:42:05.788 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:06.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:06 smithi114 ceph-mon[30820]: Updating smithi114:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:06.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:06 smithi114 ceph-mon[30820]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:06.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:06.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:06.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:06.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.388+0000 7ffa71200640 1 -- 172.21.15.114:0/2128293698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 msgr2=0x7ffa6c107840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:06.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.388+0000 7ffa71200640 1 --2- 172.21.15.114:0/2128293698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 0x7ffa6c107840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009880 tx=0x7ffa5c02a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:06.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.388+0000 7ffa71200640 1 -- 172.21.15.114:0/2128293698 shutdown_connections 2024-09-19T06:42:06.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.388+0000 7ffa71200640 1 --2- 172.21.15.114:0/2128293698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 0x7ffa6c107840 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:06.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.388+0000 7ffa71200640 1 -- 172.21.15.114:0/2128293698 >> 172.21.15.114:0/2128293698 conn(0x7ffa6c0fd0b0 msgr2=0x7ffa6c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:06.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.388+0000 7ffa71200640 1 -- 172.21.15.114:0/2128293698 shutdown_connections 2024-09-19T06:42:06.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.388+0000 7ffa71200640 1 -- 172.21.15.114:0/2128293698 wait complete. 2024-09-19T06:42:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.389+0000 7ffa71200640 1 Processor -- start 2024-09-19T06:42:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.389+0000 7ffa71200640 1 -- start start 2024-09-19T06:42:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.390+0000 7ffa71200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 0x7ffa6c196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.390+0000 7ffa71200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c10ba90 con 0x7ffa6c107460 2024-09-19T06:42:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.390+0000 7ffa6b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 0x7ffa6c196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.390+0000 7ffa6b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 0x7ffa6c196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36710/0 (socket says 172.21.15.114:36710) 2024-09-19T06:42:06.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.390+0000 7ffa6b400640 1 -- 172.21.15.114:0/4014327290 learned_addr learned my addr 172.21.15.114:0/4014327290 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:06.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.390+0000 7ffa6b400640 1 -- 172.21.15.114:0/4014327290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c1974b0 con 0x7ffa6c107460 2024-09-19T06:42:06.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.391+0000 7ffa6b400640 1 --2- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 0x7ffa6c196f70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009230 tx=0x7ffa5c003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:06.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.391+0000 7ffa68c00640 1 -- 172.21.15.114:0/4014327290 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa5c03f070 con 0x7ffa6c107460 2024-09-19T06:42:06.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.391+0000 7ffa68c00640 1 -- 172.21.15.114:0/4014327290 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa5c03bc80 con 0x7ffa6c107460 2024-09-19T06:42:06.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.391+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c1977a0 con 0x7ffa6c107460 2024-09-19T06:42:06.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.391+0000 7ffa68c00640 1 -- 172.21.15.114:0/4014327290 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffa5c035030 con 0x7ffa6c107460 2024-09-19T06:42:06.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.391+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c19b840 con 0x7ffa6c107460 2024-09-19T06:42:06.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.392+0000 7ffa68c00640 1 -- 172.21.15.114:0/4014327290 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7ffa5c042070 con 0x7ffa6c107460 2024-09-19T06:42:06.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.392+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6c102b10 con 0x7ffa6c107460 2024-09-19T06:42:06.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.393+0000 7ffa68c00640 1 --2- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa50039300 0x7ffa5003b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:06.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.393+0000 7ffa68c00640 1 -- 172.21.15.114:0/4014327290 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffa5c06d010 con 0x7ffa6c107460 2024-09-19T06:42:06.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.393+0000 7ffa63e00640 1 --2- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa50039300 0x7ffa5003b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:06.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.394+0000 7ffa63e00640 1 --2- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa50039300 0x7ffa5003b7c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa540098c0 tx=0x7ffa54006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:06.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.398+0000 7ffa68c00640 1 -- 172.21.15.114:0/4014327290 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ffa5c0372d0 con 0x7ffa6c107460 2024-09-19T06:42:06.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:06.584+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}) -- 0x7ffa6c1982d0 con 0x7ffa50039300 2024-09-19T06:42:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:07 smithi114 ceph-mon[30820]: Updating smithi114:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T06:42:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:08 smithi114 ceph-mon[30820]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:09 smithi114 ceph-mon[30820]: Deploying cephadm binary to smithi116 2024-09-19T06:42:09.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.753+0000 7ffa68c00640 1 -- 172.21.15.114:0/4014327290 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ffa6c1982d0 con 0x7ffa50039300 2024-09-19T06:42:09.755 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi116' with addr '172.21.15.116' 2024-09-19T06:42:09.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.755+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa50039300 msgr2=0x7ffa5003b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.755+0000 7ffa71200640 1 --2- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa50039300 0x7ffa5003b7c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa540098c0 tx=0x7ffa54006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.755+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 msgr2=0x7ffa6c196f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.755+0000 7ffa71200640 1 --2- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 0x7ffa6c196f70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009230 tx=0x7ffa5c003d40 comp rx=0 tx=0).stop 2024-09-19T06:42:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.755+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 shutdown_connections 2024-09-19T06:42:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.755+0000 7ffa71200640 1 --2- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa50039300 0x7ffa5003b7c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:09.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.755+0000 7ffa71200640 1 --2- 172.21.15.114:0/4014327290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa6c107460 0x7ffa6c196f70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.755+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 >> 172.21.15.114:0/4014327290 conn(0x7ffa6c0fd0b0 msgr2=0x7ffa6c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.756+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 shutdown_connections 2024-09-19T06:42:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:09.756+0000 7ffa71200640 1 -- 172.21.15.114:0/4014327290 wait complete. 2024-09-19T06:42:10.483 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-19T06:42:10.840 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:11.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:11.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:10 smithi114 ceph-mon[30820]: Added host smithi116 2024-09-19T06:42:11.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:11.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:11.124 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.387+0000 7f7d54a00640 1 -- 172.21.15.114:0/3029229368 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d4802ee50 con 0x7f7d581097c0 2024-09-19T06:42:11.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.388+0000 7f7d5cc00640 1 -- 172.21.15.114:0/3029229368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 msgr2=0x7f7d58109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:11.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.388+0000 7f7d5cc00640 1 --2- 172.21.15.114:0/3029229368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 0x7f7d58109ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009c20 tx=0x7f7d4802aa90 comp rx=0 tx=0).stop 2024-09-19T06:42:11.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.388+0000 7f7d5cc00640 1 -- 172.21.15.114:0/3029229368 shutdown_connections 2024-09-19T06:42:11.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.388+0000 7f7d5cc00640 1 --2- 172.21.15.114:0/3029229368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 0x7f7d58109ba0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:11.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.388+0000 7f7d5cc00640 1 -- 172.21.15.114:0/3029229368 >> 172.21.15.114:0/3029229368 conn(0x7f7d580ff4e0 msgr2=0x7f7d58101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:11.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.388+0000 7f7d5cc00640 1 -- 172.21.15.114:0/3029229368 shutdown_connections 2024-09-19T06:42:11.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.388+0000 7f7d5cc00640 1 -- 172.21.15.114:0/3029229368 wait complete. 2024-09-19T06:42:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.389+0000 7f7d5cc00640 1 Processor -- start 2024-09-19T06:42:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.389+0000 7f7d5cc00640 1 -- start start 2024-09-19T06:42:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.390+0000 7f7d5cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 0x7f7d58199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:11.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.390+0000 7f7d5cc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810de80 con 0x7f7d581097c0 2024-09-19T06:42:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.390+0000 7f7d55e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 0x7f7d58199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.390+0000 7f7d55e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 0x7f7d58199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60188/0 (socket says 172.21.15.114:60188) 2024-09-19T06:42:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.390+0000 7f7d55e00640 1 -- 172.21.15.114:0/641320938 learned_addr learned my addr 172.21.15.114:0/641320938 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:11.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.390+0000 7f7d55e00640 1 -- 172.21.15.114:0/641320938 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d58199870 con 0x7f7d581097c0 2024-09-19T06:42:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.390+0000 7f7d55e00640 1 --2- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 0x7f7d58199330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7d48005e30 tx=0x7f7d4802fc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.391+0000 7f7d4ea00640 1 -- 172.21.15.114:0/641320938 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d48047070 con 0x7f7d581097c0 2024-09-19T06:42:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.391+0000 7f7d4ea00640 1 -- 172.21.15.114:0/641320938 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d48003df0 con 0x7f7d581097c0 2024-09-19T06:42:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.391+0000 7f7d4ea00640 1 -- 172.21.15.114:0/641320938 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d48033050 con 0x7f7d581097c0 2024-09-19T06:42:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.391+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d58199b60 con 0x7f7d581097c0 2024-09-19T06:42:11.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.391+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d5819dc00 con 0x7f7d581097c0 2024-09-19T06:42:11.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.392+0000 7f7d4ea00640 1 -- 172.21.15.114:0/641320938 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f7d4804a070 con 0x7f7d581097c0 2024-09-19T06:42:11.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.392+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d58104f00 con 0x7f7d581097c0 2024-09-19T06:42:11.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.392+0000 7f7d4ea00640 1 --2- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d3c039300 0x7f7d3c03b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:11.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.392+0000 7f7d4ea00640 1 -- 172.21.15.114:0/641320938 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7d4803e030 con 0x7f7d581097c0 2024-09-19T06:42:11.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.393+0000 7f7d55400640 1 --2- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d3c039300 0x7f7d3c03b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:11.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.393+0000 7f7d55400640 1 --2- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d3c039300 0x7f7d3c03b7c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d440098c0 tx=0x7f7d44006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:11.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.398+0000 7f7d4ea00640 1 -- 172.21.15.114:0/641320938 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7d4804a360 con 0x7f7d581097c0 2024-09-19T06:42:11.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.579+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d5819a690 con 0x7f7d3c039300 2024-09-19T06:42:11.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.581+0000 7f7d4ea00640 1 -- 172.21.15.114:0/641320938 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f7d5819a690 con 0x7f7d3c039300 2024-09-19T06:42:11.582 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:42:11.582 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}] 2024-09-19T06:42:11.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.583+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d3c039300 msgr2=0x7f7d3c03b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:11.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.583+0000 7f7d5cc00640 1 --2- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d3c039300 0x7f7d3c03b7c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d440098c0 tx=0x7f7d44006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:11.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.583+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 msgr2=0x7f7d58199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:11.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.583+0000 7f7d5cc00640 1 --2- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 0x7f7d58199330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7d48005e30 tx=0x7f7d4802fc10 comp rx=0 tx=0).stop 2024-09-19T06:42:11.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.583+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 shutdown_connections 2024-09-19T06:42:11.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.583+0000 7f7d5cc00640 1 --2- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d3c039300 0x7f7d3c03b7c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:11.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.583+0000 7f7d5cc00640 1 --2- 172.21.15.114:0/641320938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d581097c0 0x7f7d58199330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:11.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.584+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 >> 172.21.15.114:0/641320938 conn(0x7f7d580ff4e0 msgr2=0x7f7d58100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:11.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.584+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 shutdown_connections 2024-09-19T06:42:11.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:11.584+0000 7f7d5cc00640 1 -- 172.21.15.114:0/641320938 wait complete. 2024-09-19T06:42:12.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:12 smithi114 ceph-mon[30820]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:12.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:12.320 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi118 2024-09-19T06:42:12.320 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:42:12.320 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/etc/ceph/ceph.conf 2024-09-19T06:42:12.338 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:42:12.338 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:12.396 INFO:tasks.cephadm:Adding host smithi118 to orchestrator... 2024-09-19T06:42:12.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch host add smithi118 2024-09-19T06:42:12.763 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:13 smithi114 ceph-mon[30820]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:42:13.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.304+0000 7f8c0d800640 1 -- 172.21.15.114:0/1007132520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 msgr2=0x7f8c08076b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:13.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.304+0000 7f8c0d800640 1 --2- 172.21.15.114:0/1007132520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 0x7f8c08076b00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009880 tx=0x7f8bf802a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:13.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.304+0000 7f8c0d800640 1 -- 172.21.15.114:0/1007132520 shutdown_connections 2024-09-19T06:42:13.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.304+0000 7f8c0d800640 1 --2- 172.21.15.114:0/1007132520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 0x7f8c08076b00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:13.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.304+0000 7f8c0d800640 1 -- 172.21.15.114:0/1007132520 >> 172.21.15.114:0/1007132520 conn(0x7f8c080ff4e0 msgr2=0x7f8c08101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:13.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.304+0000 7f8c0d800640 1 -- 172.21.15.114:0/1007132520 shutdown_connections 2024-09-19T06:42:13.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.304+0000 7f8c0d800640 1 -- 172.21.15.114:0/1007132520 wait complete. 2024-09-19T06:42:13.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.305+0000 7f8c0d800640 1 Processor -- start 2024-09-19T06:42:13.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.305+0000 7f8c0d800640 1 -- start start 2024-09-19T06:42:13.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.306+0000 7f8c0d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 0x7f8c081a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.306+0000 7f8c0d800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081122d0 con 0x7f8c08076720 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.306+0000 7f8c06a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 0x7f8c081a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.306+0000 7f8c06a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 0x7f8c081a33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60200/0 (socket says 172.21.15.114:60200) 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.306+0000 7f8c06a00640 1 -- 172.21.15.114:0/4248707266 learned_addr learned my addr 172.21.15.114:0/4248707266 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.306+0000 7f8c06a00640 1 -- 172.21.15.114:0/4248707266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c081a3910 con 0x7f8c08076720 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.307+0000 7f8c06a00640 1 --2- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 0x7f8c081a33d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009230 tx=0x7f8bf8003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.307+0000 7f8bff400640 1 -- 172.21.15.114:0/4248707266 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bf803f070 con 0x7f8c08076720 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.307+0000 7f8bff400640 1 -- 172.21.15.114:0/4248707266 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf803bd40 con 0x7f8c08076720 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.307+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c081a3ba0 con 0x7f8c08076720 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.307+0000 7f8bff400640 1 -- 172.21.15.114:0/4248707266 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8bf8035030 con 0x7f8c08076720 2024-09-19T06:42:13.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.307+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c0819d590 con 0x7f8c08076720 2024-09-19T06:42:13.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.308+0000 7f8bff400640 1 -- 172.21.15.114:0/4248707266 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f8bf8042070 con 0x7f8c08076720 2024-09-19T06:42:13.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.308+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c08077e60 con 0x7f8c08076720 2024-09-19T06:42:13.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.308+0000 7f8bff400640 1 --2- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8bec0392a0 0x7f8bec03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:13.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.308+0000 7f8bff400640 1 -- 172.21.15.114:0/4248707266 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8bf806d010 con 0x7f8c08076720 2024-09-19T06:42:13.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.309+0000 7f8c06000640 1 --2- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8bec0392a0 0x7f8bec03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:13.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.309+0000 7f8c06000640 1 --2- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8bec0392a0 0x7f8bec03b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8bf40098c0 tx=0x7f8bf4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:13.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.313+0000 7f8bff400640 1 -- 172.21.15.114:0/4248707266 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8bf8030d60 con 0x7f8c08076720 2024-09-19T06:42:13.493 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:13.491+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi118", "target": ["mon-mgr", ""]}) -- 0x7f8c0819eac0 con 0x7f8bec0392a0 2024-09-19T06:42:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:14 smithi114 ceph-mon[30820]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:42:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:15 smithi114 ceph-mon[30820]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi118", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:15 smithi114 ceph-mon[30820]: Updating smithi116:/etc/ceph/ceph.conf 2024-09-19T06:42:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:15 smithi114 ceph-mon[30820]: Updating smithi116:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:16 smithi114 ceph-mon[30820]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:16 smithi114 ceph-mon[30820]: Deploying cephadm binary to smithi118 2024-09-19T06:42:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:16 smithi114 ceph-mon[30820]: Updating smithi116:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T06:42:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:16 smithi114 ceph-mon[30820]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:16.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.750+0000 7f8bff400640 1 -- 172.21.15.114:0/4248707266 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8c0819eac0 con 0x7f8bec0392a0 2024-09-19T06:42:16.751 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi118' with addr '172.21.15.118' 2024-09-19T06:42:16.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8bec0392a0 msgr2=0x7f8bec03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:16.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 --2- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8bec0392a0 0x7f8bec03b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8bf40098c0 tx=0x7f8bf4006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:16.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 msgr2=0x7f8c081a33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:16.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 --2- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 0x7f8c081a33d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009230 tx=0x7f8bf8003d80 comp rx=0 tx=0).stop 2024-09-19T06:42:16.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 shutdown_connections 2024-09-19T06:42:16.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 --2- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8bec0392a0 0x7f8bec03b760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:16.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 --2- 172.21.15.114:0/4248707266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c08076720 0x7f8c081a33d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:16.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 >> 172.21.15.114:0/4248707266 conn(0x7f8c080ff4e0 msgr2=0x7f8c080ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:16.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 shutdown_connections 2024-09-19T06:42:16.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:16.752+0000 7f8c0d800640 1 -- 172.21.15.114:0/4248707266 wait complete. 2024-09-19T06:42:17.418 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-19T06:42:17.770 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:18.065 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:18.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:17 smithi114 ceph-mon[30820]: Added host smithi118 2024-09-19T06:42:18.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:18.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:17 smithi114 ceph-mon[30820]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:18.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:18.066 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.309+0000 7f866a800640 1 -- 172.21.15.114:0/85061324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 msgr2=0x7f8664109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.309+0000 7f866a800640 1 --2- 172.21.15.114:0/85061324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 0x7f8664109b20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8658009880 tx=0x7f865802a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.309+0000 7f866a800640 1 -- 172.21.15.114:0/85061324 shutdown_connections 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.309+0000 7f866a800640 1 --2- 172.21.15.114:0/85061324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 0x7f8664109b20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.309+0000 7f866a800640 1 -- 172.21.15.114:0/85061324 >> 172.21.15.114:0/85061324 conn(0x7f86640ff280 msgr2=0x7f86641016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.309+0000 7f866a800640 1 -- 172.21.15.114:0/85061324 shutdown_connections 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.309+0000 7f866a800640 1 -- 172.21.15.114:0/85061324 wait complete. 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.310+0000 7f866a800640 1 Processor -- start 2024-09-19T06:42:18.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.310+0000 7f866a800640 1 -- start start 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.310+0000 7f866a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 0x7f86641991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.310+0000 7f866a800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f866410dd50 con 0x7f8664109740 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.311+0000 7f8669400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 0x7f86641991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.311+0000 7f8669400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 0x7f86641991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54822/0 (socket says 172.21.15.114:54822) 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.311+0000 7f8669400640 1 -- 172.21.15.114:0/653880700 learned_addr learned my addr 172.21.15.114:0/653880700 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.311+0000 7f8669400640 1 -- 172.21.15.114:0/653880700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8664199730 con 0x7f8664109740 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.311+0000 7f8669400640 1 --2- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 0x7f86641991f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8658009230 tx=0x7f8658003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.312+0000 7f8662000640 1 -- 172.21.15.114:0/653880700 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f865803f070 con 0x7f8664109740 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.312+0000 7f8662000640 1 -- 172.21.15.114:0/653880700 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f865803bc80 con 0x7f8664109740 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.312+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8664199a20 con 0x7f8664109740 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.312+0000 7f8662000640 1 -- 172.21.15.114:0/653880700 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8658035030 con 0x7f8664109740 2024-09-19T06:42:18.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.312+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866419dac0 con 0x7f8664109740 2024-09-19T06:42:18.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.313+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8664104ce0 con 0x7f8664109740 2024-09-19T06:42:18.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.313+0000 7f8662000640 1 -- 172.21.15.114:0/653880700 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f8658042070 con 0x7f8664109740 2024-09-19T06:42:18.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.313+0000 7f8662000640 1 --2- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f864c0392a0 0x7f864c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:18.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.314+0000 7f8662000640 1 -- 172.21.15.114:0/653880700 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f865806d070 con 0x7f8664109740 2024-09-19T06:42:18.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.314+0000 7f8668a00640 1 --2- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f864c0392a0 0x7f864c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:18.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.314+0000 7f8668a00640 1 --2- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f864c0392a0 0x7f864c03b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f86500098c0 tx=0x7f8650006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:18.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.319+0000 7f8662000640 1 -- 172.21.15.114:0/653880700 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8658031570 con 0x7f8664109740 2024-09-19T06:42:18.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.497+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8664109bd0 con 0x7f864c0392a0 2024-09-19T06:42:18.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.498+0000 7f8662000640 1 -- 172.21.15.114:0/653880700 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f8664109bd0 con 0x7f864c0392a0 2024-09-19T06:42:18.500 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:42:18.500 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.118", "hostname": "smithi118", "labels": [], "status": ""}] 2024-09-19T06:42:18.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f864c0392a0 msgr2=0x7f864c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:18.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 --2- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f864c0392a0 0x7f864c03b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f86500098c0 tx=0x7f8650006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 msgr2=0x7f86641991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 --2- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 0x7f86641991f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8658009230 tx=0x7f8658003d40 comp rx=0 tx=0).stop 2024-09-19T06:42:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 shutdown_connections 2024-09-19T06:42:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 --2- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f864c0392a0 0x7f864c03b760 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 --2- 172.21.15.114:0/653880700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8664109740 0x7f86641991f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 >> 172.21.15.114:0/653880700 conn(0x7f86640ff280 msgr2=0x7f8664100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 shutdown_connections 2024-09-19T06:42:18.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:18.501+0000 7f866a800640 1 -- 172.21.15.114:0/653880700 wait complete. 2024-09-19T06:42:19.180 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-19T06:42:19.180 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-19T06:42:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:19.531 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:20.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.088+0000 7f4e2e600640 1 -- 172.21.15.114:0/1789567288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 msgr2=0x7f4e28109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:20.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.088+0000 7f4e2e600640 1 --2- 172.21.15.114:0/1789567288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 0x7f4e28109a50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009880 tx=0x7f4e1c02a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:20.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.089+0000 7f4e2e600640 1 -- 172.21.15.114:0/1789567288 shutdown_connections 2024-09-19T06:42:20.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.089+0000 7f4e2e600640 1 --2- 172.21.15.114:0/1789567288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 0x7f4e28109a50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:20.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.089+0000 7f4e2e600640 1 -- 172.21.15.114:0/1789567288 >> 172.21.15.114:0/1789567288 conn(0x7f4e280ff280 msgr2=0x7f4e281016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:20.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.089+0000 7f4e2e600640 1 -- 172.21.15.114:0/1789567288 shutdown_connections 2024-09-19T06:42:20.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.089+0000 7f4e2e600640 1 -- 172.21.15.114:0/1789567288 wait complete. 2024-09-19T06:42:20.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.090+0000 7f4e2e600640 1 Processor -- start 2024-09-19T06:42:20.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.090+0000 7f4e2e600640 1 -- start start 2024-09-19T06:42:20.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.090+0000 7f4e2e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 0x7f4e28199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:20.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.090+0000 7f4e2e600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2810dca0 con 0x7f4e28109670 2024-09-19T06:42:20.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.090+0000 7f4e2d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 0x7f4e28199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:20.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.090+0000 7f4e2d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 0x7f4e28199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54842/0 (socket says 172.21.15.114:54842) 2024-09-19T06:42:20.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.090+0000 7f4e2d200640 1 -- 172.21.15.114:0/3565804920 learned_addr learned my addr 172.21.15.114:0/3565804920 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:20.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.091+0000 7f4e2d200640 1 -- 172.21.15.114:0/3565804920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e28199680 con 0x7f4e28109670 2024-09-19T06:42:20.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.091+0000 7f4e2d200640 1 --2- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 0x7f4e28199140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009230 tx=0x7f4e1c003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:20.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.091+0000 7f4e25600640 1 -- 172.21.15.114:0/3565804920 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e1c03f070 con 0x7f4e28109670 2024-09-19T06:42:20.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.091+0000 7f4e25600640 1 -- 172.21.15.114:0/3565804920 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e1c03bd40 con 0x7f4e28109670 2024-09-19T06:42:20.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.091+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e28199910 con 0x7f4e28109670 2024-09-19T06:42:20.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.091+0000 7f4e25600640 1 -- 172.21.15.114:0/3565804920 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e1c035030 con 0x7f4e28109670 2024-09-19T06:42:20.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.091+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2819da10 con 0x7f4e28109670 2024-09-19T06:42:20.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.092+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e28104ce0 con 0x7f4e28109670 2024-09-19T06:42:20.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.092+0000 7f4e25600640 1 -- 172.21.15.114:0/3565804920 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f4e1c042070 con 0x7f4e28109670 2024-09-19T06:42:20.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.093+0000 7f4e25600640 1 --2- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e100392a0 0x7f4e1003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:20.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.093+0000 7f4e25600640 1 -- 172.21.15.114:0/3565804920 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4e1c06cfd0 con 0x7f4e28109670 2024-09-19T06:42:20.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.093+0000 7f4e27e00640 1 --2- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e100392a0 0x7f4e1003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:20.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.094+0000 7f4e27e00640 1 --2- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e100392a0 0x7f4e1003b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4e140098c0 tx=0x7f4e14006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:20.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.098+0000 7f4e25600640 1 -- 172.21.15.114:0/3565804920 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4e1c030d60 con 0x7f4e28109670 2024-09-19T06:42:20.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:20.278+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f4e2819a4a0 con 0x7f4e28109670 2024-09-19T06:42:20.280 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:20 smithi114 ceph-mon[30820]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:42:20.280 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:20 smithi114 ceph-mon[30820]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:21.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.111+0000 7f4e25600640 1 -- 172.21.15.114:0/3565804920 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f4e1c03b860 con 0x7f4e28109670 2024-09-19T06:42:21.113 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-09-19T06:42:21.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e100392a0 msgr2=0x7f4e1003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:21.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 --2- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e100392a0 0x7f4e1003b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4e140098c0 tx=0x7f4e14006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:21.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 msgr2=0x7f4e28199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:21.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 --2- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 0x7f4e28199140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009230 tx=0x7f4e1c003d80 comp rx=0 tx=0).stop 2024-09-19T06:42:21.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 shutdown_connections 2024-09-19T06:42:21.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 --2- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e100392a0 0x7f4e1003b760 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:21.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 --2- 172.21.15.114:0/3565804920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e28109670 0x7f4e28199140 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:21.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 >> 172.21.15.114:0/3565804920 conn(0x7f4e280ff280 msgr2=0x7f4e28100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:21.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 shutdown_connections 2024-09-19T06:42:21.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:21.114+0000 7f4e2e600640 1 -- 172.21.15.114:0/3565804920 wait complete. 2024-09-19T06:42:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:21 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3565804920' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-19T06:42:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi118", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:42:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:21.932 INFO:tasks.cephadm:Adding mon.a on smithi114 2024-09-19T06:42:21.932 INFO:tasks.cephadm:Adding mon.b on smithi116 2024-09-19T06:42:21.932 INFO:tasks.cephadm:Adding mon.c on smithi118 2024-09-19T06:42:21.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi114:172.21.15.114=a;smithi116:172.21.15.116=b;smithi118:172.21.15.118=c' 2024-09-19T06:42:22.163 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:22 smithi114 ceph-mon[30820]: Updating smithi118:/etc/ceph/ceph.conf 2024-09-19T06:42:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:22 smithi114 ceph-mon[30820]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:22 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3565804920' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-19T06:42:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:22 smithi114 ceph-mon[30820]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:22 smithi114 ceph-mon[30820]: Updating smithi118:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.597+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/3081855797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 msgr2=0x7f8d88107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.597+0000 7f8d8d8d8640 1 --2- 172.21.15.118:0/3081855797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 0x7f8d88107a40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009880 tx=0x7f8d7802a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.597+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/3081855797 shutdown_connections 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.597+0000 7f8d8d8d8640 1 --2- 172.21.15.118:0/3081855797 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 0x7f8d88107a40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.597+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/3081855797 >> 172.21.15.118:0/3081855797 conn(0x7f8d880fd2b0 msgr2=0x7f8d880ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.597+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/3081855797 shutdown_connections 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.597+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/3081855797 wait complete. 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.598+0000 7f8d8d8d8640 1 Processor -- start 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.598+0000 7f8d8d8d8640 1 -- start start 2024-09-19T06:42:22.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.598+0000 7f8d8d8d8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 0x7f8d881970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:22.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.598+0000 7f8d8d8d8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8810bc70 con 0x7f8d88107660 2024-09-19T06:42:22.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d86ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 0x7f8d881970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:22.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d86ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 0x7f8d881970a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.118:47598/0 (socket says 172.21.15.118:47598) 2024-09-19T06:42:22.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d86ffd640 1 -- 172.21.15.118:0/1921488267 learned_addr learned my addr 172.21.15.118:0/1921488267 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:42:22.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d86ffd640 1 -- 172.21.15.118:0/1921488267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d881975e0 con 0x7f8d88107660 2024-09-19T06:42:22.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d86ffd640 1 --2- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 0x7f8d881970a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009230 tx=0x7f8d78003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:22.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d8c8d6640 1 -- 172.21.15.118:0/1921488267 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d7803f070 con 0x7f8d88107660 2024-09-19T06:42:22.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d8c8d6640 1 -- 172.21.15.118:0/1921488267 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d7803bd40 con 0x7f8d88107660 2024-09-19T06:42:22.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d881978d0 con 0x7f8d88107660 2024-09-19T06:42:22.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.599+0000 7f8d8c8d6640 1 -- 172.21.15.118:0/1921488267 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d78035030 con 0x7f8d88107660 2024-09-19T06:42:22.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.600+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8819b970 con 0x7f8d88107660 2024-09-19T06:42:22.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.600+0000 7f8d8c8d6640 1 -- 172.21.15.118:0/1921488267 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f8d78042070 con 0x7f8d88107660 2024-09-19T06:42:22.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.601+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d88102d10 con 0x7f8d88107660 2024-09-19T06:42:22.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.601+0000 7f8d8c8d6640 1 --2- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8d740392a0 0x7f8d7403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:22.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.601+0000 7f8d8c8d6640 1 -- 172.21.15.118:0/1921488267 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8d7806cc60 con 0x7f8d88107660 2024-09-19T06:42:22.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.601+0000 7f8d7ffff640 1 --2- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8d740392a0 0x7f8d7403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:22.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.601+0000 7f8d7ffff640 1 --2- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8d740392a0 0x7f8d7403b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8d700098c0 tx=0x7f8d70006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:22.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.606+0000 7f8d8c8d6640 1 -- 172.21.15.118:0/1921488267 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f8d780054f0 con 0x7f8d88107660 2024-09-19T06:42:22.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.791+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi116:172.21.15.116=b;smithi118:172.21.15.118=c", "target": ["mon-mgr", ""]}) -- 0x7f8d88198400 con 0x7f8d740392a0 2024-09-19T06:42:22.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.796+0000 7f8d8c8d6640 1 -- 172.21.15.118:0/1921488267 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8d88198400 con 0x7f8d740392a0 2024-09-19T06:42:22.798 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled mon update... 2024-09-19T06:42:22.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.798+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8d740392a0 msgr2=0x7f8d7403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.798+0000 7f8d8d8d8640 1 --2- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8d740392a0 0x7f8d7403b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8d700098c0 tx=0x7f8d70006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.798+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 msgr2=0x7f8d881970a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.798+0000 7f8d8d8d8640 1 --2- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 0x7f8d881970a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009230 tx=0x7f8d78003d80 comp rx=0 tx=0).stop 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.798+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 shutdown_connections 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.798+0000 7f8d8d8d8640 1 --2- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8d740392a0 0x7f8d7403b760 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.798+0000 7f8d8d8d8640 1 --2- 172.21.15.118:0/1921488267 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d88107660 0x7f8d881970a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.798+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 >> 172.21.15.118:0/1921488267 conn(0x7f8d880fd2b0 msgr2=0x7f8d880feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.799+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 shutdown_connections 2024-09-19T06:42:22.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:22.799+0000 7f8d8d8d8640 1 -- 172.21.15.118:0/1921488267 wait complete. 2024-09-19T06:42:23.385 DEBUG:teuthology.orchestra.run.smithi116:mon.b> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mon.b.service 2024-09-19T06:42:23.388 DEBUG:teuthology.orchestra.run.smithi118:mon.c> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mon.c.service 2024-09-19T06:42:23.390 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T06:42:23.390 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T06:42:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: Updating smithi118:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T06:42:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:24 smithi114 ceph-mon[30820]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi116:172.21.15.116=b;smithi118:172.21.15.118=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:24 smithi114 ceph-mon[30820]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi116:172.21.15.116=b;smithi118:172.21.15.118=c;count:3 2024-09-19T06:42:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:24 smithi114 ceph-mon[30820]: Deploying daemon mon.c on smithi118 2024-09-19T06:42:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:24 smithi114 ceph-mon[30820]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.893+0000 7f9068592640 1 -- 172.21.15.118:0/2642128491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 msgr2=0x7f9060100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.893+0000 7f9068592640 1 --2- 172.21.15.118:0/2642128491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 0x7f9060100d70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f905c009880 tx=0x7f905c02a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.893+0000 7f9068592640 1 -- 172.21.15.118:0/2642128491 shutdown_connections 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.893+0000 7f9068592640 1 --2- 172.21.15.118:0/2642128491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 0x7f9060100d70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.893+0000 7f9068592640 1 -- 172.21.15.118:0/2642128491 >> 172.21.15.118:0/2642128491 conn(0x7f9060076250 msgr2=0x7f9060076660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.893+0000 7f9068592640 1 -- 172.21.15.118:0/2642128491 shutdown_connections 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.893+0000 7f9068592640 1 -- 172.21.15.118:0/2642128491 wait complete. 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.894+0000 7f9068592640 1 Processor -- start 2024-09-19T06:42:25.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.894+0000 7f9068592640 1 -- start start 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.894+0000 7f9068592640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 0x7f90601980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.894+0000 7f9068592640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f906010ab90 con 0x7f9060100990 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.895+0000 7f906632e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 0x7f90601980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.895+0000 7f906632e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 0x7f90601980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.118:35862/0 (socket says 172.21.15.118:35862) 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.895+0000 7f906632e640 1 -- 172.21.15.118:0/882773428 learned_addr learned my addr 172.21.15.118:0/882773428 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.895+0000 7f906632e640 1 -- 172.21.15.118:0/882773428 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90601985e0 con 0x7f9060100990 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.895+0000 7f906632e640 1 --2- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 0x7f90601980a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f905c009230 tx=0x7f905c003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.895+0000 7f90577fe640 1 -- 172.21.15.118:0/882773428 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f905c03f070 con 0x7f9060100990 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.896+0000 7f9068592640 1 -- 172.21.15.118:0/882773428 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90601988d0 con 0x7f9060100990 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.896+0000 7f90577fe640 1 -- 172.21.15.118:0/882773428 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f905c03bc80 con 0x7f9060100990 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.896+0000 7f90577fe640 1 -- 172.21.15.118:0/882773428 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f905c035030 con 0x7f9060100990 2024-09-19T06:42:25.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.896+0000 7f9068592640 1 -- 172.21.15.118:0/882773428 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9060198d90 con 0x7f9060100990 2024-09-19T06:42:25.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.897+0000 7f90557fa640 1 -- 172.21.15.118:0/882773428 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9048005290 con 0x7f9060100990 2024-09-19T06:42:25.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.898+0000 7f90577fe640 1 -- 172.21.15.118:0/882773428 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f905c042070 con 0x7f9060100990 2024-09-19T06:42:25.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.898+0000 7f90577fe640 1 --2- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f904c039300 0x7f904c03b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:25.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.898+0000 7f90577fe640 1 -- 172.21.15.118:0/882773428 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f905c06d510 con 0x7f9060100990 2024-09-19T06:42:25.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.898+0000 7f9065b2d640 1 --2- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f904c039300 0x7f904c03b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:25.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.899+0000 7f9065b2d640 1 --2- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f904c039300 0x7f904c03b7c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f90500098c0 tx=0x7f9050006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:25.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:25.903+0000 7f90577fe640 1 -- 172.21.15.118:0/882773428 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f905c04b2f0 con 0x7f9060100990 2024-09-19T06:42:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.165+0000 7f90557fa640 1 -- 172.21.15.118:0/882773428 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9048005580 con 0x7f9060100990 2024-09-19T06:42:26.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.166+0000 7f90577fe640 1 -- 172.21.15.118:0/882773428 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f905c031d50 con 0x7f9060100990 2024-09-19T06:42:26.168 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:42:26.168 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"fface640-7651-11ef-bceb-c7b262605968","modified":"2024-09-19T06:40:54.002091Z","created":"2024-09-19T06:40:54.002091Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-19T06:42:26.168 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 -- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f904c039300 msgr2=0x7f904c03b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 --2- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f904c039300 0x7f904c03b7c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f90500098c0 tx=0x7f9050006db0 comp rx=0 tx=0).stop 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 -- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 msgr2=0x7f90601980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 --2- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 0x7f90601980a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f905c009230 tx=0x7f905c003d40 comp rx=0 tx=0).stop 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 -- 172.21.15.118:0/882773428 shutdown_connections 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 --2- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f904c039300 0x7f904c03b7c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 --2- 172.21.15.118:0/882773428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9060100990 0x7f90601980a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 -- 172.21.15.118:0/882773428 >> 172.21.15.118:0/882773428 conn(0x7f9060076250 msgr2=0x7f90601020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:26.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 -- 172.21.15.118:0/882773428 shutdown_connections 2024-09-19T06:42:26.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:26.169+0000 7f90557fa640 1 -- 172.21.15.118:0/882773428 wait complete. 2024-09-19T06:42:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:26 smithi114 ceph-mon[30820]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:26 smithi118 systemd[1]: Starting Ceph mon.c for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:42:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:27 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/882773428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:27.456 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 podman[35508]: 2024-09-19 06:42:27.3517504 +0000 UTC m=+0.102487471 container create c01cce961036eda6788dd515716479301be6dbf503185577f67d6e98f15af3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-19T06:42:27.456 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 podman[35508]: 2024-09-19 06:42:27.265065709 +0000 UTC m=+0.015802794 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:42:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 podman[35508]: 2024-09-19 06:42:27.554042916 +0000 UTC m=+0.304779987 container init c01cce961036eda6788dd515716479301be6dbf503185577f67d6e98f15af3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-c, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-19T06:42:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 podman[35508]: 2024-09-19 06:42:27.55728671 +0000 UTC m=+0.308023781 container start c01cce961036eda6788dd515716479301be6dbf503185577f67d6e98f15af3f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-19T06:42:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: set uid:gid to 167:167 (ceph:ceph) 2024-09-19T06:42:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable), process ceph-mon, pid 2 2024-09-19T06:42:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pidfile_write: ignore empty --pid-file 2024-09-19T06:42:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: load: jerasure load: lrc 2024-09-19T06:42:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: RocksDB version: 7.9.2 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Git sha 0 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Compile date 2024-09-18 16:29:48 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: DB SUMMARY 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: DB Session ID: DZ8CNI8PJDUTUTP03OQL 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: CURRENT file: CURRENT 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: IDENTITY file: IDENTITY 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.error_if_exists: 0 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.create_if_missing: 0 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.paranoid_checks: 1 2024-09-19T06:42:27.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.env: 0x558570e04940 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.fs: PosixFileSystem 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.info_log: 0x558572e916e0 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_file_opening_threads: 16 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.statistics: (nil) 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.use_fsync: 0 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_log_file_size: 0 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-19T06:42:27.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.keep_log_file_num: 1000 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.recycle_log_file_num: 0 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.allow_fallocate: 1 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.allow_mmap_reads: 0 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.allow_mmap_writes: 0 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.use_direct_reads: 0 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.create_missing_column_families: 0 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.db_log_dir: 2024-09-19T06:42:27.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.wal_dir: 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.advise_random_on_open: 1 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.db_write_buffer_size: 0 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.write_buffer_manager: 0x558572ea0dc0 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.rate_limiter: (nil) 2024-09-19T06:42:27.761 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.wal_recovery_mode: 2 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.enable_thread_tracking: 0 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.enable_pipelined_write: 0 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.unordered_write: 0 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.row_cache: None 2024-09-19T06:42:27.762 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.wal_filter: None 2024-09-19T06:42:27.763 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.allow_ingest_behind: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.two_write_queues: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.manual_wal_flush: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.wal_compression: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.atomic_flush: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.log_readahead_size: 0 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-19T06:42:27.764 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.best_efforts_recovery: 0 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.allow_data_in_errors: 0 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.db_host_id: __hostname__ 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_background_jobs: 2 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_background_compactions: -1 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_subcompactions: 1 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_total_wal_size: 0 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-19T06:42:27.765 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_open_files: -1 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bytes_per_sync: 0 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_readahead_size: 0 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_background_flushes: -1 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Compression algorithms supported: 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: kZSTD supported: 0 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: kXpressCompression supported: 0 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: kBZip2Compression supported: 0 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-19T06:42:27.766 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: kLZ4Compression supported: 1 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: kZlibCompression supported: 1 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: kLZ4HCCompression supported: 1 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: kSnappyCompression supported: 1 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.merge_operator: 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_filter: None 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_filter_factory: None 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.sst_partitioner_factory: None 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-19T06:42:27.767 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558572e91320) 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: cache_index_and_filter_blocks: 1 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: pin_top_level_index_and_filter: 1 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: index_type: 0 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: data_block_index_type: 0 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: index_shortening: 1 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: checksum: 4 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: no_block_cache: 0 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: block_cache: 0x558572e851f0 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: block_cache_name: BinnedLRUCache 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: block_cache_options: 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: capacity : 536870912 2024-09-19T06:42:27.768 INFO:journalctl@ceph.mon.c.smithi118.stdout: num_shard_bits : 4 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: strict_capacity_limit : 0 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: high_pri_pool_ratio: 0.000 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: block_cache_compressed: (nil) 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: persistent_cache: (nil) 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: block_size: 4096 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: block_size_deviation: 10 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: block_restart_interval: 16 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: index_block_restart_interval: 1 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: metadata_block_size: 4096 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: partition_filters: 0 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: use_delta_encoding: 1 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: filter_policy: bloomfilter 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: whole_key_filtering: 1 2024-09-19T06:42:27.769 INFO:journalctl@ceph.mon.c.smithi118.stdout: verify_compression: 0 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout: read_amp_bytes_per_bit: 0 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout: format_version: 5 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout: enable_index_compression: 1 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout: block_align: 0 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout: max_auto_readahead_size: 262144 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout: prepopulate_block_cache: 0 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout: initial_auto_readahead_size: 8192 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout: num_file_reads_for_auto_readahead: 2 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.write_buffer_size: 33554432 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_write_buffer_number: 2 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression: NoCompression 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression: Disabled 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.prefix_extractor: nullptr 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.num_levels: 7 2024-09-19T06:42:27.770 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.level: 32767 2024-09-19T06:42:27.771 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.strategy: 0 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.enabled: false 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.target_file_size_base: 67108864 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-19T06:42:27.772 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.arena_block_size: 1048576 2024-09-19T06:42:27.773 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.disable_auto_compactions: 0 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-19T06:42:27.774 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.inplace_update_support: 0 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.bloom_locality: 0 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.max_successive_merges: 0 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.paranoid_file_checks: 0 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.force_consistency_checks: 1 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.report_bg_io_stats: 0 2024-09-19T06:42:27.775 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.ttl: 2592000 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.enable_blob_files: false 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.min_blob_size: 0 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.blob_file_size: 268435456 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-19T06:42:27.776 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.blob_file_starting_level: 0 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7a3da723-4d5d-4cfe-b432-991c8bbf17e6 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726728147584233, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726728147584623, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726728147, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7a3da723-4d5d-4cfe-b432-991c8bbf17e6", "db_session_id": "DZ8CNI8PJDUTUTP03OQL", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726728147584729, "job": 1, "event": "recovery_finished"} 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-19T06:42:27.777 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558572eb2e00 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: DB pointer 0x558572fae000 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: ** DB Stats ** 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: 2024-09-19T06:42:27.778 INFO:journalctl@ceph.mon.c.smithi118.stdout: ** Compaction Stats [default] ** 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: ** Compaction Stats [default] ** 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: 2024-09-19T06:42:27.779 INFO:journalctl@ceph.mon.c.smithi118.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: Block cache BinnedLRUCache@0x558572e851f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-19T06:42:27.780 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: using public_addr v2:172.21.15.118:0/0 -> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: starting mon.c rank -1 at public addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] at bind addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(???) e0 preinit fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).mds e1 new map 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout: e1 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout: btime 2024-09-19T06:40:58:566495+0000 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-19T06:42:27.781 INFO:journalctl@ceph.mon.c.smithi118.stdout: legacy client fscid: -1 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout: 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout: No filesystems configured 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-19T06:42:27.782 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi118", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:27.783 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Updating smithi116:/etc/ceph/ceph.conf 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Updating smithi116:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Deploying cephadm binary to smithi118 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Updating smithi116:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Added host smithi118 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.784 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3565804920' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi118", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Updating smithi118:/etc/ceph/ceph.conf 2024-09-19T06:42:27.785 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3565804920' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Updating smithi118:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Updating smithi118:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:27.786 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi116:172.21.15.116=b;smithi118:172.21.15.118=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:27.787 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi116:172.21.15.116=b;smithi118:172.21.15.118=c;count:3 2024-09-19T06:42:27.787 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: Deploying daemon mon.c on smithi118 2024-09-19T06:42:27.787 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.787 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:27.787 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/882773428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:27.787 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 ceph-mon[35527]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-19T06:42:27.787 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 bash[35508]: c01cce961036eda6788dd515716479301be6dbf503185577f67d6e98f15af3f9 2024-09-19T06:42:27.787 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:27 smithi118 systemd[1]: Started Ceph mon.c for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:42:28.155 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T06:42:28.155 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T06:42:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:42:33.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:42:33.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: mon.a calling monitor election 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: mon.c calling monitor election 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: monmap epoch 2 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: last_changed 2024-09-19T06:42:27.611837+0000 2024-09-19T06:42:33.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: min_mon_release 19 (squid) 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: election_strategy: 1 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: fsmap 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: mgrmap e13: x(active, since 41s) 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: overall HEALTH_OK 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:33.009 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:42:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: mon.a calling monitor election 2024-09-19T06:42:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: mon.c calling monitor election 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: monmap epoch 2 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: last_changed 2024-09-19T06:42:27.611837+0000 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: min_mon_release 19 (squid) 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: election_strategy: 1 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:42:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: fsmap 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: mgrmap e13: x(active, since 41s) 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: overall HEALTH_OK 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:33.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.260+0000 7f7d29099640 1 -- 172.21.15.118:0/3062150093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 msgr2=0x7f7d0c0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:33.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.260+0000 7f7d29099640 1 --2- 172.21.15.118:0/3062150093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 0x7f7d0c0059c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7d2406a9c0 tx=0x7f7d1402c490 comp rx=0 tx=0).stop 2024-09-19T06:42:33.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.260+0000 7f7d29099640 1 -- 172.21.15.118:0/3062150093 shutdown_connections 2024-09-19T06:42:33.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.260+0000 7f7d29099640 1 --2- 172.21.15.118:0/3062150093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 0x7f7d0c0059c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:33.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.260+0000 7f7d29099640 1 -- 172.21.15.118:0/3062150093 >> 172.21.15.118:0/3062150093 conn(0x7f7d240fc220 msgr2=0x7f7d240fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:33.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.260+0000 7f7d29099640 1 -- 172.21.15.118:0/3062150093 shutdown_connections 2024-09-19T06:42:33.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.260+0000 7f7d29099640 1 -- 172.21.15.118:0/3062150093 wait complete. 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.261+0000 7f7d29099640 1 Processor -- start 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d29099640 1 -- start start 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d29099640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 0x7f7d24199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d29099640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 0x7f7d2419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d29099640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d24109780 con 0x7f7d0c003530 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d29099640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7d24109600 con 0x7f7d24199ac0 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d2259c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 0x7f7d2419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d2259c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 0x7f7d2419de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39048/0 (socket says 172.21.15.118:39048) 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d2259c640 1 -- 172.21.15.118:0/2404334115 learned_addr learned my addr 172.21.15.118:0/2404334115 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:42:33.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d2259c640 1 -- 172.21.15.118:0/2404334115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 msgr2=0x7f7d2419de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-19T06:42:33.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d2259c640 1 -- 172.21.15.118:0/2404334115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 msgr2=0x7f7d2419de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T06:42:33.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d2259c640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 0x7f7d2419de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T06:42:33.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d2259c640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 0x7f7d2419de50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:42:33.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.262+0000 7f7d22d9d640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 0x7f7d24199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:33.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d22d9d640 1 -- 172.21.15.118:0/2404334115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 msgr2=0x7f7d2419de50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:42:33.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d22d9d640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 0x7f7d2419de50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:33.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d22d9d640 1 -- 172.21.15.118:0/2404334115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2419a0e0 con 0x7f7d0c003530 2024-09-19T06:42:33.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d22d9d640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 0x7f7d24199580 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7d1402ca80 tx=0x7f7d140339a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:33.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d0bfff640 1 -- 172.21.15.118:0/2404334115 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7d140329e0 con 0x7f7d0c003530 2024-09-19T06:42:33.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d0bfff640 1 -- 172.21.15.118:0/2404334115 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d14005520 con 0x7f7d0c003530 2024-09-19T06:42:33.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2419e390 con 0x7f7d0c003530 2024-09-19T06:42:33.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d0bfff640 1 -- 172.21.15.118:0/2404334115 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7d14004bc0 con 0x7f7d0c003530 2024-09-19T06:42:33.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.263+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d2419e8a0 con 0x7f7d0c003530 2024-09-19T06:42:33.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.265+0000 7f7d0bfff640 1 -- 172.21.15.118:0/2404334115 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f7d14043020 con 0x7f7d0c003530 2024-09-19T06:42:33.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.265+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d2406ac70 con 0x7f7d0c003530 2024-09-19T06:42:33.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.265+0000 7f7d0bfff640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d10039090 0x7f7d1003b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:33.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.265+0000 7f7d0bfff640 1 -- 172.21.15.118:0/2404334115 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7d14041020 con 0x7f7d0c003530 2024-09-19T06:42:33.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.265+0000 7f7d2259c640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d10039090 0x7f7d1003b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:33.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.266+0000 7f7d2259c640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d10039090 0x7f7d1003b550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009100 tx=0x7f7d18008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:33.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.270+0000 7f7d0bfff640 1 -- 172.21.15.118:0/2404334115 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7d1404a020 con 0x7f7d0c003530 2024-09-19T06:42:33.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.503+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7d24106d90 con 0x7f7d0c003530 2024-09-19T06:42:33.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.504+0000 7f7d0bfff640 1 -- 172.21.15.118:0/2404334115 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f7d1403a940 con 0x7f7d0c003530 2024-09-19T06:42:33.505 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:42:33.505 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":2,"fsid":"fface640-7651-11ef-bceb-c7b262605968","modified":"2024-09-19T06:42:27.611837Z","created":"2024-09-19T06:40:54.002091Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-19T06:42:33.505 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 2 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d10039090 msgr2=0x7f7d1003b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d10039090 0x7f7d1003b550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009100 tx=0x7f7d18008040 comp rx=0 tx=0).stop 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 msgr2=0x7f7d24199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 0x7f7d24199580 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7d1402ca80 tx=0x7f7d140339a0 comp rx=0 tx=0).stop 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 shutdown_connections 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7d10039090 0x7f7d1003b550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d24199ac0 0x7f7d2419de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 --2- 172.21.15.118:0/2404334115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d0c003530 0x7f7d24199580 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 >> 172.21.15.118:0/2404334115 conn(0x7f7d240fc220 msgr2=0x7f7d240fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 shutdown_connections 2024-09-19T06:42:33.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:33.506+0000 7f7d29099640 1 -- 172.21.15.118:0/2404334115 wait complete. 2024-09-19T06:42:33.861 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:33 smithi118 ceph-mon[35527]: Deploying daemon mon.b on smithi116 2024-09-19T06:42:33.861 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:33 smithi118 ceph-mon[35527]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:33.861 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:33 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/2404334115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:33.861 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:33 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:33 smithi114 ceph-mon[30820]: Deploying daemon mon.b on smithi116 2024-09-19T06:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:33 smithi114 ceph-mon[30820]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:33 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/2404334115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:34.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:42:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:42:34.612+0000 7fe924400640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-19T06:42:35.108 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T06:42:35.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T06:42:35.356 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:42:35.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.793+0000 7fb628d4c640 1 -- 172.21.15.118:0/2876685073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624106630 msgr2=0x7fb624106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.793+0000 7fb628d4c640 1 --2- 172.21.15.118:0/2876685073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624106630 0x7fb624106a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb614009880 tx=0x7fb61402a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.793+0000 7fb628d4c640 1 -- 172.21.15.118:0/2876685073 shutdown_connections 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.793+0000 7fb628d4c640 1 --2- 172.21.15.118:0/2876685073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624106630 0x7fb624106a10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.793+0000 7fb628d4c640 1 -- 172.21.15.118:0/2876685073 >> 172.21.15.118:0/2876685073 conn(0x7fb6240fc220 msgr2=0x7fb6240fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.793+0000 7fb628d4c640 1 -- 172.21.15.118:0/2876685073 shutdown_connections 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.793+0000 7fb628d4c640 1 -- 172.21.15.118:0/2876685073 wait complete. 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb628d4c640 1 Processor -- start 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb628d4c640 1 -- start start 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb628d4c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 0x7fb624198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb628d4c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624198790 0x7fb624198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:35.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb628d4c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb62410ac60 con 0x7fb624198790 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb628d4c640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb62410aae0 con 0x7fb624106630 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb62259c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 0x7fb624198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb62259c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 0x7fb624198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38512/0 (socket says 172.21.15.118:38512) 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb62259c640 1 -- 172.21.15.118:0/222753864 learned_addr learned my addr 172.21.15.118:0/222753864 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.794+0000 7fb62259c640 1 -- 172.21.15.118:0/222753864 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 msgr2=0x7fb624198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb62259c640 1 -- 172.21.15.118:0/222753864 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 msgr2=0x7fb624198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb62259c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 0x7fb624198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb62259c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 0x7fb624198250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:42:35.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb621d9b640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624198790 0x7fb624198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb621d9b640 1 -- 172.21.15.118:0/222753864 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 msgr2=0x7fb624198250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb621d9b640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 0x7fb624198250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb621d9b640 1 -- 172.21.15.118:0/222753864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6241a12c0 con 0x7fb624198790 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb621d9b640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624198790 0x7fb624198bf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb61800b400 tx=0x7fb61800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb6137fe640 1 -- 172.21.15.118:0/222753864 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb618011630 con 0x7fb624198790 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb6137fe640 1 -- 172.21.15.118:0/222753864 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb618011cb0 con 0x7fb624198790 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6241a1490 con 0x7fb624198790 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.795+0000 7fb6137fe640 1 -- 172.21.15.118:0/222753864 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb618010d80 con 0x7fb624198790 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.796+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6241a19f0 con 0x7fb624198790 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.796+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb624101ca0 con 0x7fb624198790 2024-09-19T06:42:35.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.796+0000 7fb6137fe640 1 -- 172.21.15.118:0/222753864 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7fb618018020 con 0x7fb624198790 2024-09-19T06:42:35.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.797+0000 7fb6137fe640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb60c039040 0x7fb60c03b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:35.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.797+0000 7fb6137fe640 1 -- 172.21.15.118:0/222753864 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb61804bf10 con 0x7fb624198790 2024-09-19T06:42:35.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.797+0000 7fb62259c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb60c039040 0x7fb60c03b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:35.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.798+0000 7fb62259c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb60c039040 0x7fb60c03b500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb6140096c0 tx=0x7fb614031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:35.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:35.802+0000 7fb6137fe640 1 -- 172.21.15.118:0/222753864 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb61801f020 con 0x7fb624198790 2024-09-19T06:42:36.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.033+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb6241a1d00 con 0x7fb624198790 2024-09-19T06:42:36.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.034+0000 7fb6137fe640 1 -- 172.21.15.118:0/222753864 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fb618017500 con 0x7fb624198790 2024-09-19T06:42:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:42:36.035 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":2,"fsid":"fface640-7651-11ef-bceb-c7b262605968","modified":"2024-09-19T06:42:27.611837Z","created":"2024-09-19T06:40:54.002091Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-19T06:42:36.035 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 2 2024-09-19T06:42:36.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb60c039040 msgr2=0x7fb60c03b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:36.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb60c039040 0x7fb60c03b500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb6140096c0 tx=0x7fb614031400 comp rx=0 tx=0).stop 2024-09-19T06:42:36.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624198790 msgr2=0x7fb624198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:36.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624198790 0x7fb624198bf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb61800b400 tx=0x7fb61800b8c0 comp rx=0 tx=0).stop 2024-09-19T06:42:36.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 shutdown_connections 2024-09-19T06:42:36.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb60c039040 0x7fb60c03b500 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:36.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb624198790 0x7fb624198bf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:36.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 --2- 172.21.15.118:0/222753864 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb624106630 0x7fb624198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:36.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 >> 172.21.15.118:0/222753864 conn(0x7fb6240fc220 msgr2=0x7fb6240fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:36.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 shutdown_connections 2024-09-19T06:42:36.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:36.036+0000 7fb628d4c640 1 -- 172.21.15.118:0/222753864 wait complete. 2024-09-19T06:42:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:36 smithi114 ceph-mon[30820]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:36 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/222753864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:36.498 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:36 smithi118 ceph-mon[35527]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:36.498 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:36 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/222753864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:36 smithi116 systemd[1]: Started Ceph mon.b for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:42:37.649 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-19T06:42:37.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-19T06:42:37.899 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:42:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.509+0000 7f6ed20bd640 1 -- 172.21.15.118:0/1730661607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 msgr2=0x7f6eb80059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.509+0000 7f6ed20bd640 1 --2- 172.21.15.118:0/1730661607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 0x7f6eb80059c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc1016c0 tx=0x7f6ebc02c150 comp rx=0 tx=0).stop 2024-09-19T06:42:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.509+0000 7f6ed20bd640 1 -- 172.21.15.118:0/1730661607 shutdown_connections 2024-09-19T06:42:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.509+0000 7f6ed20bd640 1 --2- 172.21.15.118:0/1730661607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 0x7f6eb80059c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.509+0000 7f6ed20bd640 1 -- 172.21.15.118:0/1730661607 >> 172.21.15.118:0/1730661607 conn(0x7f6ecc0fc280 msgr2=0x7f6ecc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.509+0000 7f6ed20bd640 1 -- 172.21.15.118:0/1730661607 shutdown_connections 2024-09-19T06:42:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.509+0000 7f6ed20bd640 1 -- 172.21.15.118:0/1730661607 wait complete. 2024-09-19T06:42:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.510+0000 7f6ed20bd640 1 Processor -- start 2024-09-19T06:42:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.510+0000 7f6ed20bd640 1 -- start start 2024-09-19T06:42:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ed20bd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 0x7f6ecc102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ed20bd640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 0x7f6ecc103300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ed20bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 0x7f6ecc1a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ed20bd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6ecc108690 con 0x7f6eb8003530 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ed20bd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ecc108510 con 0x7f6ecc1a3520 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ed20bd640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6ecc108810 con 0x7f6ecc102ea0 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ec3fff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 0x7f6ecc103300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ec3fff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 0x7f6ecc103300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38516/0 (socket says 172.21.15.118:38516) 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ec3fff640 1 -- 172.21.15.118:0/4034202049 learned_addr learned my addr 172.21.15.118:0/4034202049 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecb7fe640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 0x7f6ecc102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ec3fff640 1 -- 172.21.15.118:0/4034202049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 msgr2=0x7f6ecc103300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ec3fff640 1 -- 172.21.15.118:0/4034202049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 msgr2=0x7f6ecc103300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ec3fff640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 0x7f6ecc103300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T06:42:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecbfff640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 0x7f6ecc1a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ec3fff640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 0x7f6ecc103300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecbfff640 1 -- 172.21.15.118:0/4034202049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 msgr2=0x7f6ecc1a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecbfff640 1 -- 172.21.15.118:0/4034202049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 msgr2=0x7f6ecc1a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecbfff640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 0x7f6ecc1a5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecbfff640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 0x7f6ecc1a5910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecb7fe640 1 -- 172.21.15.118:0/4034202049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 msgr2=0x7f6ecc103300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecb7fe640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 0x7f6ecc103300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecb7fe640 1 -- 172.21.15.118:0/4034202049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 msgr2=0x7f6ecc1a5910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecb7fe640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 0x7f6ecc1a5910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.511+0000 7f6ecb7fe640 1 -- 172.21.15.118:0/4034202049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ecc1a5fe0 con 0x7f6eb8003530 2024-09-19T06:42:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.512+0000 7f6ecb7fe640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 0x7f6ecc102960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc036000 tx=0x7f6ebc004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.512+0000 7f6ec97fa640 1 -- 172.21.15.118:0/4034202049 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc032040 con 0x7f6eb8003530 2024-09-19T06:42:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.512+0000 7f6ec97fa640 1 -- 172.21.15.118:0/4034202049 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ebc043070 con 0x7f6eb8003530 2024-09-19T06:42:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.512+0000 7f6ec97fa640 1 -- 172.21.15.118:0/4034202049 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc036670 con 0x7f6eb8003530 2024-09-19T06:42:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.512+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ecc1a6270 con 0x7f6eb8003530 2024-09-19T06:42:41.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.512+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ecc1a6800 con 0x7f6eb8003530 2024-09-19T06:42:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.513+0000 7f6ec97fa640 1 -- 172.21.15.118:0/4034202049 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f6ebc036810 con 0x7f6eb8003530 2024-09-19T06:42:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.513+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ecc1097c0 con 0x7f6eb8003530 2024-09-19T06:42:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.513+0000 7f6ec97fa640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6eac039730 0x7f6eac03bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.513+0000 7f6ec97fa640 1 -- 172.21.15.118:0/4034202049 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6ebc06e1e0 con 0x7f6eb8003530 2024-09-19T06:42:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.513+0000 7f6ec3fff640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6eac039730 0x7f6eac03bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:41.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.514+0000 7f6ec3fff640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6eac039730 0x7f6eac03bbf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009100 tx=0x7f6eb4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:41.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.518+0000 7f6ec97fa640 1 -- 172.21.15.118:0/4034202049 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f6ebc045ba0 con 0x7f6eb8003530 2024-09-19T06:42:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:42:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: mon.a calling monitor election 2024-09-19T06:42:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: mon.c calling monitor election 2024-09-19T06:42:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: monmap epoch 3 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: last_changed 2024-09-19T06:42:36.321900+0000 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: min_mon_release 19 (squid) 2024-09-19T06:42:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: election_strategy: 1 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: fsmap 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: mgrmap e13: x(active, since 50s) 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: overall HEALTH_OK 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:41.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:42:41.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: mon.a calling monitor election 2024-09-19T06:42:41.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: mon.c calling monitor election 2024-09-19T06:42:41.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:41.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:41.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: monmap epoch 3 2024-09-19T06:42:41.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:41.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: last_changed 2024-09-19T06:42:36.321900+0000 2024-09-19T06:42:41.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:42:41.759 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: min_mon_release 19 (squid) 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: election_strategy: 1 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: fsmap 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: mgrmap e13: x(active, since 50s) 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: overall HEALTH_OK 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:41.760 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:41.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.761+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6ecc1a6b00 con 0x7f6eb8003530 2024-09-19T06:42:41.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.762+0000 7f6ec97fa640 1 -- 172.21.15.118:0/4034202049 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f6ebc03d380 con 0x7f6eb8003530 2024-09-19T06:42:41.764 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:42:41.764 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":3,"fsid":"fface640-7651-11ef-bceb-c7b262605968","modified":"2024-09-19T06:42:36.321900Z","created":"2024-09-19T06:40:54.002091Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-19T06:42:41.765 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 3 2024-09-19T06:42:41.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6eac039730 msgr2=0x7f6eac03bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6eac039730 0x7f6eac03bbf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009100 tx=0x7f6eb4008040 comp rx=0 tx=0).stop 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 msgr2=0x7f6ecc102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 0x7f6ecc102960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc036000 tx=0x7f6ebc004d30 comp rx=0 tx=0).stop 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 shutdown_connections 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6eac039730 0x7f6eac03bbf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ecc1a3520 0x7f6ecc1a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6ecc102ea0 0x7f6ecc103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 --2- 172.21.15.118:0/4034202049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6eb8003530 0x7f6ecc102960 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.764+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 >> 172.21.15.118:0/4034202049 conn(0x7f6ecc0fc280 msgr2=0x7f6ecc105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.765+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 shutdown_connections 2024-09-19T06:42:41.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:41.765+0000 7f6ed20bd640 1 -- 172.21.15.118:0/4034202049 wait complete. 2024-09-19T06:42:42.289 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-19T06:42:42.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-19T06:42:42.544 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:42.544 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:42 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/4034202049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:42.544 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:42 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/4034202049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:43.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.185+0000 7f37f1c00640 1 -- 172.21.15.114:0/1152959928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 msgr2=0x7f37ec107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:43.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.185+0000 7f37f1c00640 1 --2- 172.21.15.114:0/1152959928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 0x7f37ec107a70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009880 tx=0x7f37dc02a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.186+0000 7f37f1c00640 1 -- 172.21.15.114:0/1152959928 shutdown_connections 2024-09-19T06:42:43.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.186+0000 7f37f1c00640 1 --2- 172.21.15.114:0/1152959928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 0x7f37ec107a70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.186+0000 7f37f1c00640 1 -- 172.21.15.114:0/1152959928 >> 172.21.15.114:0/1152959928 conn(0x7f37ec076250 msgr2=0x7f37ec076660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:43.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.186+0000 7f37f1c00640 1 -- 172.21.15.114:0/1152959928 shutdown_connections 2024-09-19T06:42:43.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.186+0000 7f37f1c00640 1 -- 172.21.15.114:0/1152959928 wait complete. 2024-09-19T06:42:43.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37f1c00640 1 Processor -- start 2024-09-19T06:42:43.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37f1c00640 1 -- start start 2024-09-19T06:42:43.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37f1c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 0x7f37ec1995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:43.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37f1c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec199af0 0x7f37ec19de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:43.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37f1c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 0x7f37ec19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:43.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37f1c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec10bc90 con 0x7f37ec107690 2024-09-19T06:42:43.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37f1c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec10bb10 con 0x7f37ec19e3c0 2024-09-19T06:42:43.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37f1c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec10be10 con 0x7f37ec199af0 2024-09-19T06:42:43.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37eaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 0x7f37ec1995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.187+0000 7f37ea000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec199af0 0x7f37ec19de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37eaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 0x7f37ec1995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40484/0 (socket says 172.21.15.114:40484) 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37eaa00640 1 -- 172.21.15.114:0/1256632400 learned_addr learned my addr 172.21.15.114:0/1256632400 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37ea000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec199af0 0x7f37ec19de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:49264/0 (socket says 172.21.15.114:49264) 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37eb400640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 0x7f37ec19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37eb400640 1 -- 172.21.15.114:0/1256632400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 msgr2=0x7f37ec19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37eb400640 1 -- 172.21.15.114:0/1256632400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 msgr2=0x7f37ec19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37eb400640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 0x7f37ec19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37eb400640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 0x7f37ec19e870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37ea000640 1 -- 172.21.15.114:0/1256632400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 msgr2=0x7f37ec19e870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37ea000640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 0x7f37ec19e870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37ea000640 1 -- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 msgr2=0x7f37ec1995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37ea000640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 0x7f37ec1995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37ea000640 1 -- 172.21.15.114:0/1256632400 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec1a6130 con 0x7f37ec199af0 2024-09-19T06:42:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.188+0000 7f37ea000640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec199af0 0x7f37ec19de80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f37d800b450 tx=0x7f37d800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.189+0000 7f37e3400640 1 -- 172.21.15.114:0/1256632400 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8004230 con 0x7f37ec199af0 2024-09-19T06:42:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.189+0000 7f37e3400640 1 -- 172.21.15.114:0/1256632400 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37d80043d0 con 0x7f37ec199af0 2024-09-19T06:42:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.189+0000 7f37e3400640 1 -- 172.21.15.114:0/1256632400 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8003cb0 con 0x7f37ec199af0 2024-09-19T06:42:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.189+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ec1a6360 con 0x7f37ec199af0 2024-09-19T06:42:43.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.189+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ec1a68a0 con 0x7f37ec199af0 2024-09-19T06:42:43.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.190+0000 7f37e3400640 1 -- 172.21.15.114:0/1256632400 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f37d800f050 con 0x7f37ec199af0 2024-09-19T06:42:43.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.190+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37ec107af0 con 0x7f37ec199af0 2024-09-19T06:42:43.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.190+0000 7f37e3400640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37d0039730 0x7f37d003bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:43.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.191+0000 7f37e3400640 1 -- 172.21.15.114:0/1256632400 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f37d804bff0 con 0x7f37ec199af0 2024-09-19T06:42:43.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.191+0000 7f37eaa00640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37d0039730 0x7f37d003bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:43.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.191+0000 7f37eaa00640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37d0039730 0x7f37d003bbf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0096c0 tx=0x7f37dc02aad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:43.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.196+0000 7f37e3400640 1 -- 172.21.15.114:0/1256632400 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f37d800d040 con 0x7f37ec199af0 2024-09-19T06:42:43.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.437+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f37ec02d7c0 con 0x7f37ec199af0 2024-09-19T06:42:43.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.438+0000 7f37e3400640 1 -- 172.21.15.114:0/1256632400 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f37d8011db0 con 0x7f37ec199af0 2024-09-19T06:42:43.439 INFO:teuthology.orchestra.run.smithi114.stdout:# minimal ceph.conf for fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:43.440 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-09-19T06:42:43.440 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:43.440 INFO:teuthology.orchestra.run.smithi114.stdout: mon_host = [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37d0039730 msgr2=0x7f37d003bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37d0039730 0x7f37d003bbf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0096c0 tx=0x7f37dc02aad0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec199af0 msgr2=0x7f37ec19de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec199af0 0x7f37ec19de80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f37d800b450 tx=0x7f37d800b910 comp rx=0 tx=0).stop 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 shutdown_connections 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37d0039730 0x7f37d003bbf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37ec19e3c0 0x7f37ec19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37ec199af0 0x7f37ec19de80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 --2- 172.21.15.114:0/1256632400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37ec107690 0x7f37ec1995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 >> 172.21.15.114:0/1256632400 conn(0x7f37ec076250 msgr2=0x7f37ec1039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:43.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 shutdown_connections 2024-09-19T06:42:43.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:43.440+0000 7f37f1c00640 1 -- 172.21.15.114:0/1256632400 wait complete. 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: mon.a calling monitor election 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: mon.c calling monitor election 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: monmap epoch 3 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: last_changed 2024-09-19T06:42:36.321900+0000 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: min_mon_release 19 (squid) 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: election_strategy: 1 2024-09-19T06:42:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: fsmap 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: mgrmap e13: x(active, since 50s) 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: overall HEALTH_OK 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/4034202049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-19T06:42:43.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:44.223 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-19T06:42:44.223 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:42:44.224 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-09-19T06:42:44.287 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:42:44.287 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:44.383 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:42:44.384 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-09-19T06:42:44.406 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: mon.b calling monitor election 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: mon.b calling monitor election 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: mon.c calling monitor election 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: mon.a calling monitor election 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: monmap epoch 3 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: last_changed 2024-09-19T06:42:36.321900+0000 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: min_mon_release 19 (squid) 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: election_strategy: 1 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:42:44.407 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: fsmap 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: mgrmap e13: x(active, since 52s) 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: overall HEALTH_OK 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1256632400' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.408 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.411 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:42:44.411 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: mon.b calling monitor election 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: mon.b calling monitor election 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: mon.c calling monitor election 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: mon.a calling monitor election 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: monmap epoch 3 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: last_changed 2024-09-19T06:42:36.321900+0000 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: min_mon_release 19 (squid) 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: election_strategy: 1 2024-09-19T06:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: fsmap 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: mgrmap e13: x(active, since 52s) 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: overall HEALTH_OK 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1256632400' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.477 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:42:44.477 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.conf 2024-09-19T06:42:44.500 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: mon.b calling monitor election 2024-09-19T06:42:44.500 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: mon.b calling monitor election 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: mon.c calling monitor election 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: mon.a calling monitor election 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: monmap epoch 3 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: last_changed 2024-09-19T06:42:36.321900+0000 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: min_mon_release 19 (squid) 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: election_strategy: 1 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-09-19T06:42:44.501 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: fsmap 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: osdmap e4: 0 total, 0 up, 0 in 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: mgrmap e13: x(active, since 52s) 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: overall HEALTH_OK 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1256632400' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.502 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:44 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:44.504 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:42:44.505 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-19T06:42:44.570 INFO:tasks.cephadm:Adding mgr.x on smithi114 2024-09-19T06:42:44.570 INFO:tasks.cephadm:Adding mgr.y on smithi116 2024-09-19T06:42:44.570 INFO:tasks.cephadm:Adding mgr.z on smithi118 2024-09-19T06:42:44.571 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch apply mgr '3;smithi114=x;smithi116=y;smithi118=z' 2024-09-19T06:42:44.845 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:42:45.241 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: Updating smithi118:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: Updating smithi116:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: Updating smithi114:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:45.242 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 -- 172.21.15.118:0/1976129281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 msgr2=0x7ff0081009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 --2- 172.21.15.118:0/1976129281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 0x7ff0081009f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff004009880 tx=0x7ff00402a5d0 comp rx=0 tx=0).stop 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 -- 172.21.15.118:0/1976129281 shutdown_connections 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 --2- 172.21.15.118:0/1976129281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 0x7ff0081009f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 -- 172.21.15.118:0/1976129281 >> 172.21.15.118:0/1976129281 conn(0x7ff0080fbda0 msgr2=0x7ff0080fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 -- 172.21.15.118:0/1976129281 shutdown_connections 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 -- 172.21.15.118:0/1976129281 wait complete. 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 Processor -- start 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.322+0000 7ff00efaa640 1 -- start start 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00efaa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 0x7ff00818f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:45.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00efaa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff00818fc50 0x7ff00818c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:45.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00efaa640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff00818c6b0 0x7ff0081a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:45.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00efaa640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff008106700 con 0x7ff0081005f0 2024-09-19T06:42:45.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00efaa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff008106580 con 0x7ff00818fc50 2024-09-19T06:42:45.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00efaa640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff008106880 con 0x7ff00818c6b0 2024-09-19T06:42:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00cd46640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 0x7ff00818f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00d547640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff00818c6b0 0x7ff0081a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00cd46640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 0x7ff00818f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.118:38856/0 (socket says 172.21.15.118:38856) 2024-09-19T06:42:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00cd46640 1 -- 172.21.15.118:0/3822106649 learned_addr learned my addr 172.21.15.118:0/3822106649 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:42:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7feffffff640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff00818fc50 0x7ff00818c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00cd46640 1 -- 172.21.15.118:0/3822106649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff00818c6b0 msgr2=0x7ff0081a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00cd46640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff00818c6b0 0x7ff0081a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00cd46640 1 -- 172.21.15.118:0/3822106649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff00818fc50 msgr2=0x7ff00818c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00cd46640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff00818fc50 0x7ff00818c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7ff00cd46640 1 -- 172.21.15.118:0/3822106649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff008198df0 con 0x7ff0081005f0 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.323+0000 7feffffff640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff00818fc50 0x7ff00818c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.324+0000 7ff00cd46640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 0x7ff00818f710 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff004005ab0 tx=0x7ff004032d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.324+0000 7feffdffb640 1 -- 172.21.15.118:0/3822106649 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00403f030 con 0x7ff0081005f0 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.324+0000 7feffdffb640 1 -- 172.21.15.118:0/3822106649 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff004003ce0 con 0x7ff0081005f0 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.324+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff008199080 con 0x7ff0081005f0 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.324+0000 7feffdffb640 1 -- 172.21.15.118:0/3822106649 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff004035070 con 0x7ff0081005f0 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.324+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0081995e0 con 0x7ff0081005f0 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.325+0000 7feffdffb640 1 -- 172.21.15.118:0/3822106649 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7ff004042030 con 0x7ff0081005f0 2024-09-19T06:42:45.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.325+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff00818cdc0 con 0x7ff0081005f0 2024-09-19T06:42:45.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.325+0000 7feffdffb640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feff0039730 0x7feff003bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:45.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.325+0000 7feffdffb640 1 -- 172.21.15.118:0/3822106649 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff00406cd50 con 0x7ff0081005f0 2024-09-19T06:42:45.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.325+0000 7feffffff640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feff0039730 0x7feff003bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:45.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.326+0000 7feffffff640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feff0039730 0x7feff003bbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff00818d690 tx=0x7feff8009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:45.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.331+0000 7feffdffb640 1 -- 172.21.15.118:0/3822106649 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff00403bd90 con 0x7ff0081005f0 2024-09-19T06:42:45.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:42:45.321+0000 7fe924400640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: Updating smithi118:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: Updating smithi116:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: Updating smithi114:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: Updating smithi118:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: Updating smithi116:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: Updating smithi114:/var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-19T06:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:42:45.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.509+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi116=y;smithi118=z", "target": ["mon-mgr", ""]}) -- 0x7ff008062630 con 0x7feff0039730 2024-09-19T06:42:45.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.515+0000 7feffdffb640 1 -- 172.21.15.118:0/3822106649 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff008062630 con 0x7feff0039730 2024-09-19T06:42:45.517 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled mgr update... 2024-09-19T06:42:45.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feff0039730 msgr2=0x7feff003bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:45.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feff0039730 0x7feff003bbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff00818d690 tx=0x7feff8009190 comp rx=0 tx=0).stop 2024-09-19T06:42:45.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 msgr2=0x7ff00818f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:45.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 0x7ff00818f710 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff004005ab0 tx=0x7ff004032d00 comp rx=0 tx=0).stop 2024-09-19T06:42:45.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 shutdown_connections 2024-09-19T06:42:45.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feff0039730 0x7feff003bbf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:45.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff00818c6b0 0x7ff0081a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:45.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff00818fc50 0x7ff00818c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:45.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 --2- 172.21.15.118:0/3822106649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff0081005f0 0x7ff00818f710 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:45.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 >> 172.21.15.118:0/3822106649 conn(0x7ff0080fbda0 msgr2=0x7ff0081901b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:45.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 shutdown_connections 2024-09-19T06:42:45.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:42:45.517+0000 7ff00efaa640 1 -- 172.21.15.118:0/3822106649 wait complete. 2024-09-19T06:42:46.065 DEBUG:teuthology.orchestra.run.smithi116:mgr.y> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mgr.y.service 2024-09-19T06:42:46.068 DEBUG:teuthology.orchestra.run.smithi118:mgr.z> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mgr.z.service 2024-09-19T06:42:46.070 INFO:tasks.cephadm:Deploying OSDs... 2024-09-19T06:42:46.071 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:42:46.071 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-09-19T06:42:46.107 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-19T06:42:46.107 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-19 06:42:04.344517682 +0000 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-19 06:37:41.061003857 +0000 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-19 06:37:41.061003857 +0000 2024-09-19T06:42:46.185 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-19 06:37:41.061003857 +0000 2024-09-19T06:42:46.186 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-19T06:42:46.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: Reconfiguring mon.a (unknown last config time)... 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: Reconfiguring daemon mon.a on smithi114 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:46.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: Reconfiguring mon.a (unknown last config time)... 2024-09-19T06:42:46.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: Reconfiguring daemon mon.a on smithi114 2024-09-19T06:42:46.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:46.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:46.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:46.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:46.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.280 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-19T06:42:46.280 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-19T06:42:46.280 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000294365 s, 1.7 MB/s 2024-09-19T06:42:46.283 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-19T06:42:46.323 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-09-19T06:42:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-19T06:42:46.399 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:46.399 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-09-19T06:42:46.399 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:46.400 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:46.400 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-19 06:42:04.390516719 +0000 2024-09-19T06:42:46.400 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-19 06:37:41.541993800 +0000 2024-09-19T06:42:46.400 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-19 06:37:41.541993800 +0000 2024-09-19T06:42:46.400 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-19 06:37:41.541993800 +0000 2024-09-19T06:42:46.400 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-19T06:42:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: Reconfiguring mon.a (unknown last config time)... 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: Reconfiguring daemon mon.a on smithi114 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:46.496 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-19T06:42:46.497 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-19T06:42:46.497 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000276449 s, 1.9 MB/s 2024-09-19T06:42:46.498 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-19T06:42:46.535 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-09-19T06:42:46.610 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-19T06:42:46.610 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:46.610 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-09-19T06:42:46.610 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:46.611 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:46.611 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-19 06:42:04.436515757 +0000 2024-09-19T06:42:46.611 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-19 06:37:42.002984162 +0000 2024-09-19T06:42:46.611 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-19 06:37:42.002984162 +0000 2024-09-19T06:42:46.611 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-19 06:37:42.002984162 +0000 2024-09-19T06:42:46.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-19T06:42:46.705 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-19T06:42:46.705 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-19T06:42:46.705 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000265427 s, 1.9 MB/s 2024-09-19T06:42:46.707 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-19T06:42:46.745 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-09-19T06:42:46.823 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-19T06:42:46.823 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:46.823 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-09-19T06:42:46.823 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:46.823 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:46.823 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-19 06:42:04.480514836 +0000 2024-09-19T06:42:46.823 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-19 06:37:42.456974670 +0000 2024-09-19T06:42:46.823 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-19 06:37:42.456974670 +0000 2024-09-19T06:42:46.824 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-19 06:37:42.456974670 +0000 2024-09-19T06:42:46.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-19T06:42:46.917 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-09-19T06:42:46.917 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-09-19T06:42:46.917 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000299263 s, 1.7 MB/s 2024-09-19T06:42:46.920 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-19T06:42:46.959 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-09-19T06:42:46.959 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-09-19T06:42:46.979 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-19T06:42:46.979 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-09-19T06:42:47.037 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-19T06:42:47.037 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:47.038 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-19T06:42:47.038 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:47.038 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:47.038 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-09-19 06:42:35.783908818 +0000 2024-09-19T06:42:47.038 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-09-19 06:37:09.553103495 +0000 2024-09-19T06:42:47.038 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-09-19 06:37:09.553103495 +0000 2024-09-19T06:42:47.038 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-09-19 06:37:09.553103495 +0000 2024-09-19T06:42:47.038 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-19T06:42:47.105 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-09-19T06:42:47.105 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-09-19T06:42:47.105 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000229137 s, 2.2 MB/s 2024-09-19T06:42:47.107 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-19T06:42:47.165 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-09-19T06:42:47.224 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-19T06:42:47.224 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:47.224 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-09-19T06:42:47.224 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:47.225 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:47.225 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-09-19 06:42:35.783908818 +0000 2024-09-19T06:42:47.225 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-09-19 06:37:09.889098064 +0000 2024-09-19T06:42:47.225 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-09-19 06:37:09.889098064 +0000 2024-09-19T06:42:47.225 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-09-19 06:37:09.889098064 +0000 2024-09-19T06:42:47.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-19T06:42:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-09-19T06:42:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-09-19T06:42:47.290 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00024743 s, 2.1 MB/s 2024-09-19T06:42:47.291 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-19T06:42:47.348 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-09-19T06:42:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-19T06:42:47.405 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:47.405 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-19T06:42:47.405 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:47.405 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:47.405 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-09-19 06:42:35.784908803 +0000 2024-09-19T06:42:47.405 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-09-19 06:37:10.228092585 +0000 2024-09-19T06:42:47.405 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-09-19 06:37:10.228092585 +0000 2024-09-19T06:42:47.406 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-09-19 06:37:10.228092585 +0000 2024-09-19T06:42:47.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-19T06:42:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:47 smithi114 ceph-mon[30820]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi116=y;smithi118=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:47 smithi114 ceph-mon[30820]: Saving service mgr spec with placement smithi114=x;smithi116=y;smithi118=z;count:3 2024-09-19T06:42:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:47 smithi114 ceph-mon[30820]: Reconfiguring mgr.x (unknown last config time)... 2024-09-19T06:42:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:47 smithi114 ceph-mon[30820]: Reconfiguring daemon mgr.x on smithi114 2024-09-19T06:42:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:47 smithi116 ceph-mon[35411]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi116=y;smithi118=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:47 smithi116 ceph-mon[35411]: Saving service mgr spec with placement smithi114=x;smithi116=y;smithi118=z;count:3 2024-09-19T06:42:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:47 smithi116 ceph-mon[35411]: Reconfiguring mgr.x (unknown last config time)... 2024-09-19T06:42:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:47 smithi116 ceph-mon[35411]: Reconfiguring daemon mgr.x on smithi114 2024-09-19T06:42:47.452 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-09-19T06:42:47.452 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-09-19T06:42:47.453 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000196479 s, 2.6 MB/s 2024-09-19T06:42:47.454 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-19T06:42:47.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:47 smithi118 ceph-mon[35527]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi116=y;smithi118=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:47.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:47 smithi118 ceph-mon[35527]: Saving service mgr spec with placement smithi114=x;smithi116=y;smithi118=z;count:3 2024-09-19T06:42:47.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:47 smithi118 ceph-mon[35527]: Reconfiguring mgr.x (unknown last config time)... 2024-09-19T06:42:47.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:47 smithi118 ceph-mon[35527]: Reconfiguring daemon mgr.x on smithi114 2024-09-19T06:42:47.511 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-09-19T06:42:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-19T06:42:47.567 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:47.567 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-19T06:42:47.567 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:47.567 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:47.567 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-09-19 06:42:35.784908803 +0000 2024-09-19T06:42:47.568 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-09-19 06:37:10.555087300 +0000 2024-09-19T06:42:47.568 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-09-19 06:37:10.555087300 +0000 2024-09-19T06:42:47.568 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-09-19 06:37:10.555087300 +0000 2024-09-19T06:42:47.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-19T06:42:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-09-19T06:42:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-09-19T06:42:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000212067 s, 2.4 MB/s 2024-09-19T06:42:47.632 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-19T06:42:47.690 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-19T06:42:47.690 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-09-19T06:42:47.709 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-19T06:42:47.709 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-09-19T06:42:47.768 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-19T06:42:47.768 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:47.768 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-19T06:42:47.768 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:47.769 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:47.769 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-19 06:42:26.725312679 +0000 2024-09-19T06:42:47.769 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-19 06:37:14.774420915 +0000 2024-09-19T06:42:47.769 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-19 06:37:14.774420915 +0000 2024-09-19T06:42:47.769 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-09-19 06:37:14.774420915 +0000 2024-09-19T06:42:47.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-19T06:42:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-19T06:42:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-19T06:42:47.836 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000135203 s, 3.8 MB/s 2024-09-19T06:42:47.837 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-19T06:42:47.898 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-09-19T06:42:47.956 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-19T06:42:47.956 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:47.956 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-09-19T06:42:47.956 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:47.956 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:47.956 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-19 06:42:26.725312679 +0000 2024-09-19T06:42:47.957 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-19 06:37:15.079414078 +0000 2024-09-19T06:42:47.957 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-19 06:37:15.079414078 +0000 2024-09-19T06:42:47.957 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-09-19 06:37:15.079414078 +0000 2024-09-19T06:42:47.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-19T06:42:48.022 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-19T06:42:48.022 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-19T06:42:48.022 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000138473 s, 3.7 MB/s 2024-09-19T06:42:48.023 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-19T06:42:48.081 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-19 06:42:26.726312657 +0000 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-19 06:37:15.403406816 +0000 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-19 06:37:15.403406816 +0000 2024-09-19T06:42:48.138 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-09-19 06:37:15.403406816 +0000 2024-09-19T06:42:48.139 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-19T06:42:48.201 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-19T06:42:48.202 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-19T06:42:48.202 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000110125 s, 4.6 MB/s 2024-09-19T06:42:48.203 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-19T06:42:48.259 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-19 06:42:26.726312657 +0000 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-19 06:37:15.713399867 +0000 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-19 06:37:15.713399867 +0000 2024-09-19T06:42:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-09-19 06:37:15.713399867 +0000 2024-09-19T06:42:48.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-19T06:42:48.378 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-19T06:42:48.378 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-19T06:42:48.378 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000108159 s, 4.7 MB/s 2024-09-19T06:42:48.379 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-19T06:42:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:48 smithi114 ceph-mon[30820]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:48 smithi116 ceph-mon[35411]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:48.436 INFO:tasks.cephadm:Deploying osd.0 on smithi114 with /dev/vg_nvme/lv_4... 2024-09-19T06:42:48.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-19T06:42:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:48 smithi118 ceph-mon[35527]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:48 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:48 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:48 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:48 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:48.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:48 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:48.789 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:49.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:49 smithi114 ceph-mon[30820]: Reconfiguring mon.b (monmap changed)... 2024-09-19T06:42:49.193 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:49 smithi114 ceph-mon[30820]: Reconfiguring daemon mon.b on smithi116 2024-09-19T06:42:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:49 smithi116 ceph-mon[35411]: Reconfiguring mon.b (monmap changed)... 2024-09-19T06:42:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:49 smithi116 ceph-mon[35411]: Reconfiguring daemon mon.b on smithi116 2024-09-19T06:42:49.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:49 smithi118 ceph-mon[35527]: Reconfiguring mon.b (monmap changed)... 2024-09-19T06:42:49.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:49 smithi118 ceph-mon[35527]: Reconfiguring daemon mon.b on smithi116 2024-09-19T06:42:50.247 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:50 smithi114 ceph-mon[30820]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:50.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:50.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:50.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:50 smithi114 ceph-mon[30820]: Reconfiguring mon.c (monmap changed)... 2024-09-19T06:42:50.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:50.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:50.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:50.248 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:50 smithi114 ceph-mon[30820]: Reconfiguring daemon mon.c on smithi118 2024-09-19T06:42:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:50 smithi118 ceph-mon[35527]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:50 smithi118 ceph-mon[35527]: Reconfiguring mon.c (monmap changed)... 2024-09-19T06:42:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:50.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:50.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:50.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:50 smithi118 ceph-mon[35527]: Reconfiguring daemon mon.c on smithi118 2024-09-19T06:42:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:50 smithi116 ceph-mon[35411]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:50 smithi116 ceph-mon[35411]: Reconfiguring mon.c (monmap changed)... 2024-09-19T06:42:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-19T06:42:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-19T06:42:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:50 smithi116 ceph-mon[35411]: Reconfiguring daemon mon.c on smithi118 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T06:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: Deploying daemon mgr.z on smithi118 2024-09-19T06:42:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:51 smithi114 ceph-mon[30820]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: Deploying daemon mgr.z on smithi118 2024-09-19T06:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:51 smithi116 ceph-mon[35411]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:51.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: Deploying daemon mgr.z on smithi118 2024-09-19T06:42:51.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:51 smithi118 ceph-mon[35527]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:51.778 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:42:51.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-09-19T06:42:52.166 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:42:52.454 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 systemd[1]: Starting Ceph mgr.z for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:42:52.715 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 podman[37318]: 2024-09-19 06:42:52.356493587 +0000 UTC m=+0.015598082 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:42:52.716 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 podman[37318]: 2024-09-19 06:42:52.460204162 +0000 UTC m=+0.119308651 container create 99a0012696ac1f4a881fc956e9d9166e79933a6e829ba955851a92bec1df9f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z, RELEASE=squid-2f599a4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-19T06:42:52.716 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 podman[37318]: 2024-09-19 06:42:52.687657324 +0000 UTC m=+0.346761812 container init 99a0012696ac1f4a881fc956e9d9166e79933a6e829ba955851a92bec1df9f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-19T06:42:52.716 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 podman[37318]: 2024-09-19 06:42:52.690472046 +0000 UTC m=+0.349576539 container start 99a0012696ac1f4a881fc956e9d9166e79933a6e829ba955851a92bec1df9f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-19T06:42:52.716 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 ceph-mgr[37337]: -- 172.21.15.118:0/2305437438 <== mon.1 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aad12310e0 con 0x55aad120f400 2024-09-19T06:42:52.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.716+0000 7f9eab400640 1 -- 172.21.15.114:0/3715348963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4102e00 msgr2=0x7f9ea4103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:52.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.716+0000 7f9eab400640 1 --2- 172.21.15.114:0/3715348963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4102e00 0x7f9ea4103280 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009900 tx=0x7f9e8c02a3f0 comp rx=0 tx=0).stop 2024-09-19T06:42:52.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.716+0000 7f9eab400640 1 -- 172.21.15.114:0/3715348963 shutdown_connections 2024-09-19T06:42:52.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.716+0000 7f9eab400640 1 --2- 172.21.15.114:0/3715348963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea41037c0 0x7f9ea410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:52.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.716+0000 7f9eab400640 1 --2- 172.21.15.114:0/3715348963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4102e00 0x7f9ea4103280 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:52.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.716+0000 7f9eab400640 1 --2- 172.21.15.114:0/3715348963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea4101c00 0x7f9ea4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:52.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.716+0000 7f9eab400640 1 -- 172.21.15.114:0/3715348963 >> 172.21.15.114:0/3715348963 conn(0x7f9ea40fd390 msgr2=0x7f9ea40ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:42:52.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.716+0000 7f9eab400640 1 -- 172.21.15.114:0/3715348963 shutdown_connections 2024-09-19T06:42:52.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.717+0000 7f9eab400640 1 -- 172.21.15.114:0/3715348963 wait complete. 2024-09-19T06:42:52.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.717+0000 7f9eab400640 1 Processor -- start 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9eab400640 1 -- start start 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9eab400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4101c00 0x7f9ea419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9eab400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea4102e00 0x7f9ea419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9eab400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea41037c0 0x7f9ea41a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9eab400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea410d880 con 0x7f9ea4101c00 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9eab400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea410d700 con 0x7f9ea41037c0 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9eab400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea410da00 con 0x7f9ea4102e00 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9ea8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4101c00 0x7f9ea419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea4102e00 0x7f9ea419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9ea8a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4101c00 0x7f9ea419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33968/0 (socket says 172.21.15.114:33968) 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea4102e00 0x7f9ea419a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:57880/0 (socket says 172.21.15.114:57880) 2024-09-19T06:42:52.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9ea8a00640 1 -- 172.21.15.114:0/4268728529 learned_addr learned my addr 172.21.15.114:0/4268728529 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.718+0000 7f9ea9400640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea41037c0 0x7f9ea41a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea8a00640 1 -- 172.21.15.114:0/4268728529 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea4102e00 msgr2=0x7f9ea419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea8a00640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea4102e00 0x7f9ea419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea8a00640 1 -- 172.21.15.114:0/4268728529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea41037c0 msgr2=0x7f9ea41a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea8a00640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea41037c0 0x7f9ea41a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea8a00640 1 -- 172.21.15.114:0/4268728529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea41a1d60 con 0x7f9ea4101c00 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea8a00640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4101c00 0x7f9ea419a130 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400e9c0 tx=0x7f9e9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400cd80 con 0x7f9ea4101c00 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e940035f0 con 0x7f9ea4101c00 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94010690 con 0x7f9ea4101c00 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea41a1ff0 con 0x7f9ea4101c00 2024-09-19T06:42:52.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.719+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea41a2450 con 0x7f9ea4101c00 2024-09-19T06:42:52.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.721+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea41a26e0 con 0x7f9ea4101c00 2024-09-19T06:42:52.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.721+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46518+0+0 (secure 0 0 0) 0x7f9e940037a0 con 0x7f9ea4101c00 2024-09-19T06:42:52.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.721+0000 7f9ea2000640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9e88039500 0x7f9e8803b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:42:52.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.721+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9e9404cab0 con 0x7f9ea4101c00 2024-09-19T06:42:52.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.722+0000 7f9e9be00640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9e88039500 0x7f9e8803b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:42:52.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.722+0000 7f9e9be00640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9e88039500 0x7f9e8803b9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0314e0 tx=0x7f9e8c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:42:52.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.726+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9e9401c1c0 con 0x7f9ea4101c00 2024-09-19T06:42:52.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:52.913+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9ea4109cf0 con 0x7f9e88039500 2024-09-19T06:42:53.008 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 bash[37318]: 99a0012696ac1f4a881fc956e9d9166e79933a6e829ba955851a92bec1df9f58 2024-09-19T06:42:53.008 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 systemd[1]: Started Ceph mgr.z for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:42:53.008 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:52.811+0000 7f0b3526d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T06:42:53.008 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:52 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:52.870+0000 7f0b3526d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T06:42:53.365 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:53.016+0000 7f0b3526d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T06:42:53.365 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:53.080+0000 7f0b3526d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T06:42:53.365 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:53.225+0000 7f0b3526d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:42:53.365 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:53.362+0000 7f0b3526d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T06:42:54.006 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:53.671+0000 7f0b3526d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T06:42:54.006 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:53.731+0000 7f0b3526d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T06:42:54.006 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:53.913+0000 7f0b3526d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: Deploying daemon mgr.y on smithi116 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:53 smithi118 ceph-mon[35527]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:54.148 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: Deploying daemon mgr.y on smithi116 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:54.149 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:53 smithi116 ceph-mon[35411]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: Deploying daemon mgr.y on smithi116 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:53 smithi114 ceph-mon[30820]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:54.679 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:54 smithi116 systemd[1]: Starting Ceph mgr.y for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:42:55.044 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:54 smithi116 podman[36755]: 2024-09-19 06:42:54.783324706 +0000 UTC m=+0.109649533 container create 003d5c462a3f1979a4d63f35cafa186e0211df7bd771eb9b4807a6529cdd6d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T06:42:55.045 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:54 smithi116 podman[36755]: 2024-09-19 06:42:54.689576603 +0000 UTC m=+0.015901440 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:42:55.045 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:54 smithi116 podman[36755]: 2024-09-19 06:42:54.977248394 +0000 UTC m=+0.303573222 container init 003d5c462a3f1979a4d63f35cafa186e0211df7bd771eb9b4807a6529cdd6d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS) 2024-09-19T06:42:55.045 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:54 smithi116 podman[36755]: 2024-09-19 06:42:54.980031944 +0000 UTC m=+0.306356773 container start 003d5c462a3f1979a4d63f35cafa186e0211df7bd771eb9b4807a6529cdd6d8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-19T06:42:55.045 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 ceph-mgr[36774]: -- 172.21.15.116:0/3525734486 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55da19be10e0 con 0x55da19bbf000 2024-09-19T06:42:55.045 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 bash[36755]: 003d5c462a3f1979a4d63f35cafa186e0211df7bd771eb9b4807a6529cdd6d8b 2024-09-19T06:42:55.106 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:54 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:54.788+0000 7f0b3526d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T06:42:55.106 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:54 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:54.856+0000 7f0b3526d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T06:42:55.318 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 systemd[1]: Started Ceph mgr.y for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:42:55.318 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:55.106+0000 7f1912114040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-19T06:42:55.318 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:55.162+0000 7f1912114040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-19T06:42:55.319 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:55.317+0000 7f1912114040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-19T06:42:55.678 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:55.386+0000 7f1912114040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-19T06:42:55.679 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:55.553+0000 7f1912114040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:42:55.894 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:55 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:55.622+0000 7f0b3526d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T06:42:55.894 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:55 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:55.744+0000 7f0b3526d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T06:42:55.894 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:55 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:55.892+0000 7f0b3526d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:42:56.005 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:55 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:55.712+0000 7f1912114040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-19T06:42:56.191 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:55 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:55.959+0000 7f0b3526d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T06:42:56.191 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.111+0000 7f0b3526d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T06:42:56.191 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.188+0000 7f0b3526d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T06:42:56.191 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:56.191 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.191 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1252409490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]: dispatch 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]: dispatch 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]': finished 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: osdmap e5: 1 total, 0 up, 1 in 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:42:56.192 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.303 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:56.039+0000 7f1912114040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-19T06:42:56.303 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:56.101+0000 7f1912114040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-19T06:42:56.303 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:56.304 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.304 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.304 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.304 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.304 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:56.304 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1252409490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]: dispatch 2024-09-19T06:42:56.305 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]: dispatch 2024-09-19T06:42:56.305 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]': finished 2024-09-19T06:42:56.305 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: osdmap e5: 1 total, 0 up, 1 in 2024-09-19T06:42:56.305 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:42:56.305 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1252409490' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]: dispatch 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]: dispatch 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a97979d-58ef-4b4b-a9a1-f703d05eca0d"}]': finished 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: osdmap e5: 1 total, 0 up, 1 in 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:42:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:56.478 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.252+0000 7f0b3526d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T06:42:56.479 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.410+0000 7f0b3526d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T06:42:56.479 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.477+0000 7f0b3526d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T06:42:56.678 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:56 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:56.301+0000 7f1912114040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-19T06:42:56.747 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.543+0000 7f0b3526d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T06:42:56.747 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.628+0000 7f0b3526d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T06:42:56.747 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.687+0000 7f0b3526d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T06:42:56.747 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:56 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:56.745+0000 7f0b3526d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T06:42:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3249993891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:57 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3249993891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:42:57.678 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:57.194+0000 7f1912114040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-19T06:42:57.678 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:57.264+0000 7f1912114040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3249993891' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:42:58.006 INFO:journalctl@ceph.mgr.z.smithi118.stdout:Sep 19 06:42:57 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-z[37333]: 2024-09-19T06:42:57.527+0000 7f0b3526d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T06:42:58.233 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-mon[35411]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:58.233 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-mon[35411]: Standby manager daemon z started 2024-09-19T06:42:58.233 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-mon[35411]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T06:42:58.234 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-mon[35411]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T06:42:58.234 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-mon[35411]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T06:42:58.234 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-mon[35411]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T06:42:58.234 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:57 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:57.976+0000 7f1912114040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-19T06:42:58.234 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.089+0000 7f1912114040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-19T06:42:58.235 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.231+0000 7f1912114040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-19T06:42:58.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:58 smithi114 ceph-mon[30820]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:58.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:58 smithi114 ceph-mon[30820]: Standby manager daemon z started 2024-09-19T06:42:58.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:58 smithi114 ceph-mon[30820]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T06:42:58.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:58 smithi114 ceph-mon[30820]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T06:42:58.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:58 smithi114 ceph-mon[30820]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T06:42:58.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:58 smithi114 ceph-mon[30820]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T06:42:58.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:42:58.475+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 14) ==== 92409+0+0 (secure 0 0 0) 0x7f9e940217c0 con 0x7f9ea4101c00 2024-09-19T06:42:58.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:58 smithi118 ceph-mon[35527]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:58.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:58 smithi118 ceph-mon[35527]: Standby manager daemon z started 2024-09-19T06:42:58.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:58 smithi118 ceph-mon[35527]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-19T06:42:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:58 smithi118 ceph-mon[35527]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T06:42:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:58 smithi118 ceph-mon[35527]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-19T06:42:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:58 smithi118 ceph-mon[35527]: from='mgr.? 172.21.15.118:0/3905399861' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T06:42:58.511 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.292+0000 7f1912114040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-19T06:42:58.511 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.440+0000 7f1912114040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-19T06:42:58.511 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.510+0000 7f1912114040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-19T06:42:58.791 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.569+0000 7f1912114040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-19T06:42:58.792 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.726+0000 7f1912114040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-19T06:42:58.792 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.790+0000 7f1912114040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-19T06:42:59.055 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.856+0000 7f1912114040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-19T06:42:59.056 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.937+0000 7f1912114040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-19T06:42:59.056 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:58.996+0000 7f1912114040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-19T06:42:59.056 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:59 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:59.054+0000 7f1912114040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-19T06:42:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:59 smithi118 ceph-mon[35527]: mgrmap e14: x(active, since 67s), standbys: z 2024-09-19T06:42:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T06:42:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:42:59 smithi118 ceph-mon[35527]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:59.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:59 smithi116 ceph-mon[35411]: mgrmap e14: x(active, since 67s), standbys: z 2024-09-19T06:42:59.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T06:42:59.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:42:59 smithi116 ceph-mon[35411]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:42:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:59 smithi114 ceph-mon[30820]: mgrmap e14: x(active, since 67s), standbys: z 2024-09-19T06:42:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-19T06:42:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:42:59 smithi114 ceph-mon[30820]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:00.178 INFO:journalctl@ceph.mgr.y.smithi116.stdout:Sep 19 06:42:59 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-y[36770]: 2024-09-19T06:42:59.842+0000 7f1912114040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-19T06:43:00.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:00.489+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 15) ==== 138300+0+0 (secure 0 0 0) 0x7f9e9401b820 con 0x7f9ea4101c00 2024-09-19T06:43:00.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:00 smithi114 ceph-mon[30820]: Standby manager daemon y started 2024-09-19T06:43:00.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:00 smithi114 ceph-mon[30820]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T06:43:00.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:00 smithi114 ceph-mon[30820]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T06:43:00.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:00 smithi114 ceph-mon[30820]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T06:43:00.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:00 smithi114 ceph-mon[30820]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T06:43:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:00 smithi118 ceph-mon[35527]: Standby manager daemon y started 2024-09-19T06:43:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:00 smithi118 ceph-mon[35527]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T06:43:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:00 smithi118 ceph-mon[35527]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T06:43:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:00 smithi118 ceph-mon[35527]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T06:43:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:00 smithi118 ceph-mon[35527]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T06:43:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:00 smithi116 ceph-mon[35411]: Standby manager daemon y started 2024-09-19T06:43:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:00 smithi116 ceph-mon[35411]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-19T06:43:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:00 smithi116 ceph-mon[35411]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-19T06:43:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:00 smithi116 ceph-mon[35411]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-19T06:43:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:00 smithi116 ceph-mon[35411]: from='mgr.? 172.21.15.116:0/2005943080' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-19T06:43:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:01 smithi118 ceph-mon[35527]: mgrmap e15: x(active, since 69s), standbys: z, y 2024-09-19T06:43:01.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:01 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T06:43:01.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:01 smithi118 ceph-mon[35527]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:01.780 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:01 smithi114 ceph-mon[30820]: mgrmap e15: x(active, since 69s), standbys: z, y 2024-09-19T06:43:01.780 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T06:43:01.780 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:01 smithi114 ceph-mon[30820]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:01 smithi116 ceph-mon[35411]: mgrmap e15: x(active, since 69s), standbys: z, y 2024-09-19T06:43:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-19T06:43:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:01 smithi116 ceph-mon[35411]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:04 smithi114 ceph-mon[30820]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:04 smithi116 ceph-mon[35411]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:04.459 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:04 smithi118 ceph-mon[35527]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-19T06:43:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-19T06:43:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:05.471 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-19T06:43:05.471 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:06 smithi114 ceph-mon[30820]: Deploying daemon osd.0 on smithi114 2024-09-19T06:43:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:06 smithi114 ceph-mon[30820]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:06 smithi116 ceph-mon[35411]: Deploying daemon osd.0 on smithi114 2024-09-19T06:43:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:06 smithi116 ceph-mon[35411]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:06 smithi118 ceph-mon[35527]: Deploying daemon osd.0 on smithi114 2024-09-19T06:43:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:06 smithi118 ceph-mon[35527]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:08.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:08 smithi114 ceph-mon[30820]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:08 smithi116 ceph-mon[35411]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:08.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:08 smithi118 ceph-mon[35527]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:10 smithi114 ceph-mon[30820]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:10 smithi116 ceph-mon[35411]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:10.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:10 smithi118 ceph-mon[35527]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:12.221 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:12 smithi114 ceph-mon[30820]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:12 smithi116 ceph-mon[35411]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:12.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:12 smithi118 ceph-mon[35527]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:13.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:13.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:13.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:14.340 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:14 smithi114 ceph-mon[30820]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:14 smithi116 ceph-mon[35411]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:14.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:14 smithi118 ceph-mon[35527]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:16 smithi114 ceph-mon[30820]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:16 smithi116 ceph-mon[35411]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:16 smithi118 ceph-mon[35527]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:16.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:16.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:17.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.884+0000 7f9ea2000640 1 -- 172.21.15.114:0/4268728529 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9ea4109cf0 con 0x7f9e88039500 2024-09-19T06:43:17.886 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 0 on host 'smithi114' 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9e88039500 msgr2=0x7f9e8803b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9e88039500 0x7f9e8803b9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0314e0 tx=0x7f9e8c000bc0 comp rx=0 tx=0).stop 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4101c00 msgr2=0x7f9ea419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4101c00 0x7f9ea419a130 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400e9c0 tx=0x7f9e9400ee80 comp rx=0 tx=0).stop 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 shutdown_connections 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9e88039500 0x7f9e8803b9c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea41037c0 0x7f9ea41a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ea4102e00 0x7f9ea419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 --2- 172.21.15.114:0/4268728529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ea4101c00 0x7f9ea419a130 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:17.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 >> 172.21.15.114:0/4268728529 conn(0x7f9ea40fd390 msgr2=0x7f9ea41085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:43:17.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 shutdown_connections 2024-09-19T06:43:17.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:43:17.887+0000 7f9eab400640 1 -- 172.21.15.114:0/4268728529 wait complete. 2024-09-19T06:43:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:18 smithi114 ceph-mon[30820]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:18 smithi116 ceph-mon[35411]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:18 smithi118 ceph-mon[35527]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:18.705 DEBUG:teuthology.orchestra.run.smithi114:osd.0> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.0.service 2024-09-19T06:43:18.709 INFO:tasks.cephadm:Deploying osd.1 on smithi116 with /dev/vg_nvme/lv_4... 2024-09-19T06:43:18.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-19T06:43:18.962 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T06:43:19.666 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:43:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-0[44399]: 2024-09-19T06:43:19.406+0000 7f3bd241d640 -1 osd.0 0 log_to_monitors true 2024-09-19T06:43:20.143 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:20 smithi116 ceph-mon[35411]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:20.143 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:20 smithi116 ceph-mon[35411]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-19T06:43:20.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:20 smithi114 ceph-mon[30820]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:20.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:20 smithi114 ceph-mon[30820]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-19T06:43:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:20 smithi118 ceph-mon[35527]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:20 smithi118 ceph-mon[35527]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-19T06:43:21.190 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:21 smithi116 ceph-mon[35411]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-19T06:43:21.190 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:21 smithi116 ceph-mon[35411]: osdmap e6: 1 total, 0 up, 1 in 2024-09-19T06:43:21.191 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:21 smithi116 ceph-mon[35411]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:43:21.191 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:21.191 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:21 smithi116 ceph-mon[35411]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:43:21.191 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:21 smithi116 ceph-mon[35411]: osdmap e7: 1 total, 0 up, 1 in 2024-09-19T06:43:21.204 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:43:21.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-09-19T06:43:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:21 smithi114 ceph-mon[30820]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-19T06:43:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:21 smithi114 ceph-mon[30820]: osdmap e6: 1 total, 0 up, 1 in 2024-09-19T06:43:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:21 smithi114 ceph-mon[30820]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:43:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:21 smithi114 ceph-mon[30820]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:43:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:21 smithi114 ceph-mon[30820]: osdmap e7: 1 total, 0 up, 1 in 2024-09-19T06:43:21.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:43:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-0[44399]: 2024-09-19T06:43:21.112+0000 7f3bcd000640 -1 osd.0 0 waiting for initial osdmap 2024-09-19T06:43:21.418 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:43:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-0[44399]: 2024-09-19T06:43:21.115+0000 7f3bc8000640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:43:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T06:43:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:21 smithi118 ceph-mon[35527]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-19T06:43:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:21 smithi118 ceph-mon[35527]: osdmap e6: 1 total, 0 up, 1 in 2024-09-19T06:43:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:21 smithi118 ceph-mon[35527]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:43:21.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:21 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:21.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:21 smithi118 ceph-mon[35527]: from='osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:43:21.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:21 smithi118 ceph-mon[35527]: osdmap e7: 1 total, 0 up, 1 in 2024-09-19T06:43:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.921+0000 7f372c7a9640 1 -- 172.21.15.116:0/1467833863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724101d00 msgr2=0x7f3724102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.921+0000 7f372c7a9640 1 --2- 172.21.15.116:0/1467833863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724101d00 0x7f3724102180 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3714009900 tx=0x7f371402a410 comp rx=0 tx=0).stop 2024-09-19T06:43:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 -- 172.21.15.116:0/1467833863 shutdown_connections 2024-09-19T06:43:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 --2- 172.21.15.116:0/1467833863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37241026c0 0x7f3724108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 --2- 172.21.15.116:0/1467833863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724101d00 0x7f3724102180 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 --2- 172.21.15.116:0/1467833863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3724100b00 0x7f3724100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 -- 172.21.15.116:0/1467833863 >> 172.21.15.116:0/1467833863 conn(0x7f37240fc270 msgr2=0x7f37240fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:43:21.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 -- 172.21.15.116:0/1467833863 shutdown_connections 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 -- 172.21.15.116:0/1467833863 wait complete. 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 Processor -- start 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 -- start start 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724100b00 0x7f372406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3724101d00 0x7f372419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37241026c0 0x7f37241a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f372410c800 con 0x7f3724101d00 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.922+0000 7f372c7a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f372410c680 con 0x7f3724100b00 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372c7a9640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f372410c980 con 0x7f37241026c0 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724100b00 0x7f372406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:43:21.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724100b00 0x7f372406b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48586/0 (socket says 172.21.15.116:48586) 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 -- 172.21.15.116:0/3156123685 learned_addr learned my addr 172.21.15.116:0/3156123685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f3729d44640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3724101d00 0x7f372419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372ad46640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37241026c0 0x7f37241a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 -- 172.21.15.116:0/3156123685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37241026c0 msgr2=0x7f37241a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37241026c0 0x7f37241a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 -- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3724101d00 msgr2=0x7f372419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3724101d00 0x7f372419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 -- 172.21.15.116:0/3156123685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37241a1eb0 con 0x7f3724100b00 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f3729d44640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3724101d00 0x7f372419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372a545640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724100b00 0x7f372406b2f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f372000d950 tx=0x7f372000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:43:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f371b7fe640 1 -- 172.21.15.116:0/3156123685 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720017070 con 0x7f3724100b00 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f371b7fe640 1 -- 172.21.15.116:0/3156123685 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3720009c30 con 0x7f3724100b00 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37241a2140 con 0x7f3724100b00 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f371b7fe640 1 -- 172.21.15.116:0/3156123685 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372000f600 con 0x7f3724100b00 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.923+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37241a2680 con 0x7f3724100b00 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.924+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3724107fc0 con 0x7f3724100b00 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.925+0000 7f371b7fe640 1 -- 172.21.15.116:0/3156123685 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) ==== 138300+0+0 (secure 0 0 0) 0x7f372001a070 con 0x7f3724100b00 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.926+0000 7f371b7fe640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37100a31e0 0x7f37100a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.926+0000 7f371b7fe640 1 -- 172.21.15.116:0/3156123685 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1487+0+0 (secure 0 0 0) 0x7f37200cd850 con 0x7f3724100b00 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.926+0000 7f3729d44640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37100a31e0 0x7f37100a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.927+0000 7f3729d44640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37100a31e0 0x7f37100a56a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f372406b070 tx=0x7f37140315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:43:21.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:21.930+0000 7f371b7fe640 1 -- 172.21.15.116:0/3156123685 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f372009c100 con 0x7f3724100b00 2024-09-19T06:43:22.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:22.112+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3724062630 con 0x7f37100a31e0 2024-09-19T06:43:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:22 smithi114 ceph-mon[30820]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:22 smithi116 ceph-mon[35411]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:22.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:22 smithi118 ceph-mon[35527]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:22.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:22.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] boot 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: osdmap e8: 1 total, 1 up, 1 in 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:43:23.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:23.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:23.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.418 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:43:23.418 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:43:23.418 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] boot 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: osdmap e8: 1 total, 1 up, 1 in 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:23.419 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: osd.0 [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] boot 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: osdmap e8: 1 total, 1 up, 1 in 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:43:23.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:23.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:23.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:23.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:24 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi114 2024-09-19T06:43:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:24 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 17089M 2024-09-19T06:43:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:24 smithi114 ceph-mon[30820]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:24 smithi114 ceph-mon[30820]: osdmap e9: 1 total, 1 up, 1 in 2024-09-19T06:43:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:24 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi114 2024-09-19T06:43:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:24 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 17089M 2024-09-19T06:43:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:24 smithi116 ceph-mon[35411]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:24 smithi116 ceph-mon[35411]: osdmap e9: 1 total, 1 up, 1 in 2024-09-19T06:43:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:24 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi114 2024-09-19T06:43:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:24 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi114 to 17089M 2024-09-19T06:43:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:24 smithi118 ceph-mon[35527]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-19T06:43:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:24 smithi118 ceph-mon[35527]: osdmap e9: 1 total, 1 up, 1 in 2024-09-19T06:43:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:25 smithi116 ceph-mon[35411]: from='client.? 172.21.15.116:0/2341223930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]: dispatch 2024-09-19T06:43:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:25 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]: dispatch 2024-09-19T06:43:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:25 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]': finished 2024-09-19T06:43:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:25 smithi116 ceph-mon[35411]: osdmap e10: 2 total, 1 up, 2 in 2024-09-19T06:43:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:25 smithi116 ceph-mon[35411]: from='client.? 172.21.15.116:0/1505291783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:43:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:25 smithi114 ceph-mon[30820]: from='client.? 172.21.15.116:0/2341223930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]: dispatch 2024-09-19T06:43:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:25 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]: dispatch 2024-09-19T06:43:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:25 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]': finished 2024-09-19T06:43:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:25 smithi114 ceph-mon[30820]: osdmap e10: 2 total, 1 up, 2 in 2024-09-19T06:43:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:25 smithi114 ceph-mon[30820]: from='client.? 172.21.15.116:0/1505291783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:43:25.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:25 smithi118 ceph-mon[35527]: from='client.? 172.21.15.116:0/2341223930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]: dispatch 2024-09-19T06:43:25.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:25 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]: dispatch 2024-09-19T06:43:25.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:25 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97cfe3c4-42fe-4873-a2ba-058f81d9c7d9"}]': finished 2024-09-19T06:43:25.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:25 smithi118 ceph-mon[35527]: osdmap e10: 2 total, 1 up, 2 in 2024-09-19T06:43:25.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:25.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:25 smithi118 ceph-mon[35527]: from='client.? 172.21.15.116:0/1505291783' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:43:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:26 smithi114 ceph-mon[30820]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:26 smithi116 ceph-mon[35411]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:26.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:26 smithi118 ceph-mon[35527]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:28 smithi114 ceph-mon[30820]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:28 smithi116 ceph-mon[35411]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:28.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:28 smithi118 ceph-mon[35527]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:30.379 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:30 smithi116 ceph-mon[35411]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:30 smithi114 ceph-mon[30820]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:30.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:30 smithi118 ceph-mon[35527]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:32.298 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:32 smithi116 ceph-mon[35411]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:32 smithi114 ceph-mon[30820]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:32 smithi118 ceph-mon[35527]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-19T06:43:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-19T06:43:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:33.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:33 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-19T06:43:33.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:33 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:34.221 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:34 smithi116 ceph-mon[35411]: Deploying daemon osd.1 on smithi116 2024-09-19T06:43:34.221 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:34 smithi116 ceph-mon[35411]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:34.407 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:34 smithi118 ceph-mon[35527]: Deploying daemon osd.1 on smithi116 2024-09-19T06:43:34.407 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:34 smithi118 ceph-mon[35527]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:34 smithi114 ceph-mon[30820]: Deploying daemon osd.1 on smithi116 2024-09-19T06:43:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:34 smithi114 ceph-mon[30820]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:36 smithi114 ceph-mon[30820]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:36 smithi116 ceph-mon[35411]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:36 smithi118 ceph-mon[35527]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:38 smithi114 ceph-mon[30820]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:38 smithi116 ceph-mon[35411]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:38.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:38 smithi118 ceph-mon[35527]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:39.382 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:39.382 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:39.382 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:39 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:39 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:39 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:40 smithi114 ceph-mon[30820]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:40.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:40 smithi116 ceph-mon[35411]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:40.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:40 smithi118 ceph-mon[35527]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:42.356 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:42 smithi116 ceph-mon[35411]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:42.356 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.356 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.356 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:42.356 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:42.356 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.397+0000 7f371b7fe640 1 -- 172.21.15.116:0/3156123685 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3724062630 con 0x7f37100a31e0 2024-09-19T06:43:42.399 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 1 on host 'smithi116' 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37100a31e0 msgr2=0x7f37100a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37100a31e0 0x7f37100a56a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f372406b070 tx=0x7f37140315b0 comp rx=0 tx=0).stop 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724100b00 msgr2=0x7f372406b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724100b00 0x7f372406b2f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f372000d950 tx=0x7f372000de10 comp rx=0 tx=0).stop 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 shutdown_connections 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37100a31e0 0x7f37100a56a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37241026c0 0x7f37241a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3724101d00 0x7f372419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 --2- 172.21.15.116:0/3156123685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724100b00 0x7f372406b2f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:42.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.399+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 >> 172.21.15.116:0/3156123685 conn(0x7f37240fc270 msgr2=0x7f3724105190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:43:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.400+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 shutdown_connections 2024-09-19T06:43:42.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:43:42.400+0000 7f372c7a9640 1 -- 172.21.15.116:0/3156123685 wait complete. 2024-09-19T06:43:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:42 smithi114 ceph-mon[30820]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:42 smithi118 ceph-mon[35527]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:42.999 DEBUG:teuthology.orchestra.run.smithi116:osd.1> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.1.service 2024-09-19T06:43:43.001 INFO:tasks.cephadm:Deploying osd.2 on smithi118 with /dev/vg_nvme/lv_4... 2024-09-19T06:43:43.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-19T06:43:43.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:43.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:43.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:43.251 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:43:43.372 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:43.372 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:43.372 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:43:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:44 smithi114 ceph-mon[30820]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:44 smithi116 ceph-mon[35411]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:44.496 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:44 smithi118 ceph-mon[35527]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:44.929 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:43:44 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:43:44.552+0000 7fac1f0b1640 -1 osd.1 0 log_to_monitors true 2024-09-19T06:43:45.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:45 smithi118 ceph-mon[35527]: from='osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T06:43:45.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:45 smithi118 ceph-mon[35527]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T06:43:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:45 smithi116 ceph-mon[35411]: from='osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T06:43:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:45 smithi116 ceph-mon[35411]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T06:43:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:45 smithi114 ceph-mon[30820]: from='osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T06:43:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:45 smithi114 ceph-mon[30820]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-19T06:43:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:43:45.716 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi118:vg_nvme/lv_4 2024-09-19T06:43:45.963 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:43:46.337 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:46.337 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-19T06:43:46.337 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: osdmap e11: 2 total, 1 up, 2 in 2024-09-19T06:43:46.337 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:46.337 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:43:46.337 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:43:46.337 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.338 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.338 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:43:46.338 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.338 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:46.338 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:46.338 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.416+0000 7fd986875640 1 -- 172.21.15.118:0/4075084402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 msgr2=0x7fd980102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 --2- 172.21.15.118:0/4075084402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 0x7fd980102000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd970009900 tx=0x7fd97002a410 comp rx=0 tx=0).stop 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 -- 172.21.15.118:0/4075084402 shutdown_connections 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 --2- 172.21.15.118:0/4075084402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9801037c0 0x7fd98010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 --2- 172.21.15.118:0/4075084402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd980102e00 0x7fd980103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 --2- 172.21.15.118:0/4075084402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 0x7fd980102000 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 -- 172.21.15.118:0/4075084402 >> 172.21.15.118:0/4075084402 conn(0x7fd9800fd390 msgr2=0x7fd9800ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 -- 172.21.15.118:0/4075084402 shutdown_connections 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 -- 172.21.15.118:0/4075084402 wait complete. 2024-09-19T06:43:46.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 Processor -- start 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.417+0000 7fd986875640 1 -- start start 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd986875640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 0x7fd9801980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd986875640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd980102e00 0x7fd9801985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd986875640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9801037c0 0x7fd98019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd986875640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd98010d960 con 0x7fd980102e00 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd986875640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd98010d7e0 con 0x7fd9801037c0 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd986875640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd98010dae0 con 0x7fd980101c00 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 0x7fd9801980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 0x7fd9801980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43094/0 (socket says 172.21.15.118:43094) 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 -- 172.21.15.118:0/453300467 learned_addr learned my addr 172.21.15.118:0/453300467 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd984e12640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9801037c0 0x7fd98019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:43:46.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 -- 172.21.15.118:0/453300467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9801037c0 msgr2=0x7fd98019f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9801037c0 0x7fd98019f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 -- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd980102e00 msgr2=0x7fd9801985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97f7fe640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd980102e00 0x7fd9801985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd980102e00 0x7fd9801985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 -- 172.21.15.118:0/453300467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98019fcd0 con 0x7fd980101c00 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd984e12640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9801037c0 0x7fd98019f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97f7fe640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd980102e00 0x7fd9801985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97ffff640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 0x7fd9801980a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd970031610 tx=0x7fd9700043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97d7fa640 1 -- 172.21.15.118:0/453300467 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97003f030 con 0x7fd980101c00 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97d7fa640 1 -- 172.21.15.118:0/453300467 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd970003ce0 con 0x7fd980101c00 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98019ff60 con 0x7fd980101c00 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.418+0000 7fd97d7fa640 1 -- 172.21.15.118:0/453300467 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970035070 con 0x7fd980101c00 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.419+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9801a03e0 con 0x7fd980101c00 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.420+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9801090a0 con 0x7fd980101c00 2024-09-19T06:43:46.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.421+0000 7fd97d7fa640 1 -- 172.21.15.118:0/453300467 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 15) ==== 138300+0+0 (secure 0 0 0) 0x7fd970042030 con 0x7fd980101c00 2024-09-19T06:43:46.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.421+0000 7fd97d7fa640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd9680a31e0 0x7fd9680a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:43:46.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.421+0000 7fd97d7fa640 1 -- 172.21.15.118:0/453300467 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2029+0+0 (secure 0 0 0) 0x7fd9700ee830 con 0x7fd980101c00 2024-09-19T06:43:46.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.422+0000 7fd97f7fe640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd9680a31e0 0x7fd9680a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:43:46.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.422+0000 7fd97f7fe640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd9680a31e0 0x7fd9680a56a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd974006fd0 tx=0x7fd974008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:43:46.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.425+0000 7fd97d7fa640 1 -- 172.21.15.118:0/453300467 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd9700bcec0 con 0x7fd980101c00 2024-09-19T06:43:46.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:43:46.599+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd980062630 con 0x7fd9680a31e0 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: osdmap e11: 2 total, 1 up, 2 in 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.678 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:43:46.180+0000 7fac1b034640 -1 osd.1 0 waiting for initial osdmap 2024-09-19T06:43:46.678 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:43:46.183+0000 7fac16e5c640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: osdmap e11: 2 total, 1 up, 2 in 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:43:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:43:47.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi116 2024-09-19T06:43:47.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi116 to 17088M 2024-09-19T06:43:47.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:43:47.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: osdmap e12: 2 total, 1 up, 2 in 2024-09-19T06:43:47.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:47.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:47.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:43:47.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:43:47.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:47 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi116 2024-09-19T06:43:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 17088M 2024-09-19T06:43:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:43:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: osdmap e12: 2 total, 1 up, 2 in 2024-09-19T06:43:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:43:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:43:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi116 2024-09-19T06:43:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 17088M 2024-09-19T06:43:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:43:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: osdmap e12: 2 total, 1 up, 2 in 2024-09-19T06:43:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:43:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:43:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:48 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:43:48.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:48 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:43:48.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:48 smithi118 ceph-mon[35527]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:43:48.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:48 smithi118 ceph-mon[35527]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:48.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:48 smithi118 ceph-mon[35527]: osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] boot 2024-09-19T06:43:48.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:48 smithi118 ceph-mon[35527]: osdmap e13: 2 total, 2 up, 2 in 2024-09-19T06:43:48.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:48 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:48 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:48 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:48 smithi114 ceph-mon[30820]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:48 smithi114 ceph-mon[30820]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:48 smithi114 ceph-mon[30820]: osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] boot 2024-09-19T06:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:48 smithi114 ceph-mon[30820]: osdmap e13: 2 total, 2 up, 2 in 2024-09-19T06:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:48 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:43:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:48 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:43:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:48 smithi116 ceph-mon[35411]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:43:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:48 smithi116 ceph-mon[35411]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-19T06:43:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:48 smithi116 ceph-mon[35411]: osd.1 [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] boot 2024-09-19T06:43:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:48 smithi116 ceph-mon[35411]: osdmap e13: 2 total, 2 up, 2 in 2024-09-19T06:43:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-19T06:43:49.419 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:49 smithi118 ceph-mon[35527]: osdmap e14: 2 total, 2 up, 2 in 2024-09-19T06:43:49.419 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:49 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/1254465345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]: dispatch 2024-09-19T06:43:49.419 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:49 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]: dispatch 2024-09-19T06:43:49.419 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:49 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]': finished 2024-09-19T06:43:49.419 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:49 smithi118 ceph-mon[35527]: osdmap e15: 3 total, 2 up, 3 in 2024-09-19T06:43:49.419 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:43:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:49 smithi114 ceph-mon[30820]: osdmap e14: 2 total, 2 up, 2 in 2024-09-19T06:43:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:49 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/1254465345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]: dispatch 2024-09-19T06:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:49 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]: dispatch 2024-09-19T06:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:49 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]': finished 2024-09-19T06:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:49 smithi114 ceph-mon[30820]: osdmap e15: 3 total, 2 up, 3 in 2024-09-19T06:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:49 smithi116 ceph-mon[35411]: osdmap e14: 2 total, 2 up, 2 in 2024-09-19T06:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:49 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/1254465345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]: dispatch 2024-09-19T06:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:49 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]: dispatch 2024-09-19T06:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:49 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6db87a1e-8734-4c62-86bb-b3c2297b7ef6"}]': finished 2024-09-19T06:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:49 smithi116 ceph-mon[35411]: osdmap e15: 3 total, 2 up, 3 in 2024-09-19T06:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:43:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:50 smithi118 ceph-mon[35527]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:50 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/1718480220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:43:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:50 smithi114 ceph-mon[30820]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:50 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/1718480220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:43:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:50 smithi116 ceph-mon[35411]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:50 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/1718480220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:43:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:52 smithi118 ceph-mon[35527]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:52 smithi114 ceph-mon[30820]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:52 smithi116 ceph-mon[35411]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:54.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:54 smithi118 ceph-mon[35527]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:54 smithi114 ceph-mon[30820]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:54 smithi116 ceph-mon[35411]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:56.421 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:56 smithi118 ceph-mon[35527]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:56 smithi114 ceph-mon[30820]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:56 smithi116 ceph-mon[35411]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-19T06:43:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-19T06:43:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-19T06:43:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:43:58.481 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:58 smithi118 ceph-mon[35527]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:58.481 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:43:58 smithi118 ceph-mon[35527]: Deploying daemon osd.2 on smithi118 2024-09-19T06:43:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:58 smithi114 ceph-mon[30820]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:43:58 smithi114 ceph-mon[30820]: Deploying daemon osd.2 on smithi118 2024-09-19T06:43:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:58 smithi116 ceph-mon[35411]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:43:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:43:58 smithi116 ceph-mon[35411]: Deploying daemon osd.2 on smithi118 2024-09-19T06:44:00.474 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:00 smithi118 ceph-mon[35527]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:00 smithi114 ceph-mon[30820]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:00 smithi116 ceph-mon[35411]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:02.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:02 smithi118 ceph-mon[35527]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:02 smithi114 ceph-mon[30820]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:02 smithi116 ceph-mon[35411]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:04.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:04 smithi116 ceph-mon[35411]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:04.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:04.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:04.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:04.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:04 smithi118 ceph-mon[35527]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:04.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:04 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:04.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:04 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:04.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:04 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:04 smithi114 ceph-mon[30820]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:06 smithi118 ceph-mon[35527]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:06.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:06 smithi114 ceph-mon[30820]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:06 smithi116 ceph-mon[35411]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:07.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.509+0000 7fd97d7fa640 1 -- 172.21.15.118:0/453300467 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd980062630 con 0x7fd9680a31e0 2024-09-19T06:44:07.512 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 2 on host 'smithi118' 2024-09-19T06:44:07.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.511+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd9680a31e0 msgr2=0x7fd9680a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:07.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.511+0000 7fd986875640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd9680a31e0 0x7fd9680a56a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd974006fd0 tx=0x7fd974008040 comp rx=0 tx=0).stop 2024-09-19T06:44:07.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.511+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 msgr2=0x7fd9801980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:07.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.511+0000 7fd986875640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 0x7fd9801980a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd970031610 tx=0x7fd9700043b0 comp rx=0 tx=0).stop 2024-09-19T06:44:07.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.512+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 shutdown_connections 2024-09-19T06:44:07.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.512+0000 7fd986875640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd9680a31e0 0x7fd9680a56a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:07.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.512+0000 7fd986875640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9801037c0 0x7fd98019f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:07.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.512+0000 7fd986875640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd980102e00 0x7fd9801985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:07.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.512+0000 7fd986875640 1 --2- 172.21.15.118:0/453300467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd980101c00 0x7fd9801980a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:07.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.512+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 >> 172.21.15.118:0/453300467 conn(0x7fd9800fd390 msgr2=0x7fd980106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:44:07.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.512+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 shutdown_connections 2024-09-19T06:44:07.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:44:07.512+0000 7fd986875640 1 -- 172.21.15.118:0/453300467 wait complete. 2024-09-19T06:44:08.136 DEBUG:teuthology.orchestra.run.smithi118:osd.2> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.2.service 2024-09-19T06:44:08.139 INFO:tasks.cephadm:Deploying osd.3 on smithi114 with /dev/vg_nvme/lv_3... 2024-09-19T06:44:08.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-19T06:44:08.455 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:08 smithi118 ceph-mon[35527]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:08.455 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:08.455 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:08.455 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:08.508 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:44:08.646 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:08 smithi114 ceph-mon[30820]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:08.647 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:08.647 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:08.647 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:08 smithi116 ceph-mon[35411]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:09.578 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:09 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-2[40489]: 2024-09-19T06:44:09.282+0000 7f765c023640 -1 osd.2 0 log_to_monitors true 2024-09-19T06:44:10.243 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:10 smithi114 ceph-mon[30820]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:10.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:10 smithi114 ceph-mon[30820]: from='osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T06:44:10.244 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:10 smithi114 ceph-mon[30820]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T06:44:10.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:10 smithi118 ceph-mon[35527]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:10.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:10 smithi118 ceph-mon[35527]: from='osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T06:44:10.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:10 smithi118 ceph-mon[35527]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T06:44:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:10 smithi116 ceph-mon[35411]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:10 smithi116 ceph-mon[35411]: from='osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T06:44:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:10 smithi116 ceph-mon[35411]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-19T06:44:11.506 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-2[40489]: 2024-09-19T06:44:11.249+0000 7f7657fa6640 -1 osd.2 0 waiting for initial osdmap 2024-09-19T06:44:11.506 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-2[40489]: 2024-09-19T06:44:11.256+0000 7f7653dce640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: osdmap e16: 3 total, 2 up, 3 in 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:11.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-19T06:44:11.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:44:11.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: osdmap e16: 3 total, 2 up, 3 in 2024-09-19T06:44:11.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:11.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:44:11.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.540 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.541 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:11.541 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.541 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:11.541 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:11.541 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.545 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:44:11.578 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: osdmap e16: 3 total, 2 up, 3 in 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:11.948 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:44:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:12 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi118 2024-09-19T06:44:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:12 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi118 to 17088M 2024-09-19T06:44:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:12 smithi114 ceph-mon[30820]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:12 smithi114 ceph-mon[30820]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:44:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:12 smithi114 ceph-mon[30820]: osdmap e17: 3 total, 2 up, 3 in 2024-09-19T06:44:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:12.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:12 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi118 2024-09-19T06:44:12.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:12 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi118 to 17088M 2024-09-19T06:44:12.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:12 smithi118 ceph-mon[35527]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:12.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:12 smithi118 ceph-mon[35527]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:44:12.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:12 smithi118 ceph-mon[35527]: osdmap e17: 3 total, 2 up, 3 in 2024-09-19T06:44:12.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:12 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:12.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:12 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:12.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.550+0000 7f2360e00640 1 -- 172.21.15.114:0/2401031207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c1098b0 msgr2=0x7f235c10bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:12.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.550+0000 7f2360e00640 1 --2- 172.21.15.114:0/2401031207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c1098b0 0x7f235c10bd70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2354009ca0 tx=0x7f235402a8c0 comp rx=0 tx=0).stop 2024-09-19T06:44:12.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.551+0000 7f2360e00640 1 -- 172.21.15.114:0/2401031207 shutdown_connections 2024-09-19T06:44:12.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.551+0000 7f2360e00640 1 --2- 172.21.15.114:0/2401031207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c1098b0 0x7f235c10bd70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:12.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.551+0000 7f2360e00640 1 --2- 172.21.15.114:0/2401031207 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f235c074b40 0x7f235c074fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:12.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.551+0000 7f2360e00640 1 --2- 172.21.15.114:0/2401031207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c0761a0 0x7f235c074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:12.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.551+0000 7f2360e00640 1 -- 172.21.15.114:0/2401031207 >> 172.21.15.114:0/2401031207 conn(0x7f235c0fd1f0 msgr2=0x7f235c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:44:12.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.551+0000 7f2360e00640 1 -- 172.21.15.114:0/2401031207 shutdown_connections 2024-09-19T06:44:12.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.551+0000 7f2360e00640 1 -- 172.21.15.114:0/2401031207 wait complete. 2024-09-19T06:44:12.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2360e00640 1 Processor -- start 2024-09-19T06:44:12.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2360e00640 1 -- start start 2024-09-19T06:44:12.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2360e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c074b40 0x7f235c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:44:12.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2360e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c0761a0 0x7f235c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:44:12.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2360e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f235c1098b0 0x7f235c1a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2360e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f235c10f900 con 0x7f235c074b40 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2360e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f235c10f780 con 0x7f235c0761a0 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2360e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f235c10fa80 con 0x7f235c1098b0 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f235b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c074b40 0x7f235c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f235b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c074b40 0x7f235c199ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39620/0 (socket says 172.21.15.114:39620) 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f235b400640 1 -- 172.21.15.114:0/2693704447 learned_addr learned my addr 172.21.15.114:0/2693704447 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f235be00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f235c1098b0 0x7f235c1a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.552+0000 7f2353e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c0761a0 0x7f235c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2353e00640 1 -- 172.21.15.114:0/2693704447 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f235c1098b0 msgr2=0x7f235c1a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:12.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2353e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f235c1098b0 0x7f235c1a1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2353e00640 1 -- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c074b40 msgr2=0x7f235c199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2353e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c074b40 0x7f235c199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2353e00640 1 -- 172.21.15.114:0/2693704447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235c1a1b00 con 0x7f235c0761a0 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f235b400640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c074b40 0x7f235c199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2353e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c0761a0 0x7f235c19a410 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f234400e9f0 tx=0x7f234400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2358c00640 1 -- 172.21.15.114:0/2693704447 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234400ccc0 con 0x7f235c0761a0 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2358c00640 1 -- 172.21.15.114:0/2693704447 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2344005020 con 0x7f235c0761a0 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.553+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235c1a1df0 con 0x7f235c0761a0 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.554+0000 7f2358c00640 1 -- 172.21.15.114:0/2693704447 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23440049e0 con 0x7f235c0761a0 2024-09-19T06:44:12.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.554+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235c1a22a0 con 0x7f235c0761a0 2024-09-19T06:44:12.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.555+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f235c1086c0 con 0x7f235c0761a0 2024-09-19T06:44:12.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.556+0000 7f2358c00640 1 -- 172.21.15.114:0/2693704447 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) ==== 138300+0+0 (secure 0 0 0) 0x7f2344005560 con 0x7f235c0761a0 2024-09-19T06:44:12.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.558+0000 7f2358c00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f23400a3260 0x7f23400a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:44:12.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.558+0000 7f2358c00640 1 -- 172.21.15.114:0/2693704447 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2851+0+0 (secure 0 0 0) 0x7f23440ce470 con 0x7f235c0761a0 2024-09-19T06:44:12.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.558+0000 7f235b400640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f23400a3260 0x7f23400a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:44:12.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.559+0000 7f235b400640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f23400a3260 0x7f23400a5720 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f234c011550 tx=0x7f234c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:44:12.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.560+0000 7f2358c00640 1 -- 172.21.15.114:0/2693704447 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f23440051d0 con 0x7f235c0761a0 2024-09-19T06:44:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:12 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi118 2024-09-19T06:44:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:12 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi118 to 17088M 2024-09-19T06:44:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:12 smithi116 ceph-mon[35411]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:12 smithi116 ceph-mon[35411]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:44:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:12 smithi116 ceph-mon[35411]: osdmap e17: 3 total, 2 up, 3 in 2024-09-19T06:44:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:12.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:12.744+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f235c062630 con 0x7f23400a3260 2024-09-19T06:44:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] boot 2024-09-19T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: osdmap e18: 3 total, 3 up, 3 in 2024-09-19T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:44:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:44:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:44:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] boot 2024-09-19T06:44:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: osdmap e18: 3 total, 3 up, 3 in 2024-09-19T06:44:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:44:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:44:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:14.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:44:14.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:44:14.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:14.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: osd.2 [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] boot 2024-09-19T06:44:14.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: osdmap e18: 3 total, 3 up, 3 in 2024-09-19T06:44:14.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-19T06:44:14.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:44:14.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:44:14.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:14 smithi114 ceph-mon[30820]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:44:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:14 smithi114 ceph-mon[30820]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-19T06:44:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-19T06:44:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:14 smithi114 ceph-mon[30820]: osdmap e19: 3 total, 3 up, 3 in 2024-09-19T06:44:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T06:44:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:14 smithi116 ceph-mon[35411]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:44:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:14 smithi116 ceph-mon[35411]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-19T06:44:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-19T06:44:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:14 smithi116 ceph-mon[35411]: osdmap e19: 3 total, 3 up, 3 in 2024-09-19T06:44:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T06:44:15.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:14 smithi118 ceph-mon[35527]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:44:15.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:14 smithi118 ceph-mon[35527]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-19T06:44:15.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-19T06:44:15.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-19T06:44:15.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:14 smithi118 ceph-mon[35527]: osdmap e19: 3 total, 3 up, 3 in 2024-09-19T06:44:15.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-19T06:44:15.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:44:15 smithi114 sudo[48936]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T06:44:15.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:44:15 smithi114 sudo[48936]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:15.633 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:44:15 smithi114 sudo[48936]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:15.634 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:44:15 smithi114 sudo[48936]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:15.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:15.635+0000 7f2358c00640 1 -- 172.21.15.114:0/2693704447 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f2344098690 con 0x7f235c0761a0 2024-09-19T06:44:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-19T06:44:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:15 smithi114 ceph-mon[30820]: osdmap e20: 3 total, 3 up, 3 in 2024-09-19T06:44:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:15 smithi114 ceph-mon[30820]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:15.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:44:15 smithi114 sudo[48964]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T06:44:15.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:44:15 smithi114 sudo[48964]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:15.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:44:15 smithi114 sudo[48964]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:15.917 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:44:15 smithi114 sudo[48964]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-19T06:44:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:15 smithi116 ceph-mon[35411]: osdmap e20: 3 total, 3 up, 3 in 2024-09-19T06:44:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:15 smithi116 ceph-mon[35411]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:16.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-19T06:44:16.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:15 smithi118 ceph-mon[35527]: osdmap e20: 3 total, 3 up, 3 in 2024-09-19T06:44:16.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:15 smithi118 ceph-mon[35527]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:16.355 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:44:16 smithi116 sudo[43707]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T06:44:16.355 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:44:16 smithi116 sudo[43707]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:16.356 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:44:16 smithi116 sudo[43707]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:16.356 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:44:16 smithi116 sudo[43707]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:16.632 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:44:16 smithi116 sudo[43710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T06:44:16.632 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:44:16 smithi116 sudo[43710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:16.632 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:44:16 smithi116 sudo[43710]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:16.632 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:44:16 smithi116 sudo[43710]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:16 smithi114 ceph-mon[30820]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-19T06:44:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:16 smithi114 ceph-mon[30820]: osdmap e21: 3 total, 3 up, 3 in 2024-09-19T06:44:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:16 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/847780094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c36094-adb2-4adb-8aa0-dd87da73028e"}]: dispatch 2024-09-19T06:44:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:16 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/847780094' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c36094-adb2-4adb-8aa0-dd87da73028e"}]': finished 2024-09-19T06:44:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:16 smithi114 ceph-mon[30820]: osdmap e22: 4 total, 3 up, 4 in 2024-09-19T06:44:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:16.886 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:16 smithi118 sudo[44559]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T06:44:16.886 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:16 smithi118 sudo[44559]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:16.887 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:16 smithi118 sudo[44559]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:16.887 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:16 smithi118 sudo[44559]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:16.887 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:16 smithi118 ceph-mon[35527]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-19T06:44:16.887 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:16 smithi118 ceph-mon[35527]: osdmap e21: 3 total, 3 up, 3 in 2024-09-19T06:44:16.887 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:16.887 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:16 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/847780094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c36094-adb2-4adb-8aa0-dd87da73028e"}]: dispatch 2024-09-19T06:44:16.887 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:16 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/847780094' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c36094-adb2-4adb-8aa0-dd87da73028e"}]': finished 2024-09-19T06:44:16.887 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:16 smithi118 ceph-mon[35527]: osdmap e22: 4 total, 3 up, 4 in 2024-09-19T06:44:16.888 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:16.888 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:16 smithi116 ceph-mon[35411]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-19T06:44:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:16 smithi116 ceph-mon[35411]: osdmap e21: 3 total, 3 up, 3 in 2024-09-19T06:44:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:16 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/847780094' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5c36094-adb2-4adb-8aa0-dd87da73028e"}]: dispatch 2024-09-19T06:44:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:16 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/847780094' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5c36094-adb2-4adb-8aa0-dd87da73028e"}]': finished 2024-09-19T06:44:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:16 smithi116 ceph-mon[35411]: osdmap e22: 4 total, 3 up, 4 in 2024-09-19T06:44:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:17.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:16 smithi118 sudo[44562]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T06:44:17.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:16 smithi118 sudo[44562]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:17.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:16 smithi118 sudo[44562]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:17.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Sep 19 06:44:16 smithi118 sudo[44562]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 sudo[49063]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T06:44:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 sudo[49063]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 sudo[49063]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 sudo[49063]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:18.248 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:17 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3047041060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:44:18.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:17 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:18.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:17 smithi118 ceph-mon[35527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:18.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 sudo[44565]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T06:44:18.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 sudo[44565]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:18.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 sudo[44565]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:18.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 sudo[44565]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 sudo[49188]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T06:44:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 sudo[49188]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 sudo[49188]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 sudo[49188]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3047041060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:44:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:17 smithi114 ceph-mon[30820]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:17 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3047041060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:44:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:17 smithi116 ceph-mon[35411]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 sudo[44568]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T06:44:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 sudo[44568]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 sudo[44568]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 sudo[44568]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:18.986 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 sudo[43715]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-19T06:44:18.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 sudo[43715]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:18.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 sudo[43715]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:18.987 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 sudo[43715]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:19.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 sudo[43718]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-19T06:44:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 sudo[43718]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 sudo[43718]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 sudo[43718]: pam_unix(sudo:session): session closed for user root 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-19T06:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-19T06:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-19T06:44:20.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:19 smithi118 ceph-mon[35527]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:20.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:19 smithi118 ceph-mon[35527]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:20.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:19 smithi118 ceph-mon[35527]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:19 smithi114 ceph-mon[30820]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:19 smithi114 ceph-mon[30820]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:19 smithi114 ceph-mon[30820]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:19 smithi116 ceph-mon[35411]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-19T06:44:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:19 smithi116 ceph-mon[35411]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-19T06:44:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:19 smithi116 ceph-mon[35411]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:22.360 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:22 smithi114 ceph-mon[30820]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:22 smithi116 ceph-mon[35411]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:22.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:22 smithi118 ceph-mon[35527]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:24.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:24 smithi114 ceph-mon[30820]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:24 smithi116 ceph-mon[35411]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:24 smithi118 ceph-mon[35527]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:26.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:26 smithi114 ceph-mon[30820]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-19T06:44:26.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-19T06:44:26.352 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:26 smithi116 ceph-mon[35411]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-19T06:44:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-19T06:44:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:26.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:26 smithi118 ceph-mon[35527]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-19T06:44:26.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:26 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-19T06:44:26.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:26 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:27.388 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:27 smithi114 ceph-mon[30820]: Deploying daemon osd.3 on smithi114 2024-09-19T06:44:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:27 smithi116 ceph-mon[35411]: Deploying daemon osd.3 on smithi114 2024-09-19T06:44:27.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:27 smithi118 ceph-mon[35527]: Deploying daemon osd.3 on smithi114 2024-09-19T06:44:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:28 smithi114 ceph-mon[30820]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-19T06:44:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:28 smithi116 ceph-mon[35411]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-19T06:44:28.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:28 smithi118 ceph-mon[35527]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-19T06:44:30.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:30 smithi114 ceph-mon[30820]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-19T06:44:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:30 smithi116 ceph-mon[35411]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-19T06:44:30.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:30 smithi118 ceph-mon[35527]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-19T06:44:32.327 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:32 smithi114 ceph-mon[30820]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:32 smithi116 ceph-mon[35411]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:32 smithi118 ceph-mon[35527]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:34.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:34 smithi114 ceph-mon[30820]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:34.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:34.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:34.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:34 smithi116 ceph-mon[35411]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:34 smithi118 ceph-mon[35527]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:36.159 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:36 smithi114 ceph-mon[30820]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:36 smithi116 ceph-mon[35411]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:36 smithi118 ceph-mon[35527]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-19T06:44:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:37 smithi114 ceph-mon[30820]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:37 smithi116 ceph-mon[35411]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:38.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:38.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:38.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:38.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:37 smithi118 ceph-mon[35527]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:38.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.470+0000 7f2358c00640 1 -- 172.21.15.114:0/2693704447 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f235c062630 con 0x7f23400a3260 2024-09-19T06:44:38.472 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 3 on host 'smithi114' 2024-09-19T06:44:38.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f23400a3260 msgr2=0x7f23400a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:38.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f23400a3260 0x7f23400a5720 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f234c011550 tx=0x7f234c005f90 comp rx=0 tx=0).stop 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c0761a0 msgr2=0x7f235c19a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c0761a0 0x7f235c19a410 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f234400e9f0 tx=0x7f234400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 shutdown_connections 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f23400a3260 0x7f23400a5720 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f235c1098b0 0x7f235c1a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f235c0761a0 0x7f235c19a410 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 --2- 172.21.15.114:0/2693704447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f235c074b40 0x7f235c199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 >> 172.21.15.114:0/2693704447 conn(0x7f235c0fd1f0 msgr2=0x7f235c105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 shutdown_connections 2024-09-19T06:44:38.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:44:38.474+0000 7f2360e00640 1 -- 172.21.15.114:0/2693704447 wait complete. 2024-09-19T06:44:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:39.206 DEBUG:teuthology.orchestra.run.smithi114:osd.3> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.3.service 2024-09-19T06:44:39.211 INFO:tasks.cephadm:Deploying osd.4 on smithi116 with /dev/vg_nvme/lv_3... 2024-09-19T06:44:39.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-19T06:44:39.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:38 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:44:39.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:38 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:39.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:38 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:39.432 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 19 06:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-3[51029]: 2024-09-19T06:44:39.339+0000 7f868443f640 -1 osd.3 0 log_to_monitors true 2024-09-19T06:44:39.465 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T06:44:39.902 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:39 smithi116 ceph-mon[35411]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:39.902 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:39 smithi116 ceph-mon[35411]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-19T06:44:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:39 smithi114 ceph-mon[30820]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:39 smithi114 ceph-mon[30820]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-19T06:44:40.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:39 smithi118 ceph-mon[35527]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:40.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:39 smithi118 ceph-mon[35527]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-19T06:44:40.853 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:40 smithi114 ceph-mon[30820]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-19T06:44:40.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:40 smithi114 ceph-mon[30820]: osdmap e23: 4 total, 3 up, 4 in 2024-09-19T06:44:40.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:40 smithi114 ceph-mon[30820]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:44:40.856 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:41.091 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:40 smithi116 ceph-mon[35411]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-19T06:44:41.092 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:40 smithi116 ceph-mon[35411]: osdmap e23: 4 total, 3 up, 4 in 2024-09-19T06:44:41.092 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:40 smithi116 ceph-mon[35411]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:44:41.092 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:41.122 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 19 06:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-3[51029]: 2024-09-19T06:44:40.851+0000 7f867f000640 -1 osd.3 0 waiting for initial osdmap 2024-09-19T06:44:41.122 INFO:journalctl@ceph.osd.3.smithi114.stdout:Sep 19 06:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-3[51029]: 2024-09-19T06:44:40.859+0000 7f867a000640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:44:41.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:40 smithi118 ceph-mon[35527]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-19T06:44:41.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:40 smithi118 ceph-mon[35527]: osdmap e23: 4 total, 3 up, 4 in 2024-09-19T06:44:41.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:40 smithi118 ceph-mon[35527]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:44:41.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:41 smithi116 ceph-mon[35411]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:41 smithi116 ceph-mon[35411]: osdmap e24: 4 total, 3 up, 4 in 2024-09-19T06:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:41 smithi116 ceph-mon[35411]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:41.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:44:42.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-09-19T06:44:42.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:41 smithi114 ceph-mon[30820]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:44:42.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:41 smithi114 ceph-mon[30820]: osdmap e24: 4 total, 3 up, 4 in 2024-09-19T06:44:42.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:42.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:42.116 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:41 smithi114 ceph-mon[30820]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:42.255 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T06:44:42.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:41 smithi118 ceph-mon[35527]: from='osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:44:42.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:41 smithi118 ceph-mon[35527]: osdmap e24: 4 total, 3 up, 4 in 2024-09-19T06:44:42.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:42.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:41 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:42.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:41 smithi118 ceph-mon[35527]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-19T06:44:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 -- 172.21.15.116:0/3241414154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 msgr2=0x7f1530102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 --2- 172.21.15.116:0/3241414154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 0x7f1530102190 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1524009900 tx=0x7f152402a410 comp rx=0 tx=0).stop 2024-09-19T06:44:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 -- 172.21.15.116:0/3241414154 shutdown_connections 2024-09-19T06:44:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 --2- 172.21.15.116:0/3241414154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15301026d0 0x7f1530108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 --2- 172.21.15.116:0/3241414154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 0x7f1530102190 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 --2- 172.21.15.116:0/3241414154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1530100b10 0x7f1530100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 -- 172.21.15.116:0/3241414154 >> 172.21.15.116:0/3241414154 conn(0x7f15300fc2a0 msgr2=0x7f15300fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 -- 172.21.15.116:0/3241414154 shutdown_connections 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.764+0000 7f153619d640 1 -- 172.21.15.116:0/3241414154 wait complete. 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f153619d640 1 Processor -- start 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f153619d640 1 -- start start 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f153619d640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1530100b10 0x7f1530197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f153619d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 0x7f1530198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f153619d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15301026d0 0x7f153019f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f153619d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f153010d8c0 con 0x7f15301026d0 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f153619d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f153010d740 con 0x7f1530101d10 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f153619d640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f153010da40 con 0x7f1530100b10 2024-09-19T06:44:42.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f152effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 0x7f1530198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f152effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 0x7f1530198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54116/0 (socket says 172.21.15.116:54116) 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f152effd640 1 -- 172.21.15.116:0/286575649 learned_addr learned my addr 172.21.15.116:0/286575649 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f152ffff640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15301026d0 0x7f153019f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.765+0000 7f152f7fe640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1530100b10 0x7f1530197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152effd640 1 -- 172.21.15.116:0/286575649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1530100b10 msgr2=0x7f1530197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152effd640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1530100b10 0x7f1530197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152effd640 1 -- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15301026d0 msgr2=0x7f153019f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152effd640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15301026d0 0x7f153019f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152effd640 1 -- 172.21.15.116:0/286575649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153019fa70 con 0x7f1530101d10 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152ffff640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15301026d0 0x7f153019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152effd640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 0x7f1530198380 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1524009230 tx=0x7f1524004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152cff9640 1 -- 172.21.15.116:0/286575649 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f152403f030 con 0x7f1530101d10 2024-09-19T06:44:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f153019fd00 con 0x7f1530101d10 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152cff9640 1 -- 172.21.15.116:0/286575649 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1524003ce0 con 0x7f1530101d10 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f152cff9640 1 -- 172.21.15.116:0/286575649 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524035070 con 0x7f1530101d10 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.766+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15301a01d0 con 0x7f1530101d10 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.767+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1530107e90 con 0x7f1530101d10 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.768+0000 7f152cff9640 1 -- 172.21.15.116:0/286575649 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f1524042030 con 0x7f1530101d10 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.768+0000 7f152cff9640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f15180a3270 0x7f15180a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.769+0000 7f152cff9640 1 -- 172.21.15.116:0/286575649 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 3702+0+0 (secure 0 0 0) 0x7f15240eedd0 con 0x7f1530101d10 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.769+0000 7f152f7fe640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f15180a3270 0x7f15180a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.769+0000 7f152f7fe640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f15180a3270 0x7f15180a5730 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15200058c0 tx=0x7f1520005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:44:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.773+0000 7f152cff9640 1 -- 172.21.15.116:0/286575649 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f15240bccf0 con 0x7f1530101d10 2024-09-19T06:44:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:42 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:44:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:42 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:44:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:42 smithi116 ceph-mon[35411]: osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] boot 2024-09-19T06:44:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:42 smithi116 ceph-mon[35411]: osdmap e25: 4 total, 4 up, 4 in 2024-09-19T06:44:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:42 smithi116 ceph-mon[35411]: osdmap e26: 4 total, 4 up, 4 in 2024-09-19T06:44:42.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:44:42.951+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1530062630 con 0x7f15180a3270 2024-09-19T06:44:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:42 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:44:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:42 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:44:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:42 smithi114 ceph-mon[30820]: osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] boot 2024-09-19T06:44:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:42 smithi114 ceph-mon[30820]: osdmap e25: 4 total, 4 up, 4 in 2024-09-19T06:44:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:42 smithi114 ceph-mon[30820]: osdmap e26: 4 total, 4 up, 4 in 2024-09-19T06:44:43.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:42 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:44:43.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:42 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:44:43.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:42 smithi118 ceph-mon[35527]: osd.3 [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] boot 2024-09-19T06:44:43.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:42 smithi118 ceph-mon[35527]: osdmap e25: 4 total, 4 up, 4 in 2024-09-19T06:44:43.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-19T06:44:43.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:42 smithi118 ceph-mon[35527]: osdmap e26: 4 total, 4 up, 4 in 2024-09-19T06:44:44.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi114 2024-09-19T06:44:44.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi114 to 8544M 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:44.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:44.258 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:43 smithi118 ceph-mon[35527]: osdmap e27: 4 total, 4 up, 4 in 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi114 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 8544M 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:44.323 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:44:44.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:44:44.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:44:44.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:44.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:44.324 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:43 smithi116 ceph-mon[35411]: osdmap e27: 4 total, 4 up, 4 in 2024-09-19T06:44:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi114 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 8544M 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:44:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:44:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:43 smithi114 ceph-mon[30820]: osdmap e27: 4 total, 4 up, 4 in 2024-09-19T06:44:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:45 smithi116 ceph-mon[35411]: osdmap e28: 4 total, 4 up, 4 in 2024-09-19T06:44:45.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:45 smithi118 ceph-mon[35527]: osdmap e28: 4 total, 4 up, 4 in 2024-09-19T06:44:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:45 smithi114 ceph-mon[30820]: osdmap e28: 4 total, 4 up, 4 in 2024-09-19T06:44:46.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:46 smithi118 ceph-mon[35527]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:46.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:46 smithi118 ceph-mon[35527]: osdmap e29: 4 total, 4 up, 4 in 2024-09-19T06:44:46.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:46 smithi118 ceph-mon[35527]: from='client.? 172.21.15.116:0/2548499799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]: dispatch 2024-09-19T06:44:46.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:46 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]: dispatch 2024-09-19T06:44:46.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:46 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]': finished 2024-09-19T06:44:46.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:46 smithi118 ceph-mon[35527]: osdmap e30: 5 total, 4 up, 5 in 2024-09-19T06:44:46.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:44:46.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:46 smithi118 ceph-mon[35527]: from='client.? 172.21.15.116:0/1518338149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:44:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:46 smithi114 ceph-mon[30820]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:46 smithi114 ceph-mon[30820]: osdmap e29: 4 total, 4 up, 4 in 2024-09-19T06:44:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:46 smithi114 ceph-mon[30820]: from='client.? 172.21.15.116:0/2548499799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]: dispatch 2024-09-19T06:44:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:46 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]: dispatch 2024-09-19T06:44:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:46 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]': finished 2024-09-19T06:44:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:46 smithi114 ceph-mon[30820]: osdmap e30: 5 total, 4 up, 5 in 2024-09-19T06:44:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:44:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:46 smithi114 ceph-mon[30820]: from='client.? 172.21.15.116:0/1518338149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:44:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:46 smithi116 ceph-mon[35411]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:46 smithi116 ceph-mon[35411]: osdmap e29: 4 total, 4 up, 4 in 2024-09-19T06:44:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:46 smithi116 ceph-mon[35411]: from='client.? 172.21.15.116:0/2548499799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]: dispatch 2024-09-19T06:44:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:46 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]: dispatch 2024-09-19T06:44:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:46 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3898d363-6848-46d1-b062-93850b994735"}]': finished 2024-09-19T06:44:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:46 smithi116 ceph-mon[35411]: osdmap e30: 5 total, 4 up, 5 in 2024-09-19T06:44:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:44:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:46 smithi116 ceph-mon[35411]: from='client.? 172.21.15.116:0/1518338149' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:44:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:48 smithi118 ceph-mon[35527]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:48 smithi114 ceph-mon[30820]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:48 smithi116 ceph-mon[35411]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:44:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:50 smithi118 ceph-mon[35527]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-19T06:44:50.622 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:50 smithi116 ceph-mon[35411]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-19T06:44:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:50 smithi114 ceph-mon[30820]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-19T06:44:52.490 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:52 smithi116 ceph-mon[35411]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-19T06:44:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:52 smithi118 ceph-mon[35527]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-19T06:44:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:52 smithi114 ceph-mon[30820]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-19T06:44:54.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:54 smithi118 ceph-mon[35527]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T06:44:54.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:54 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-19T06:44:54.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:54 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:54 smithi114 ceph-mon[30820]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T06:44:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-19T06:44:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:54 smithi116 ceph-mon[35411]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T06:44:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-19T06:44:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:44:55.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:55 smithi116 ceph-mon[35411]: Deploying daemon osd.4 on smithi116 2024-09-19T06:44:55.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:55 smithi118 ceph-mon[35527]: Deploying daemon osd.4 on smithi116 2024-09-19T06:44:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:55 smithi114 ceph-mon[30820]: Deploying daemon osd.4 on smithi116 2024-09-19T06:44:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:56 smithi118 ceph-mon[35527]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-19T06:44:56.632 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:56 smithi116 ceph-mon[35411]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-19T06:44:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:56 smithi114 ceph-mon[30820]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-19T06:44:58.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:44:58 smithi118 ceph-mon[35527]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-19T06:44:58.512 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:44:58 smithi116 ceph-mon[35411]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-19T06:44:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:44:58 smithi114 ceph-mon[30820]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-19T06:45:00.505 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:00 smithi116 ceph-mon[35411]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:45:00.505 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:00.505 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:00.505 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:00 smithi118 ceph-mon[35527]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:45:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:00 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:00 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:00 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:00 smithi114 ceph-mon[30820]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:45:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:02 smithi116 ceph-mon[35411]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:02 smithi114 ceph-mon[30820]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:02 smithi118 ceph-mon[35527]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:02 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:02 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:02 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:02.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:02 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:02.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:02 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:03.335 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:03 smithi116 ceph-mon[35411]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:03 smithi114 ceph-mon[30820]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:03.669 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 4 on host 'smithi116' 2024-09-19T06:45:03.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.667+0000 7f152cff9640 1 -- 172.21.15.116:0/286575649 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1530062630 con 0x7f15180a3270 2024-09-19T06:45:03.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f15180a3270 msgr2=0x7f15180a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:03.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f15180a3270 0x7f15180a5730 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15200058c0 tx=0x7f1520005850 comp rx=0 tx=0).stop 2024-09-19T06:45:03.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 msgr2=0x7f1530198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 0x7f1530198380 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1524009230 tx=0x7f1524004360 comp rx=0 tx=0).stop 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 shutdown_connections 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f15180a3270 0x7f15180a5730 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f15301026d0 0x7f153019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1530101d10 0x7f1530198380 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 --2- 172.21.15.116:0/286575649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1530100b10 0x7f1530197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 >> 172.21.15.116:0/286575649 conn(0x7f15300fc2a0 msgr2=0x7f1530105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 shutdown_connections 2024-09-19T06:45:03.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:45:03.670+0000 7f153619d640 1 -- 172.21.15.116:0/286575649 wait complete. 2024-09-19T06:45:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:03 smithi118 ceph-mon[35527]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:04.308 DEBUG:teuthology.orchestra.run.smithi116:osd.4> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.4.service 2024-09-19T06:45:04.310 INFO:tasks.cephadm:Deploying osd.5 on smithi118 with /dev/vg_nvme/lv_3... 2024-09-19T06:45:04.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-19T06:45:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:04.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:04.524 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:04.565 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:45:04.657 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:04 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:04.657 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:04 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:04.657 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:04 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:05.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:05 smithi118 ceph-mon[35527]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:05.527 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:05 smithi116 ceph-mon[35411]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:05 smithi114 ceph-mon[30820]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:06.179 INFO:journalctl@ceph.osd.4.smithi116.stdout:Sep 19 06:45:05 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-4[46486]: 2024-09-19T06:45:05.805+0000 7fb3575b5640 -1 osd.4 0 log_to_monitors true 2024-09-19T06:45:06.482 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:06 smithi116 ceph-mon[35411]: from='osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T06:45:06.483 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:06 smithi116 ceph-mon[35411]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T06:45:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:06 smithi118 ceph-mon[35527]: from='osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T06:45:06.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:06 smithi118 ceph-mon[35527]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T06:45:06.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:06 smithi114 ceph-mon[30820]: from='osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T06:45:06.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:06 smithi114 ceph-mon[30820]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-19T06:45:07.235 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:45:07.262 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi118:vg_nvme/lv_3 2024-09-19T06:45:07.428 INFO:journalctl@ceph.osd.4.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-4[46486]: 2024-09-19T06:45:07.288+0000 7fb353538640 -1 osd.4 0 waiting for initial osdmap 2024-09-19T06:45:07.429 INFO:journalctl@ceph.osd.4.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-4[46486]: 2024-09-19T06:45:07.294+0000 7fb34f360640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: osdmap e31: 5 total, 4 up, 5 in 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi116 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 8544M 2024-09-19T06:45:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-19T06:45:07.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: osdmap e31: 5 total, 4 up, 5 in 2024-09-19T06:45:07.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:07.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:45:07.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi116 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi116 to 8544M 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:07.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.509 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:45:07.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-19T06:45:07.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: osdmap e31: 5 total, 4 up, 5 in 2024-09-19T06:45:07.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:07.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:45:07.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:45:07.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-09-19T06:45:07.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi116 2024-09-19T06:45:07.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:07.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:07.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 8544M 2024-09-19T06:45:07.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:07.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:07.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:07.544 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4040206427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 msgr2=0x7f81f8100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4040206427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 0x7f81f8100f00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009900 tx=0x7f81e802a410 comp rx=0 tx=0).stop 2024-09-19T06:45:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4040206427 shutdown_connections 2024-09-19T06:45:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4040206427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f81026c0 0x7f81f8108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4040206427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81f8101d00 0x7f81f8102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4040206427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 0x7f81f8100f00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4040206427 >> 172.21.15.118:0/4040206427 conn(0x7f81f80fc270 msgr2=0x7f81f80fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4040206427 shutdown_connections 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.978+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4040206427 wait complete. 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81ff9f0640 1 Processor -- start 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81ff9f0640 1 -- start start 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81ff9f0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 0x7f81f8196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81ff9f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f8101d00 0x7f81f8197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81ff9f0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81f81026c0 0x7f81f819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81ff9f0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f81f810c760 con 0x7f81f81026c0 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81ff9f0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81f810c5e0 con 0x7f81f8101d00 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81ff9f0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f81f810c8e0 con 0x7f81f8100b00 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 0x7f81f8196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:45:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 0x7f81f8196e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:47058/0 (socket says 172.21.15.118:47058) 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 -- 172.21.15.118:0/4167224507 learned_addr learned my addr 172.21.15.118:0/4167224507 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fdf8d640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81f81026c0 0x7f81f819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fcf8b640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f8101d00 0x7f81f8197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 -- 172.21.15.118:0/4167224507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f8101d00 msgr2=0x7f81f8197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f8101d00 0x7f81f8197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 -- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81f81026c0 msgr2=0x7f81f819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81f81026c0 0x7f81f819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 -- 172.21.15.118:0/4167224507 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f819ea50 con 0x7f81f8100b00 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.979+0000 7f81fd78c640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 0x7f81f8196e20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f81e8006010 tx=0x7f81e8031b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.980+0000 7f81ee7fc640 1 -- 172.21.15.118:0/4167224507 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e803f030 con 0x7f81f8100b00 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.980+0000 7f81ee7fc640 1 -- 172.21.15.118:0/4167224507 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81e80044a0 con 0x7f81f8100b00 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.980+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f819ece0 con 0x7f81f8100b00 2024-09-19T06:45:08.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.980+0000 7f81ee7fc640 1 -- 172.21.15.118:0/4167224507 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e8035070 con 0x7f81f8100b00 2024-09-19T06:45:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.980+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f819f1c0 con 0x7f81f8100b00 2024-09-19T06:45:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.981+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f8107fa0 con 0x7f81f8100b00 2024-09-19T06:45:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.982+0000 7f81ee7fc640 1 -- 172.21.15.118:0/4167224507 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f81e8042030 con 0x7f81f8100b00 2024-09-19T06:45:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.983+0000 7f81ee7fc640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f81e40a3270 0x7f81e40a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:45:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.983+0000 7f81ee7fc640 1 -- 172.21.15.118:0/4167224507 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 3821+0+0 (secure 0 0 0) 0x7f81e80eefe0 con 0x7f81f8100b00 2024-09-19T06:45:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.983+0000 7f81fcf8b640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f81e40a3270 0x7f81e40a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:45:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.983+0000 7f81fcf8b640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f81e40a3270 0x7f81e40a5730 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f81f4005fd0 tx=0x7f81f4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:45:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:07.990+0000 7f81ee7fc640 1 -- 172.21.15.118:0/4167224507 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f81e80bce90 con 0x7f81f8100b00 2024-09-19T06:45:08.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:08.171+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f81f8062630 con 0x7f81e40a3270 2024-09-19T06:45:08.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:08 smithi118 ceph-mon[35527]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:45:08.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:08 smithi118 ceph-mon[35527]: osdmap e32: 5 total, 4 up, 5 in 2024-09-19T06:45:08.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:08.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:08.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:08 smithi118 ceph-mon[35527]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:45:08.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:45:08.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:45:08.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:08 smithi114 ceph-mon[30820]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:45:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:08 smithi114 ceph-mon[30820]: osdmap e32: 5 total, 4 up, 5 in 2024-09-19T06:45:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:08 smithi114 ceph-mon[30820]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:45:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:45:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:45:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:08 smithi116 ceph-mon[35411]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:45:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:08 smithi116 ceph-mon[35411]: osdmap e32: 5 total, 4 up, 5 in 2024-09-19T06:45:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:08 smithi116 ceph-mon[35411]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:45:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:45:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:45:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:09 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:45:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:09 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:45:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:09 smithi114 ceph-mon[30820]: osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] boot 2024-09-19T06:45:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:09 smithi114 ceph-mon[30820]: osdmap e33: 5 total, 5 up, 5 in 2024-09-19T06:45:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:09 smithi114 ceph-mon[30820]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:09 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:45:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:09 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:45:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:09 smithi116 ceph-mon[35411]: osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] boot 2024-09-19T06:45:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:09 smithi116 ceph-mon[35411]: osdmap e33: 5 total, 5 up, 5 in 2024-09-19T06:45:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:09 smithi116 ceph-mon[35411]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:09.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:09 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:45:09.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:09 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:45:09.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:09 smithi118 ceph-mon[35527]: osd.4 [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] boot 2024-09-19T06:45:09.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:09 smithi118 ceph-mon[35527]: osdmap e33: 5 total, 5 up, 5 in 2024-09-19T06:45:09.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:09 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-19T06:45:09.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:09 smithi118 ceph-mon[35527]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:10 smithi114 ceph-mon[30820]: osdmap e34: 5 total, 5 up, 5 in 2024-09-19T06:45:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:10 smithi116 ceph-mon[35411]: osdmap e34: 5 total, 5 up, 5 in 2024-09-19T06:45:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:10 smithi118 ceph-mon[35527]: osdmap e34: 5 total, 5 up, 5 in 2024-09-19T06:45:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:11 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/486076189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]: dispatch 2024-09-19T06:45:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:11 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]: dispatch 2024-09-19T06:45:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:11 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]': finished 2024-09-19T06:45:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:11 smithi114 ceph-mon[30820]: osdmap e35: 6 total, 5 up, 6 in 2024-09-19T06:45:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:11 smithi114 ceph-mon[30820]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:11 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/1176346150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:45:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:11 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/486076189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]: dispatch 2024-09-19T06:45:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:11 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]: dispatch 2024-09-19T06:45:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:11 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]': finished 2024-09-19T06:45:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:11 smithi116 ceph-mon[35411]: osdmap e35: 6 total, 5 up, 6 in 2024-09-19T06:45:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:11 smithi116 ceph-mon[35411]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:11 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/1176346150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:45:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:11 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/486076189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]: dispatch 2024-09-19T06:45:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:11 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]: dispatch 2024-09-19T06:45:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:11 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd64637-ce99-4ea8-bee2-049d202c4720"}]': finished 2024-09-19T06:45:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:11 smithi118 ceph-mon[35527]: osdmap e35: 6 total, 5 up, 6 in 2024-09-19T06:45:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:11.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:11 smithi118 ceph-mon[35527]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:11.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:11 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/1176346150' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:45:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:14 smithi114 ceph-mon[30820]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:14 smithi116 ceph-mon[35411]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:14.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:14 smithi118 ceph-mon[35527]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:16.207 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:16 smithi118 ceph-mon[35527]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:16 smithi114 ceph-mon[30820]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:16 smithi116 ceph-mon[35411]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:18.212 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:18 smithi118 ceph-mon[35527]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:18 smithi114 ceph-mon[30820]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:18 smithi116 ceph-mon[35411]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:20.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:20 smithi118 ceph-mon[35527]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:20.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-19T06:45:20.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:20 smithi114 ceph-mon[30820]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-19T06:45:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:20 smithi116 ceph-mon[35411]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-19T06:45:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:21.381 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:21 smithi118 ceph-mon[35527]: Deploying daemon osd.5 on smithi118 2024-09-19T06:45:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:21 smithi114 ceph-mon[30820]: Deploying daemon osd.5 on smithi118 2024-09-19T06:45:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:21 smithi116 ceph-mon[35411]: Deploying daemon osd.5 on smithi118 2024-09-19T06:45:22.393 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:22 smithi118 ceph-mon[35527]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:22 smithi114 ceph-mon[30820]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:22 smithi116 ceph-mon[35411]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:24 smithi114 ceph-mon[30820]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:24 smithi116 ceph-mon[35411]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:24 smithi118 ceph-mon[35527]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:26.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:26 smithi118 ceph-mon[35527]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:26.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:26 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:26.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:26 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:26.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:26 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:26 smithi114 ceph-mon[30820]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:26 smithi116 ceph-mon[35411]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:28 smithi114 ceph-mon[30820]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:28 smithi116 ceph-mon[35411]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:28.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:28 smithi118 ceph-mon[35527]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:29 smithi114 ceph-mon[30820]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:29 smithi116 ceph-mon[35411]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:29.712 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.712 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.712 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:29.712 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:29.712 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:29.712 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:29 smithi118 ceph-mon[35527]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:29.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.754+0000 7f81ee7fc640 1 -- 172.21.15.118:0/4167224507 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f81f8062630 con 0x7f81e40a3270 2024-09-19T06:45:29.755 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 5 on host 'smithi118' 2024-09-19T06:45:29.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f81e40a3270 msgr2=0x7f81e40a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:29.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f81e40a3270 0x7f81e40a5730 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f81f4005fd0 tx=0x7f81f4005850 comp rx=0 tx=0).stop 2024-09-19T06:45:29.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 msgr2=0x7f81f8196e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:29.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 0x7f81f8196e20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f81e8006010 tx=0x7f81e8031b60 comp rx=0 tx=0).stop 2024-09-19T06:45:29.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 shutdown_connections 2024-09-19T06:45:29.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f81e40a3270 0x7f81e40a5730 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:29.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81f81026c0 0x7f81f819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:29.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f8101d00 0x7f81f8197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:29.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 --2- 172.21.15.118:0/4167224507 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8100b00 0x7f81f8196e20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:29.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 >> 172.21.15.118:0/4167224507 conn(0x7f81f80fc270 msgr2=0x7f81f8105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:45:29.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 shutdown_connections 2024-09-19T06:45:29.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:45:29.756+0000 7f81ff9f0640 1 -- 172.21.15.118:0/4167224507 wait complete. 2024-09-19T06:45:30.449 DEBUG:teuthology.orchestra.run.smithi118:osd.5> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.5.service 2024-09-19T06:45:30.456 INFO:tasks.cephadm:Deploying osd.6 on smithi114 with /dev/vg_nvme/lv_2... 2024-09-19T06:45:30.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-19T06:45:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:30.725 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:30 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:30.725 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:30 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:30.725 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:30 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:30.834 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:45:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:31.628 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:31 smithi118 ceph-mon[35527]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:31.628 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:31 smithi118 ceph-mon[35527]: from='osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T06:45:31.628 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:31 smithi118 ceph-mon[35527]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T06:45:31.628 INFO:journalctl@ceph.osd.5.smithi118.stdout:Sep 19 06:45:31 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-5[47391]: 2024-09-19T06:45:31.284+0000 7fe90e11b640 -1 osd.5 0 log_to_monitors true 2024-09-19T06:45:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:31 smithi114 ceph-mon[30820]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:31 smithi114 ceph-mon[30820]: from='osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T06:45:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:31 smithi114 ceph-mon[30820]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T06:45:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:31 smithi116 ceph-mon[35411]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:31 smithi116 ceph-mon[35411]: from='osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T06:45:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:31 smithi116 ceph-mon[35411]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-19T06:45:32.740 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:32 smithi118 ceph-mon[35527]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-19T06:45:32.740 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:32 smithi118 ceph-mon[35527]: from='osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:45:32.740 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:32 smithi118 ceph-mon[35527]: osdmap e36: 6 total, 5 up, 6 in 2024-09-19T06:45:32.740 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:32.740 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:32 smithi118 ceph-mon[35527]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:45:32.740 INFO:journalctl@ceph.osd.5.smithi118.stdout:Sep 19 06:45:32 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-5[47391]: 2024-09-19T06:45:32.438+0000 7fe90a09e640 -1 osd.5 0 waiting for initial osdmap 2024-09-19T06:45:32.741 INFO:journalctl@ceph.osd.5.smithi118.stdout:Sep 19 06:45:32 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-5[47391]: 2024-09-19T06:45:32.443+0000 7fe905ec6640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:45:32.750 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:32 smithi114 ceph-mon[30820]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-19T06:45:32.750 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:32 smithi114 ceph-mon[30820]: from='osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:45:32.750 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:32 smithi114 ceph-mon[30820]: osdmap e36: 6 total, 5 up, 6 in 2024-09-19T06:45:32.750 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:32.750 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:32 smithi114 ceph-mon[30820]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:45:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:32 smithi116 ceph-mon[35411]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-19T06:45:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:32 smithi116 ceph-mon[35411]: from='osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:45:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:32 smithi116 ceph-mon[35411]: osdmap e36: 6 total, 5 up, 6 in 2024-09-19T06:45:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:32 smithi116 ceph-mon[35411]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:45:33.695 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:33 smithi114 ceph-mon[30820]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:45:33.695 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:33 smithi114 ceph-mon[30820]: osdmap e37: 6 total, 5 up, 6 in 2024-09-19T06:45:33.695 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:33.695 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:33.695 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:33 smithi114 ceph-mon[30820]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:33.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:33 smithi118 ceph-mon[35527]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:45:33.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:33 smithi118 ceph-mon[35527]: osdmap e37: 6 total, 5 up, 6 in 2024-09-19T06:45:33.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:33 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:33.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:33 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:33.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:33 smithi118 ceph-mon[35527]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:33 smithi116 ceph-mon[35411]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:45:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:33 smithi116 ceph-mon[35411]: osdmap e37: 6 total, 5 up, 6 in 2024-09-19T06:45:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:33 smithi116 ceph-mon[35411]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-09-19T06:45:34.097 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:45:34.132 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-09-19T06:45:34.508 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:45:34.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:45:34.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] boot 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: osdmap e38: 6 total, 6 up, 6 in 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi118 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi118 to 8544M 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:34.726 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:34 smithi114 ceph-mon[30820]: osdmap e39: 6 total, 6 up, 6 in 2024-09-19T06:45:34.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:45:34.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] boot 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: osdmap e38: 6 total, 6 up, 6 in 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi118 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi118 to 8544M 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:34.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:34 smithi118 ceph-mon[35527]: osdmap e39: 6 total, 6 up, 6 in 2024-09-19T06:45:34.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:45:34.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:45:34.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: osd.5 [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] boot 2024-09-19T06:45:34.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: osdmap e38: 6 total, 6 up, 6 in 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi118 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi118 to 8544M 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:34.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:34 smithi116 ceph-mon[35411]: osdmap e39: 6 total, 6 up, 6 in 2024-09-19T06:45:35.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.162+0000 7f2564400640 1 -- 172.21.15.114:0/2423940581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 msgr2=0x7f255c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.162+0000 7f2564400640 1 --2- 172.21.15.114:0/2423940581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 0x7f255c1053e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2548009900 tx=0x7f254802a3f0 comp rx=0 tx=0).stop 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.162+0000 7f2564400640 1 -- 172.21.15.114:0/2423940581 shutdown_connections 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.163+0000 7f2564400640 1 --2- 172.21.15.114:0/2423940581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f255c105920 0x7f255c10c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.163+0000 7f2564400640 1 --2- 172.21.15.114:0/2423940581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 0x7f255c1053e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.163+0000 7f2564400640 1 --2- 172.21.15.114:0/2423940581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f255c103e00 0x7f255c104200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.163+0000 7f2564400640 1 -- 172.21.15.114:0/2423940581 >> 172.21.15.114:0/2423940581 conn(0x7f255c0ff5b0 msgr2=0x7f255c1019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.163+0000 7f2564400640 1 -- 172.21.15.114:0/2423940581 shutdown_connections 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.163+0000 7f2564400640 1 -- 172.21.15.114:0/2423940581 wait complete. 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.164+0000 7f2564400640 1 Processor -- start 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.164+0000 7f2564400640 1 -- start start 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.164+0000 7f2564400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f255c103e00 0x7f255c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:45:35.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.164+0000 7f2564400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 0x7f255c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.164+0000 7f2564400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f255c105920 0x7f255c1a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.164+0000 7f2564400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f255c10fab0 con 0x7f255c105920 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.164+0000 7f2564400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f255c10f930 con 0x7f255c104f60 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.164+0000 7f2564400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f255c10fc30 con 0x7f255c103e00 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 0x7f255c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 0x7f255c19a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:35822/0 (socket says 172.21.15.114:35822) 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2562400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f255c105920 0x7f255c1a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f255c103e00 0x7f255c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 -- 172.21.15.114:0/1429579335 learned_addr learned my addr 172.21.15.114:0/1429579335 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 -- 172.21.15.114:0/1429579335 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f255c103e00 msgr2=0x7f255c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f255c103e00 0x7f255c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:35.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 -- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f255c105920 msgr2=0x7f255c1a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f255c105920 0x7f255c1a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 -- 172.21.15.114:0/1429579335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255c1a1d30 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.165+0000 7f2561000640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 0x7f255c19a640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2548031610 tx=0x7f2548004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.166+0000 7f255a000640 1 -- 172.21.15.114:0/1429579335 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254803f030 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.166+0000 7f255a000640 1 -- 172.21.15.114:0/1429579335 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2548003ad0 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.166+0000 7f255a000640 1 -- 172.21.15.114:0/1429579335 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548035070 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.166+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255c1a1fc0 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.166+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f255c1a2360 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.167+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f255c10b240 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.169+0000 7f255a000640 1 -- 172.21.15.114:0/1429579335 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f2548042030 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.170+0000 7f255a000640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f25440a3270 0x7f25440a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.170+0000 7f255a000640 1 -- 172.21.15.114:0/1429579335 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4562+0+0 (secure 0 0 0) 0x7f25480ef260 con 0x7f255c104f60 2024-09-19T06:45:35.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.170+0000 7f2561a00640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f25440a3270 0x7f25440a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:45:35.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.171+0000 7f2561a00640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f25440a3270 0x7f25440a5730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2550009bd0 tx=0x7f2550005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:45:35.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.176+0000 7f255a000640 1 -- 172.21.15.114:0/1429579335 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f25480bc730 con 0x7f255c104f60 2024-09-19T06:45:35.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:35.364+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f255c062630 con 0x7f25440a3270 2024-09-19T06:45:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:36 smithi116 ceph-mon[35411]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:36 smithi116 ceph-mon[35411]: osdmap e40: 6 total, 6 up, 6 in 2024-09-19T06:45:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:45:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:45:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:36 smithi118 ceph-mon[35527]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:36 smithi118 ceph-mon[35527]: osdmap e40: 6 total, 6 up, 6 in 2024-09-19T06:45:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:36 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:45:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:36 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:45:36.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:36 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:36 smithi114 ceph-mon[30820]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:36 smithi114 ceph-mon[30820]: osdmap e40: 6 total, 6 up, 6 in 2024-09-19T06:45:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:45:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:45:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:37 smithi114 ceph-mon[30820]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:45:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:37 smithi114 ceph-mon[30820]: osdmap e41: 6 total, 6 up, 6 in 2024-09-19T06:45:37.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:37 smithi118 ceph-mon[35527]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:45:37.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:37 smithi118 ceph-mon[35527]: osdmap e41: 6 total, 6 up, 6 in 2024-09-19T06:45:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:37 smithi116 ceph-mon[35411]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:45:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:37 smithi116 ceph-mon[35411]: osdmap e41: 6 total, 6 up, 6 in 2024-09-19T06:45:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:38 smithi114 ceph-mon[30820]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:38 smithi114 ceph-mon[30820]: osdmap e42: 6 total, 6 up, 6 in 2024-09-19T06:45:38.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:38 smithi118 ceph-mon[35527]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:38.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:38 smithi118 ceph-mon[35527]: osdmap e42: 6 total, 6 up, 6 in 2024-09-19T06:45:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:38 smithi116 ceph-mon[35411]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:38 smithi116 ceph-mon[35411]: osdmap e42: 6 total, 6 up, 6 in 2024-09-19T06:45:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:39 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/372060939' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc797e6c-f130-4fd1-ba32-cb8a989cfd81"}]: dispatch 2024-09-19T06:45:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:39 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/372060939' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc797e6c-f130-4fd1-ba32-cb8a989cfd81"}]': finished 2024-09-19T06:45:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:39 smithi114 ceph-mon[30820]: osdmap e43: 7 total, 6 up, 7 in 2024-09-19T06:45:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:45:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:39 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/68386067' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:45:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:39 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/372060939' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc797e6c-f130-4fd1-ba32-cb8a989cfd81"}]: dispatch 2024-09-19T06:45:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:39 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/372060939' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc797e6c-f130-4fd1-ba32-cb8a989cfd81"}]': finished 2024-09-19T06:45:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:39 smithi118 ceph-mon[35527]: osdmap e43: 7 total, 6 up, 7 in 2024-09-19T06:45:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:39 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:45:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:39 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/68386067' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:45:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:39 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/372060939' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc797e6c-f130-4fd1-ba32-cb8a989cfd81"}]: dispatch 2024-09-19T06:45:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:39 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/372060939' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc797e6c-f130-4fd1-ba32-cb8a989cfd81"}]': finished 2024-09-19T06:45:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:39 smithi116 ceph-mon[35411]: osdmap e43: 7 total, 6 up, 7 in 2024-09-19T06:45:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:45:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:39 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/68386067' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:45:40.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:40 smithi118 ceph-mon[35527]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-19T06:45:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:40 smithi114 ceph-mon[30820]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-19T06:45:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:40 smithi116 ceph-mon[35411]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-19T06:45:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:42 smithi118 ceph-mon[35527]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-19T06:45:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:42 smithi114 ceph-mon[30820]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-19T06:45:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:42 smithi116 ceph-mon[35411]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-19T06:45:44.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:44 smithi118 ceph-mon[35527]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-19T06:45:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:44 smithi114 ceph-mon[30820]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-19T06:45:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:44 smithi116 ceph-mon[35411]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-09-19T06:45:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:46 smithi116 ceph-mon[35411]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T06:45:46.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:46 smithi114 ceph-mon[30820]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T06:45:46.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:46 smithi118 ceph-mon[35527]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-19T06:45:48.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:48 smithi114 ceph-mon[30820]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-19T06:45:48.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-19T06:45:48.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:48 smithi116 ceph-mon[35411]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-19T06:45:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-19T06:45:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:48.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:48 smithi118 ceph-mon[35527]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-19T06:45:48.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:48 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-19T06:45:48.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:48 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:49.529 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:49 smithi114 ceph-mon[30820]: Deploying daemon osd.6 on smithi114 2024-09-19T06:45:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:49 smithi116 ceph-mon[35411]: Deploying daemon osd.6 on smithi114 2024-09-19T06:45:49.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:49 smithi118 ceph-mon[35527]: Deploying daemon osd.6 on smithi114 2024-09-19T06:45:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:50 smithi114 ceph-mon[30820]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:50 smithi116 ceph-mon[35411]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:50.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:50 smithi118 ceph-mon[35527]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:51.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:51 smithi114 ceph-mon[30820]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:51 smithi116 ceph-mon[35411]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:51.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:51 smithi118 ceph-mon[35527]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:54 smithi114 ceph-mon[30820]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:54 smithi116 ceph-mon[35411]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:54.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:54 smithi118 ceph-mon[35527]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:56 smithi114 ceph-mon[30820]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:56 smithi116 ceph-mon[35411]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:56.512 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:56.513 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:56.513 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:56.513 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:56 smithi118 ceph-mon[35527]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:56.513 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:45:56.513 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:56.513 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:56 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:58 smithi114 ceph-mon[30820]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:58 smithi116 ceph-mon[35411]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:58.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:58 smithi118 ceph-mon[35527]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:45:59 smithi114 ceph-mon[30820]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:59.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:59.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:45:59 smithi118 ceph-mon[35527]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:45:59.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.888+0000 7f255a000640 1 -- 172.21.15.114:0/1429579335 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f255c062630 con 0x7f25440a3270 2024-09-19T06:45:59.890 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 6 on host 'smithi114' 2024-09-19T06:45:59.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.890+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f25440a3270 msgr2=0x7f25440a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:59.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.890+0000 7f2564400640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f25440a3270 0x7f25440a5730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2550009bd0 tx=0x7f2550005f90 comp rx=0 tx=0).stop 2024-09-19T06:45:59.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.890+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 msgr2=0x7f255c19a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:45:59.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.890+0000 7f2564400640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 0x7f255c19a640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2548031610 tx=0x7f2548004270 comp rx=0 tx=0).stop 2024-09-19T06:45:59.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.891+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 shutdown_connections 2024-09-19T06:45:59.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.891+0000 7f2564400640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f25440a3270 0x7f25440a5730 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:59.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.891+0000 7f2564400640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f255c105920 0x7f255c1a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:59.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.891+0000 7f2564400640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f255c104f60 0x7f255c19a640 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:59.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.891+0000 7f2564400640 1 --2- 172.21.15.114:0/1429579335 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f255c103e00 0x7f255c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:45:59.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.891+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 >> 172.21.15.114:0/1429579335 conn(0x7f255c0ff5b0 msgr2=0x7f255c108410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:45:59.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.891+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 shutdown_connections 2024-09-19T06:45:59.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:45:59.891+0000 7f2564400640 1 -- 172.21.15.114:0/1429579335 wait complete. 2024-09-19T06:45:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:45:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:45:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:45:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:45:59 smithi116 ceph-mon[35411]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:46:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:00 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:00 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:00 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:00.765 DEBUG:teuthology.orchestra.run.smithi114:osd.6> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.6.service 2024-09-19T06:46:00.770 INFO:tasks.cephadm:Deploying osd.7 on smithi116 with /dev/vg_nvme/lv_2... 2024-09-19T06:46:00.770 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-19T06:46:00.798 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:00.798 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:00.798 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:00.996 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 19 06:46:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-6[57978]: 2024-09-19T06:46:00.914+0000 7f5c00c21640 -1 osd.6 0 log_to_monitors true 2024-09-19T06:46:01.025 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T06:46:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:01 smithi114 ceph-mon[30820]: from='osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T06:46:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:01 smithi114 ceph-mon[30820]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T06:46:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:01 smithi114 ceph-mon[30820]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:46:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:01 smithi116 ceph-mon[35411]: from='osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T06:46:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:01 smithi116 ceph-mon[35411]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T06:46:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:01 smithi116 ceph-mon[35411]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:46:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:01 smithi118 ceph-mon[35527]: from='osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T06:46:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:01 smithi118 ceph-mon[35527]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-19T06:46:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:01 smithi118 ceph-mon[35527]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:46:02.724 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:02 smithi114 ceph-mon[30820]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-19T06:46:02.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:02 smithi114 ceph-mon[30820]: from='osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:46:02.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:02 smithi114 ceph-mon[30820]: osdmap e44: 7 total, 6 up, 7 in 2024-09-19T06:46:02.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:02.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:02 smithi114 ceph-mon[30820]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:46:02.744 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:02 smithi116 ceph-mon[35411]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-19T06:46:02.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:02 smithi116 ceph-mon[35411]: from='osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:46:02.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:02 smithi116 ceph-mon[35411]: osdmap e44: 7 total, 6 up, 7 in 2024-09-19T06:46:02.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:02.745 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:02 smithi116 ceph-mon[35411]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:46:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:02 smithi118 ceph-mon[35527]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-19T06:46:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:02 smithi118 ceph-mon[35527]: from='osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:46:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:02 smithi118 ceph-mon[35527]: osdmap e44: 7 total, 6 up, 7 in 2024-09-19T06:46:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:02 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:02.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:02 smithi118 ceph-mon[35527]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:46:03.416 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 19 06:46:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-6[57978]: 2024-09-19T06:46:03.016+0000 7f5bfb800640 -1 osd.6 0 waiting for initial osdmap 2024-09-19T06:46:03.416 INFO:journalctl@ceph.osd.6.smithi114.stdout:Sep 19 06:46:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-6[57978]: 2024-09-19T06:46:03.034+0000 7f5bf1800640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:46:03.590 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:03 smithi116 ceph-mon[35411]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:46:03.591 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:03 smithi116 ceph-mon[35411]: osdmap e45: 7 total, 6 up, 7 in 2024-09-19T06:46:03.591 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:03.591 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:03.591 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:03 smithi116 ceph-mon[35411]: from='osd.6 ' entity='osd.6' 2024-09-19T06:46:03.591 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:03 smithi116 ceph-mon[35411]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:46:03.593 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:46:03.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-09-19T06:46:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:03 smithi118 ceph-mon[35527]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:46:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:03 smithi118 ceph-mon[35527]: osdmap e45: 7 total, 6 up, 7 in 2024-09-19T06:46:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:03 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:03 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:03 smithi118 ceph-mon[35527]: from='osd.6 ' entity='osd.6' 2024-09-19T06:46:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:03 smithi118 ceph-mon[35527]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:46:03.908 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T06:46:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:03 smithi114 ceph-mon[30820]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:46:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:03 smithi114 ceph-mon[30820]: osdmap e45: 7 total, 6 up, 7 in 2024-09-19T06:46:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:03 smithi114 ceph-mon[30820]: from='osd.6 ' entity='osd.6' 2024-09-19T06:46:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:03 smithi114 ceph-mon[30820]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.392+0000 7fc35466b640 1 -- 172.21.15.116:0/1402081227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c1037c0 msgr2=0x7fc34c10a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.392+0000 7fc35466b640 1 --2- 172.21.15.116:0/1402081227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c1037c0 0x7fc34c10a050 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc344009ca0 tx=0x7fc34402a8c0 comp rx=0 tx=0).stop 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 -- 172.21.15.116:0/1402081227 shutdown_connections 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 --2- 172.21.15.116:0/1402081227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c1037c0 0x7fc34c10a050 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 --2- 172.21.15.116:0/1402081227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc34c102e00 0x7fc34c103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 --2- 172.21.15.116:0/1402081227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc34c101c00 0x7fc34c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 -- 172.21.15.116:0/1402081227 >> 172.21.15.116:0/1402081227 conn(0x7fc34c0fd390 msgr2=0x7fc34c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 -- 172.21.15.116:0/1402081227 shutdown_connections 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 -- 172.21.15.116:0/1402081227 wait complete. 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 Processor -- start 2024-09-19T06:46:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 -- start start 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc34c101c00 0x7fc34c075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c102e00 0x7fc34c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc34c1037c0 0x7fc34c074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc34c10d880 con 0x7fc34c101c00 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc34c10d700 con 0x7fc34c102e00 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.393+0000 7fc35466b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc34c10da00 con 0x7fc34c1037c0 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c102e00 0x7fc34c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c102e00 0x7fc34c0745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36344/0 (socket says 172.21.15.116:36344) 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 -- 172.21.15.116:0/2591763369 learned_addr learned my addr 172.21.15.116:0/2591763369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc352407640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc34c101c00 0x7fc34c075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc352c08640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc34c1037c0 0x7fc34c074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 -- 172.21.15.116:0/2591763369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc34c1037c0 msgr2=0x7fc34c074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc34c1037c0 0x7fc34c074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 -- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc34c101c00 msgr2=0x7fc34c075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc34c101c00 0x7fc34c075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 -- 172.21.15.116:0/2591763369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34c0753e0 con 0x7fc34c102e00 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc352c08640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc34c1037c0 0x7fc34c074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc351c06640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c102e00 0x7fc34c0745e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc34800d770 tx=0x7fc34800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc3437fe640 1 -- 172.21.15.116:0/2591763369 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348017070 con 0x7fc34c102e00 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc3437fe640 1 -- 172.21.15.116:0/2591763369 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc348005500 con 0x7fc34c102e00 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc34c1a2060 con 0x7fc34c102e00 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc3437fe640 1 -- 172.21.15.116:0/2591763369 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348004e00 con 0x7fc34c102e00 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.394+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc34c1a2570 con 0x7fc34c102e00 2024-09-19T06:46:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.395+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc34c108fb0 con 0x7fc34c102e00 2024-09-19T06:46:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.396+0000 7fc3437fe640 1 -- 172.21.15.116:0/2591763369 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fc348004fa0 con 0x7fc34c102e00 2024-09-19T06:46:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.397+0000 7fc3437fe640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc3380a32c0 0x7fc3380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.397+0000 7fc3437fe640 1 -- 172.21.15.116:0/2591763369 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 4965+0+0 (secure 0 0 0) 0x7fc3480ce5d0 con 0x7fc34c102e00 2024-09-19T06:46:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.397+0000 7fc352407640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc3380a32c0 0x7fc3380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.398+0000 7fc352407640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc3380a32c0 0x7fc3380a5780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc34c102c60 tx=0x7fc33c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:46:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.401+0000 7fc3437fe640 1 -- 172.21.15.116:0/2591763369 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc34809c000 con 0x7fc34c102e00 2024-09-19T06:46:04.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:04.578+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc34c062630 con 0x7fc3380a32c0 2024-09-19T06:46:04.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:04 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:46:04.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:04 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:46:04.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:04.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:04 smithi116 ceph-mon[35411]: osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] boot 2024-09-19T06:46:04.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:04 smithi116 ceph-mon[35411]: osdmap e46: 7 total, 7 up, 7 in 2024-09-19T06:46:04.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:04 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:46:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:04 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:46:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:04 smithi114 ceph-mon[30820]: osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] boot 2024-09-19T06:46:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:04 smithi114 ceph-mon[30820]: osdmap e46: 7 total, 7 up, 7 in 2024-09-19T06:46:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:04 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:46:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:04 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:46:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:04 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:04 smithi118 ceph-mon[35527]: osd.6 [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] boot 2024-09-19T06:46:04.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:04 smithi118 ceph-mon[35527]: osdmap e46: 7 total, 7 up, 7 in 2024-09-19T06:46:04.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:04 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-19T06:46:05.650 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi114 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.651 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 5696M 2024-09-19T06:46:05.652 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.652 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:05.652 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:05.652 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.652 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:05 smithi116 ceph-mon[35411]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:05.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:05.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi114 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi114 to 5696M 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:05 smithi118 ceph-mon[35527]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi114 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:05.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 5696M 2024-09-19T06:46:05.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:05.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:05.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:05.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:05 smithi114 ceph-mon[30820]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:06 smithi116 ceph-mon[35411]: osdmap e47: 7 total, 7 up, 7 in 2024-09-19T06:46:07.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:06 smithi118 ceph-mon[35527]: osdmap e47: 7 total, 7 up, 7 in 2024-09-19T06:46:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:06 smithi114 ceph-mon[30820]: osdmap e47: 7 total, 7 up, 7 in 2024-09-19T06:46:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:07 smithi116 ceph-mon[35411]: from='client.? 172.21.15.116:0/511343906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]: dispatch 2024-09-19T06:46:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:07 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]: dispatch 2024-09-19T06:46:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:07 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]': finished 2024-09-19T06:46:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:07 smithi116 ceph-mon[35411]: osdmap e48: 8 total, 7 up, 8 in 2024-09-19T06:46:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:07 smithi116 ceph-mon[35411]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:07 smithi116 ceph-mon[35411]: from='client.? 172.21.15.116:0/1031592565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:46:08.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:07 smithi118 ceph-mon[35527]: from='client.? 172.21.15.116:0/511343906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]: dispatch 2024-09-19T06:46:08.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:07 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]: dispatch 2024-09-19T06:46:08.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:07 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]': finished 2024-09-19T06:46:08.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:07 smithi118 ceph-mon[35527]: osdmap e48: 8 total, 7 up, 8 in 2024-09-19T06:46:08.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:08.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:07 smithi118 ceph-mon[35527]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:08.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:07 smithi118 ceph-mon[35527]: from='client.? 172.21.15.116:0/1031592565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:46:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:07 smithi114 ceph-mon[30820]: from='client.? 172.21.15.116:0/511343906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]: dispatch 2024-09-19T06:46:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:07 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]: dispatch 2024-09-19T06:46:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:07 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39e63a0c-eb89-434a-977e-2efd436ee732"}]': finished 2024-09-19T06:46:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:07 smithi114 ceph-mon[30820]: osdmap e48: 8 total, 7 up, 8 in 2024-09-19T06:46:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:07 smithi114 ceph-mon[30820]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:07 smithi114 ceph-mon[30820]: from='client.? 172.21.15.116:0/1031592565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:46:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:10 smithi114 ceph-mon[30820]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:10 smithi116 ceph-mon[35411]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:10.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:10 smithi118 ceph-mon[35527]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:12 smithi114 ceph-mon[30820]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:12 smithi116 ceph-mon[35411]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:12.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:12 smithi118 ceph-mon[35527]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:14 smithi116 ceph-mon[35411]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:14 smithi114 ceph-mon[30820]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:14.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:14 smithi118 ceph-mon[35527]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:15.401 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-19T06:46:15.401 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-19T06:46:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-19T06:46:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:16 smithi116 ceph-mon[35411]: Deploying daemon osd.7 on smithi116 2024-09-19T06:46:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:16 smithi116 ceph-mon[35411]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:16.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:16 smithi118 ceph-mon[35527]: Deploying daemon osd.7 on smithi116 2024-09-19T06:46:16.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:16 smithi118 ceph-mon[35527]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:16 smithi114 ceph-mon[30820]: Deploying daemon osd.7 on smithi116 2024-09-19T06:46:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:16 smithi114 ceph-mon[30820]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:18 smithi114 ceph-mon[30820]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:18.381 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:18 smithi116 ceph-mon[35411]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:18 smithi118 ceph-mon[35527]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:20.248 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:20 smithi116 ceph-mon[35411]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:20 smithi114 ceph-mon[30820]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:20 smithi118 ceph-mon[35527]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:21 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:21 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:21 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:22 smithi114 ceph-mon[30820]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:22 smithi116 ceph-mon[35411]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:22.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:22 smithi118 ceph-mon[35527]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:24.362 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:24 smithi116 ceph-mon[35411]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:24.362 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.362 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.362 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:24.362 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:24.363 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:24 smithi114 ceph-mon[30820]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:24 smithi118 ceph-mon[35527]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:24.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:24.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:24.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.695+0000 7fc3437fe640 1 -- 172.21.15.116:0/2591763369 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc34c062630 con 0x7fc3380a32c0 2024-09-19T06:46:24.697 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 7 on host 'smithi116' 2024-09-19T06:46:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc3380a32c0 msgr2=0x7fc3380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc3380a32c0 0x7fc3380a5780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc34c102c60 tx=0x7fc33c00b390 comp rx=0 tx=0).stop 2024-09-19T06:46:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c102e00 msgr2=0x7fc34c0745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c102e00 0x7fc34c0745e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc34800d770 tx=0x7fc34800dc30 comp rx=0 tx=0).stop 2024-09-19T06:46:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 shutdown_connections 2024-09-19T06:46:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc3380a32c0 0x7fc3380a5780 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc34c1037c0 0x7fc34c074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc34c102e00 0x7fc34c0745e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 --2- 172.21.15.116:0/2591763369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc34c101c00 0x7fc34c075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 >> 172.21.15.116:0/2591763369 conn(0x7fc34c0fd390 msgr2=0x7fc34c106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:46:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 shutdown_connections 2024-09-19T06:46:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:46:24.698+0000 7fc35466b640 1 -- 172.21.15.116:0/2591763369 wait complete. 2024-09-19T06:46:25.350 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:25.350 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:25.350 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:25.385 DEBUG:teuthology.orchestra.run.smithi116:osd.7> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.7.service 2024-09-19T06:46:25.387 INFO:tasks.cephadm:Deploying osd.8 on smithi118 with /dev/vg_nvme/lv_2... 2024-09-19T06:46:25.387 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-19T06:46:25.415 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:25.416 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:25.416 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:25.641 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:46:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:26 smithi114 ceph-mon[30820]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:26 smithi116 ceph-mon[35411]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:26.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:26 smithi118 ceph-mon[35527]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:26.760 INFO:journalctl@ceph.osd.7.smithi116.stdout:Sep 19 06:46:26 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-7[52973]: 2024-09-19T06:46:26.664+0000 7fca27a8a640 -1 osd.7 0 log_to_monitors true 2024-09-19T06:46:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:27 smithi114 ceph-mon[30820]: from='osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T06:46:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:27 smithi114 ceph-mon[30820]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T06:46:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:27 smithi116 ceph-mon[35411]: from='osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T06:46:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:27 smithi116 ceph-mon[35411]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T06:46:27.486 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:27 smithi118 ceph-mon[35527]: from='osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T06:46:27.486 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:27 smithi118 ceph-mon[35527]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-19T06:46:28.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:28 smithi118 ceph-mon[35527]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:28.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:28 smithi118 ceph-mon[35527]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-19T06:46:28.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:28 smithi118 ceph-mon[35527]: osdmap e49: 8 total, 7 up, 8 in 2024-09-19T06:46:28.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:28 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:28.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:28 smithi118 ceph-mon[35527]: from='osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:46:28.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:28 smithi118 ceph-mon[35527]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:46:28.286 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:46:28.319 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi118:vg_nvme/lv_2 2024-09-19T06:46:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:28 smithi114 ceph-mon[30820]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:28 smithi114 ceph-mon[30820]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-19T06:46:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:28 smithi114 ceph-mon[30820]: osdmap e49: 8 total, 7 up, 8 in 2024-09-19T06:46:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:28 smithi114 ceph-mon[30820]: from='osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:46:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:28 smithi114 ceph-mon[30820]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:46:28.428 INFO:journalctl@ceph.osd.7.smithi116.stdout:Sep 19 06:46:28 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-7[52973]: 2024-09-19T06:46:28.148+0000 7fca23a0d640 -1 osd.7 0 waiting for initial osdmap 2024-09-19T06:46:28.428 INFO:journalctl@ceph.osd.7.smithi116.stdout:Sep 19 06:46:28 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-7[52973]: 2024-09-19T06:46:28.156+0000 7fca1f835640 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:46:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:28 smithi116 ceph-mon[35411]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:28 smithi116 ceph-mon[35411]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-19T06:46:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:28 smithi116 ceph-mon[35411]: osdmap e49: 8 total, 7 up, 8 in 2024-09-19T06:46:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:28 smithi116 ceph-mon[35411]: from='osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:46:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:28 smithi116 ceph-mon[35411]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:46:28.603 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:46:29.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 -- 172.21.15.118:0/4093485108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55641004d0 msgr2=0x7f55641008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:29.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 --2- 172.21.15.118:0/4093485108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55641004d0 0x7f55641008d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5558009900 tx=0x7f555802a410 comp rx=0 tx=0).stop 2024-09-19T06:46:29.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 -- 172.21.15.118:0/4093485108 shutdown_connections 2024-09-19T06:46:29.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 --2- 172.21.15.118:0/4093485108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55641017b0 0x7f5564101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:29.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 --2- 172.21.15.118:0/4093485108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5564100e10 0x7f5564101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 --2- 172.21.15.118:0/4093485108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55641004d0 0x7f55641008d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 -- 172.21.15.118:0/4093485108 >> 172.21.15.118:0/4093485108 conn(0x7f55640fc280 msgr2=0x7f55640fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 -- 172.21.15.118:0/4093485108 shutdown_connections 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.079+0000 7f556ab58640 1 -- 172.21.15.118:0/4093485108 wait complete. 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f556ab58640 1 Processor -- start 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f556ab58640 1 -- start start 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f556ab58640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55641004d0 0x7f556419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f556ab58640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5564100e10 0x7f556419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f556ab58640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55641017b0 0x7f55641a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f556ab58640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f556410fa90 con 0x7f55641017b0 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f556ab58640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f556410f910 con 0x7f55641004d0 2024-09-19T06:46:29.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f556ab58640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f556410fc10 con 0x7f5564100e10 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f5563fff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5564100e10 0x7f556419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f5563fff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5564100e10 0x7f556419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43262/0 (socket says 172.21.15.118:43262) 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f5563fff640 1 -- 172.21.15.118:0/2394180701 learned_addr learned my addr 172.21.15.118:0/2394180701 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f55688f4640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55641004d0 0x7f556419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f55690f5640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55641017b0 0x7f55641a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f5563fff640 1 -- 172.21.15.118:0/2394180701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55641004d0 msgr2=0x7f556419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f5563fff640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55641004d0 0x7f556419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f5563fff640 1 -- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55641017b0 msgr2=0x7f55641a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f5563fff640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55641017b0 0x7f55641a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.080+0000 7f5563fff640 1 -- 172.21.15.118:0/2394180701 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55641a1d10 con 0x7f5564100e10 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.081+0000 7f5563fff640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5564100e10 0x7f556419a620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f555000e9f0 tx=0x7f555000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.081+0000 7f5561ffb640 1 -- 172.21.15.118:0/2394180701 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555000cd80 con 0x7f5564100e10 2024-09-19T06:46:29.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.081+0000 7f5561ffb640 1 -- 172.21.15.118:0/2394180701 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5550010910 con 0x7f5564100e10 2024-09-19T06:46:29.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.081+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55641a1fa0 con 0x7f5564100e10 2024-09-19T06:46:29.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.081+0000 7f5561ffb640 1 -- 172.21.15.118:0/2394180701 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550018990 con 0x7f5564100e10 2024-09-19T06:46:29.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.081+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55641a24b0 con 0x7f5564100e10 2024-09-19T06:46:29.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.082+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f556410b220 con 0x7f5564100e10 2024-09-19T06:46:29.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.084+0000 7f5561ffb640 1 -- 172.21.15.118:0/2394180701 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f5550010ac0 con 0x7f5564100e10 2024-09-19T06:46:29.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.085+0000 7f5561ffb640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55540a3270 0x7f55540a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:29.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.085+0000 7f5561ffb640 1 -- 172.21.15.118:0/2394180701 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5117+0+0 (secure 0 0 0) 0x7f55500cf480 con 0x7f5564100e10 2024-09-19T06:46:29.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.085+0000 7f55688f4640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55540a3270 0x7f55540a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:29.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.086+0000 7f55688f4640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55540a3270 0x7f55540a5730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5558006010 tx=0x7f5558000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:46:29.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.088+0000 7f5561ffb640 1 -- 172.21.15.118:0/2394180701 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f555009ce20 con 0x7f5564100e10 2024-09-19T06:46:29.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:29.273+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5564062630 con 0x7f55540a3270 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: osdmap e50: 8 total, 7 up, 8 in 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi116 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 5696M 2024-09-19T06:46:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: osdmap e50: 8 total, 7 up, 8 in 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi116 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 5696M 2024-09-19T06:46:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:46:29.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: osdmap e50: 8 total, 7 up, 8 in 2024-09-19T06:46:29.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:29.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi116 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi116 to 5696M 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:29.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:29 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:46:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] boot 2024-09-19T06:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: osdmap e51: 8 total, 8 up, 8 in 2024-09-19T06:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: osdmap e52: 8 total, 8 up, 8 in 2024-09-19T06:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:46:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] boot 2024-09-19T06:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: osdmap e51: 8 total, 8 up, 8 in 2024-09-19T06:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: osdmap e52: 8 total, 8 up, 8 in 2024-09-19T06:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 192 MiB used, 626 GiB / 626 GiB avail 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: osd.7 [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] boot 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: osdmap e51: 8 total, 8 up, 8 in 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: osdmap e52: 8 total, 8 up, 8 in 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:30.435 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:30 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:31.378 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:31 smithi118 ceph-mon[35527]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:31.378 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:31 smithi118 ceph-mon[35527]: osdmap e53: 8 total, 8 up, 8 in 2024-09-19T06:46:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:31 smithi114 ceph-mon[30820]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:31 smithi114 ceph-mon[30820]: osdmap e53: 8 total, 8 up, 8 in 2024-09-19T06:46:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:31 smithi116 ceph-mon[35411]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:31 smithi116 ceph-mon[35411]: osdmap e53: 8 total, 8 up, 8 in 2024-09-19T06:46:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:32 smithi118 ceph-mon[35527]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 219 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:32.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:32 smithi118 ceph-mon[35527]: osdmap e54: 8 total, 8 up, 8 in 2024-09-19T06:46:32.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:32 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/1062892522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]: dispatch 2024-09-19T06:46:32.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:32 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]: dispatch 2024-09-19T06:46:32.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:32 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]': finished 2024-09-19T06:46:32.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:32 smithi118 ceph-mon[35527]: osdmap e55: 9 total, 8 up, 9 in 2024-09-19T06:46:32.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:32 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:32 smithi114 ceph-mon[30820]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 219 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:32 smithi114 ceph-mon[30820]: osdmap e54: 8 total, 8 up, 8 in 2024-09-19T06:46:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:32 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/1062892522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]: dispatch 2024-09-19T06:46:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:32 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]: dispatch 2024-09-19T06:46:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:32 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]': finished 2024-09-19T06:46:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:32 smithi114 ceph-mon[30820]: osdmap e55: 9 total, 8 up, 9 in 2024-09-19T06:46:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:32 smithi116 ceph-mon[35411]: pgmap v183: 1 pgs: 1 remapped+peering; 577 KiB data, 219 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:32 smithi116 ceph-mon[35411]: osdmap e54: 8 total, 8 up, 8 in 2024-09-19T06:46:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:32 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/1062892522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]: dispatch 2024-09-19T06:46:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:32 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]: dispatch 2024-09-19T06:46:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:32 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c245abe-b78b-46a0-8d12-943863e93026"}]': finished 2024-09-19T06:46:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:32 smithi116 ceph-mon[35411]: osdmap e55: 9 total, 8 up, 9 in 2024-09-19T06:46:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:33 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/3065286249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:46:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:33 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/3065286249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:46:33.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:33 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/3065286249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:46:34.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:34 smithi116 ceph-mon[35411]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 219 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:34.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:34 smithi116 ceph-mon[35411]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-19T06:46:34.657 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:34 smithi118 ceph-mon[35527]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 219 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:34.658 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:34 smithi118 ceph-mon[35527]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-19T06:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:34 smithi114 ceph-mon[30820]: pgmap v186: 1 pgs: 1 remapped+peering; 577 KiB data, 219 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:34 smithi114 ceph-mon[30820]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-19T06:46:35.601 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:35 smithi116 ceph-mon[35411]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:35.602 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:35 smithi116 ceph-mon[35411]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-19T06:46:35.602 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:35 smithi116 ceph-mon[35411]: Cluster is now healthy 2024-09-19T06:46:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:35 smithi114 ceph-mon[30820]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:35 smithi114 ceph-mon[30820]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-19T06:46:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:35 smithi114 ceph-mon[30820]: Cluster is now healthy 2024-09-19T06:46:35.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:35 smithi118 ceph-mon[35527]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:35.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:35 smithi118 ceph-mon[35527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-19T06:46:35.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:35 smithi118 ceph-mon[35527]: Cluster is now healthy 2024-09-19T06:46:38.369 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:38 smithi114 ceph-mon[30820]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:38 smithi116 ceph-mon[35411]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:38.500 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:38 smithi118 ceph-mon[35527]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:40.368 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:40 smithi118 ceph-mon[35527]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:40.368 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-19T06:46:40.368 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:40 smithi114 ceph-mon[30820]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-19T06:46:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:40 smithi116 ceph-mon[35411]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-19T06:46:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:41.347 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:41 smithi118 ceph-mon[35527]: Deploying daemon osd.8 on smithi118 2024-09-19T06:46:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:41 smithi114 ceph-mon[30820]: Deploying daemon osd.8 on smithi118 2024-09-19T06:46:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:41 smithi116 ceph-mon[35411]: Deploying daemon osd.8 on smithi118 2024-09-19T06:46:42.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:42 smithi118 ceph-mon[35527]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:42 smithi114 ceph-mon[30820]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:42 smithi116 ceph-mon[35411]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:44 smithi114 ceph-mon[30820]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:44 smithi116 ceph-mon[35411]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:44.468 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:44 smithi118 ceph-mon[35527]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:46 smithi114 ceph-mon[30820]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:46 smithi116 ceph-mon[35411]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:46.439 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:46 smithi118 ceph-mon[35527]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:46.439 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:46.439 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:46.439 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:46 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:48 smithi114 ceph-mon[30820]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:48 smithi116 ceph-mon[35411]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:48 smithi118 ceph-mon[35527]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:49.872 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 8 on host 'smithi118' 2024-09-19T06:46:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.870+0000 7f5561ffb640 1 -- 172.21.15.118:0/2394180701 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5564062630 con 0x7f55540a3270 2024-09-19T06:46:49.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55540a3270 msgr2=0x7f55540a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:49.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55540a3270 0x7f55540a5730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5558006010 tx=0x7f5558000bc0 comp rx=0 tx=0).stop 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5564100e10 msgr2=0x7f556419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5564100e10 0x7f556419a620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f555000e9f0 tx=0x7f555000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 shutdown_connections 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55540a3270 0x7f55540a5730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55641017b0 0x7f55641a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5564100e10 0x7f556419a620 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 --2- 172.21.15.118:0/2394180701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55641004d0 0x7f556419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 >> 172.21.15.118:0/2394180701 conn(0x7f55640fc280 msgr2=0x7f55640fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 shutdown_connections 2024-09-19T06:46:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:46:49.873+0000 7f556ab58640 1 -- 172.21.15.118:0/2394180701 wait complete. 2024-09-19T06:46:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:49 smithi114 ceph-mon[30820]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:49 smithi116 ceph-mon[35411]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:50.240 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.240 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.240 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:50.240 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:50.240 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.240 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:49 smithi118 ceph-mon[35527]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:50.583 DEBUG:teuthology.orchestra.run.smithi118:osd.8> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.8.service 2024-09-19T06:46:50.585 INFO:tasks.cephadm:Deploying osd.9 on smithi114 with /dev/vg_nvme/lv_1... 2024-09-19T06:46:50.585 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-19T06:46:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:50.954 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:46:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:51.186 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:46:51.186 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:51.186 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:52.115 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:51 smithi118 ceph-mon[35527]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:52.116 INFO:journalctl@ceph.osd.8.smithi118.stdout:Sep 19 06:46:52 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-8[54060]: 2024-09-19T06:46:52.012+0000 7f3d13c0c640 -1 osd.8 0 log_to_monitors true 2024-09-19T06:46:52.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:51 smithi114 ceph-mon[30820]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:51 smithi116 ceph-mon[35411]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:52 smithi114 ceph-mon[30820]: from='osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T06:46:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:52 smithi114 ceph-mon[30820]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T06:46:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:52 smithi116 ceph-mon[35411]: from='osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T06:46:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:52 smithi116 ceph-mon[35411]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T06:46:53.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:52 smithi118 ceph-mon[35527]: from='osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T06:46:53.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:52 smithi118 ceph-mon[35527]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-19T06:46:54.006 INFO:journalctl@ceph.osd.8.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-8[54060]: 2024-09-19T06:46:53.873+0000 7f3d0fb8f640 -1 osd.8 0 waiting for initial osdmap 2024-09-19T06:46:54.006 INFO:journalctl@ceph.osd.8.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-8[54060]: 2024-09-19T06:46:53.883+0000 7f3d0b9b7640 -1 osd.8 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: osdmap e56: 9 total, 8 up, 9 in 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:54.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:54.008 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: osdmap e56: 9 total, 8 up, 9 in 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:54.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:54.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-19T06:46:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: osdmap e56: 9 total, 8 up, 9 in 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:46:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:46:54.278 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:46:54.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-09-19T06:46:54.685 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi118 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi118 to 5696M 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: osdmap e57: 9 total, 8 up, 9 in 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] boot 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: osdmap e58: 9 total, 9 up, 9 in 2024-09-19T06:46:55.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi118 2024-09-19T06:46:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi118 to 5696M 2024-09-19T06:46:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:46:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: osdmap e57: 9 total, 8 up, 9 in 2024-09-19T06:46:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] boot 2024-09-19T06:46:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: osdmap e58: 9 total, 9 up, 9 in 2024-09-19T06:46:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi118 2024-09-19T06:46:55.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi118 to 5696M 2024-09-19T06:46:55.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:46:55.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: osdmap e57: 9 total, 8 up, 9 in 2024-09-19T06:46:55.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: osd.8 [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] boot 2024-09-19T06:46:55.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: osdmap e58: 9 total, 9 up, 9 in 2024-09-19T06:46:55.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:54 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-19T06:46:55.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 -- 172.21.15.114:0/853041816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38104e50 msgr2=0x7f2f381052d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:55.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/853041816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38104e50 0x7f2f381052d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009900 tx=0x7f2f2402a410 comp rx=0 tx=0).stop 2024-09-19T06:46:55.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 -- 172.21.15.114:0/853041816 shutdown_connections 2024-09-19T06:46:55.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/853041816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f38105810 0x7f2f3810c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:55.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/853041816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38104e50 0x7f2f381052d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:55.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/853041816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f38103c50 0x7f2f38104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:55.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 -- 172.21.15.114:0/853041816 >> 172.21.15.114:0/853041816 conn(0x7f2f380ff3e0 msgr2=0x7f2f38101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:46:55.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 -- 172.21.15.114:0/853041816 shutdown_connections 2024-09-19T06:46:55.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.273+0000 7f2f3cc00640 1 -- 172.21.15.114:0/853041816 wait complete. 2024-09-19T06:46:55.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.274+0000 7f2f3cc00640 1 Processor -- start 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f3cc00640 1 -- start start 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f3cc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f38103c50 0x7f2f38199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f3cc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f38104e50 0x7f2f3819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f3cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38105810 0x7f2f381a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f3cc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3810f920 con 0x7f2f38105810 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f3cc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3810f7a0 con 0x7f2f38104e50 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f3cc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3810faa0 con 0x7f2f38103c50 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f37e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38105810 0x7f2f381a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f37e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38105810 0x7f2f381a14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34780/0 (socket says 172.21.15.114:34780) 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f37e00640 1 -- 172.21.15.114:0/4107008451 learned_addr learned my addr 172.21.15.114:0/4107008451 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f37400640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f38103c50 0x7f2f38199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:55.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.275+0000 7f2f36a00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f38104e50 0x7f2f3819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:55.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f37e00640 1 -- 172.21.15.114:0/4107008451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f38103c50 msgr2=0x7f2f38199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:55.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f37e00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f38103c50 0x7f2f38199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:55.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f37e00640 1 -- 172.21.15.114:0/4107008451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f38104e50 msgr2=0x7f2f3819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f37e00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f38104e50 0x7f2f3819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f37e00640 1 -- 172.21.15.114:0/4107008451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f381a1b50 con 0x7f2f38105810 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f37400640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f38103c50 0x7f2f38199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f37e00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38105810 0x7f2f381a14e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000eb10 tx=0x7f2f3000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f2fe00640 1 -- 172.21.15.114:0/4107008451 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30004b20 con 0x7f2f38105810 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f2fe00640 1 -- 172.21.15.114:0/4107008451 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f30004cc0 con 0x7f2f38105810 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.276+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f381a1e40 con 0x7f2f38105810 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.277+0000 7f2f2fe00640 1 -- 172.21.15.114:0/4107008451 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30010760 con 0x7f2f38105810 2024-09-19T06:46:55.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.277+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f381a22f0 con 0x7f2f38105810 2024-09-19T06:46:55.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.278+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f3810b000 con 0x7f2f38105810 2024-09-19T06:46:55.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.281+0000 7f2f2fe00640 1 -- 172.21.15.114:0/4107008451 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f2f30010900 con 0x7f2f38105810 2024-09-19T06:46:55.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.282+0000 7f2f2fe00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2f1c0a32f0 0x7f2f1c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:46:55.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.282+0000 7f2f2fe00640 1 -- 172.21.15.114:0/4107008451 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5858+0+0 (secure 0 0 0) 0x7f2f300cf6c0 con 0x7f2f38105810 2024-09-19T06:46:55.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.283+0000 7f2f37400640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2f1c0a32f0 0x7f2f1c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:46:55.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.283+0000 7f2f37400640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2f1c0a32f0 0x7f2f1c0a57b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2f38104cb0 tx=0x7f2f28005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:46:55.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.285+0000 7f2f2fe00640 1 -- 172.21.15.114:0/4107008451 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2f30054210 con 0x7f2f38105810 2024-09-19T06:46:55.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:46:55.473+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2f38062630 con 0x7f2f1c0a32f0 2024-09-19T06:46:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:55 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:46:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:55 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:46:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:55 smithi114 ceph-mon[30820]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:46:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:55 smithi114 ceph-mon[30820]: osdmap e59: 9 total, 9 up, 9 in 2024-09-19T06:46:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:55 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:46:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:55 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:46:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:55 smithi116 ceph-mon[35411]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:46:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:55 smithi116 ceph-mon[35411]: osdmap e59: 9 total, 9 up, 9 in 2024-09-19T06:46:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:56.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:55 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:46:56.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:55 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:46:56.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:55 smithi118 ceph-mon[35527]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:46:56.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:55 smithi118 ceph-mon[35527]: osdmap e59: 9 total, 9 up, 9 in 2024-09-19T06:46:56.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:55 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:46:56.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:55 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:46:56.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:55 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:46:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:57 smithi118 ceph-mon[35527]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:57 smithi118 ceph-mon[35527]: osdmap e60: 9 total, 9 up, 9 in 2024-09-19T06:46:57.606 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:57 smithi114 ceph-mon[30820]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:57.606 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:57 smithi114 ceph-mon[30820]: osdmap e60: 9 total, 9 up, 9 in 2024-09-19T06:46:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:57 smithi116 ceph-mon[35411]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:46:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:57 smithi116 ceph-mon[35411]: osdmap e60: 9 total, 9 up, 9 in 2024-09-19T06:46:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:58 smithi114 ceph-mon[30820]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:46:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:58 smithi114 ceph-mon[30820]: osdmap e61: 9 total, 9 up, 9 in 2024-09-19T06:46:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:58 smithi116 ceph-mon[35411]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:46:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:58 smithi116 ceph-mon[35411]: osdmap e61: 9 total, 9 up, 9 in 2024-09-19T06:46:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:58 smithi118 ceph-mon[35527]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:46:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:58 smithi118 ceph-mon[35527]: osdmap e61: 9 total, 9 up, 9 in 2024-09-19T06:46:59.384 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:59 smithi114 ceph-mon[30820]: osdmap e62: 9 total, 9 up, 9 in 2024-09-19T06:46:59.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:59 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1354889982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e29bdc9d-0f06-4b54-ba01-d15193a6ab0e"}]: dispatch 2024-09-19T06:46:59.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:59 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1354889982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e29bdc9d-0f06-4b54-ba01-d15193a6ab0e"}]': finished 2024-09-19T06:46:59.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:59 smithi114 ceph-mon[30820]: osdmap e63: 10 total, 9 up, 10 in 2024-09-19T06:46:59.385 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:46:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:46:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:59 smithi116 ceph-mon[35411]: osdmap e62: 9 total, 9 up, 9 in 2024-09-19T06:46:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:59 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1354889982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e29bdc9d-0f06-4b54-ba01-d15193a6ab0e"}]: dispatch 2024-09-19T06:46:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:59 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1354889982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e29bdc9d-0f06-4b54-ba01-d15193a6ab0e"}]': finished 2024-09-19T06:46:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:59 smithi116 ceph-mon[35411]: osdmap e63: 10 total, 9 up, 10 in 2024-09-19T06:46:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:46:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:46:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:59 smithi118 ceph-mon[35527]: osdmap e62: 9 total, 9 up, 9 in 2024-09-19T06:46:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:59 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1354889982' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e29bdc9d-0f06-4b54-ba01-d15193a6ab0e"}]: dispatch 2024-09-19T06:46:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:59 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1354889982' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e29bdc9d-0f06-4b54-ba01-d15193a6ab0e"}]': finished 2024-09-19T06:46:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:59 smithi118 ceph-mon[35527]: osdmap e63: 10 total, 9 up, 10 in 2024-09-19T06:46:59.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:46:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:00 smithi114 ceph-mon[30820]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:00 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2230370050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:00 smithi116 ceph-mon[35411]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:00 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2230370050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:00 smithi118 ceph-mon[35527]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:00 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2230370050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:02 smithi114 ceph-mon[30820]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-19T06:47:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:02 smithi116 ceph-mon[35411]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-19T06:47:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:02 smithi118 ceph-mon[35527]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-19T06:47:04.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:04 smithi114 ceph-mon[30820]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T06:47:04.558 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:04 smithi118 ceph-mon[35527]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T06:47:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:04 smithi116 ceph-mon[35411]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T06:47:06.393 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:06 smithi114 ceph-mon[30820]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-19T06:47:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:06 smithi116 ceph-mon[35411]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-19T06:47:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:06 smithi118 ceph-mon[35527]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-19T06:47:08.573 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:08 smithi114 ceph-mon[30820]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T06:47:08.573 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-19T06:47:08.573 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:08 smithi116 ceph-mon[35411]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T06:47:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-19T06:47:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:08 smithi118 ceph-mon[35527]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T06:47:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-19T06:47:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:09.606 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:09 smithi114 ceph-mon[30820]: Deploying daemon osd.9 on smithi114 2024-09-19T06:47:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:09 smithi116 ceph-mon[35411]: Deploying daemon osd.9 on smithi114 2024-09-19T06:47:09.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:09 smithi118 ceph-mon[35527]: Deploying daemon osd.9 on smithi114 2024-09-19T06:47:10.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:10 smithi114 ceph-mon[30820]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-19T06:47:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:10 smithi116 ceph-mon[35411]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-19T06:47:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:10 smithi118 ceph-mon[35527]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-19T06:47:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:11 smithi114 ceph-mon[30820]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:47:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:11 smithi116 ceph-mon[35411]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:47:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:11 smithi118 ceph-mon[35527]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:14 smithi114 ceph-mon[30820]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:14 smithi116 ceph-mon[35411]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:14.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:14 smithi118 ceph-mon[35527]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:16 smithi114 ceph-mon[30820]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:16 smithi116 ceph-mon[35411]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:16 smithi118 ceph-mon[35527]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:16.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:16.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:18 smithi114 ceph-mon[30820]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:18 smithi116 ceph-mon[35411]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:18 smithi118 ceph-mon[35527]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:20 smithi114 ceph-mon[30820]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:20.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.306+0000 7f2f2fe00640 1 -- 172.21.15.114:0/4107008451 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2f38062630 con 0x7f2f1c0a32f0 2024-09-19T06:47:20.308 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 9 on host 'smithi114' 2024-09-19T06:47:20.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2f1c0a32f0 msgr2=0x7f2f1c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2f1c0a32f0 0x7f2f1c0a57b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2f38104cb0 tx=0x7f2f28005f90 comp rx=0 tx=0).stop 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38105810 msgr2=0x7f2f381a14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38105810 0x7f2f381a14e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000eb10 tx=0x7f2f3000c2d0 comp rx=0 tx=0).stop 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 shutdown_connections 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2f1c0a32f0 0x7f2f1c0a57b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38105810 0x7f2f381a14e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f38104e50 0x7f2f3819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 --2- 172.21.15.114:0/4107008451 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f38103c50 0x7f2f38199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 >> 172.21.15.114:0/4107008451 conn(0x7f2f380ff3e0 msgr2=0x7f2f381081d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 shutdown_connections 2024-09-19T06:47:20.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:47:20.309+0000 7f2f3cc00640 1 -- 172.21.15.114:0/4107008451 wait complete. 2024-09-19T06:47:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:20 smithi116 ceph-mon[35411]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:20 smithi118 ceph-mon[35527]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:21.134 DEBUG:teuthology.orchestra.run.smithi114:osd.9> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.9.service 2024-09-19T06:47:21.138 INFO:tasks.cephadm:Deploying osd.10 on smithi116 with /dev/vg_nvme/lv_1... 2024-09-19T06:47:21.139 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-19T06:47:21.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:21.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:21.346 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:21.391 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T06:47:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:21 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:21 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:21.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:21 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:21.917 INFO:journalctl@ceph.osd.9.smithi114.stdout:Sep 19 06:47:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-9[64528]: 2024-09-19T06:47:21.796+0000 7f8bbaa5f640 -1 osd.9 0 log_to_monitors true 2024-09-19T06:47:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:22 smithi114 ceph-mon[30820]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:22 smithi114 ceph-mon[30820]: from='osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T06:47:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:22 smithi114 ceph-mon[30820]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T06:47:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:22 smithi116 ceph-mon[35411]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:22 smithi116 ceph-mon[35411]: from='osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T06:47:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:22 smithi116 ceph-mon[35411]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T06:47:22.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:22 smithi118 ceph-mon[35527]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:22.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:22 smithi118 ceph-mon[35527]: from='osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T06:47:22.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:22 smithi118 ceph-mon[35527]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-19T06:47:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:23 smithi114 ceph-mon[30820]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-19T06:47:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:23 smithi114 ceph-mon[30820]: osdmap e64: 10 total, 9 up, 10 in 2024-09-19T06:47:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:23 smithi114 ceph-mon[30820]: from='osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:47:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:23 smithi114 ceph-mon[30820]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:47:23.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:23 smithi114 ceph-mon[30820]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:23.610 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:23 smithi116 ceph-mon[35411]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-19T06:47:23.610 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:23 smithi116 ceph-mon[35411]: osdmap e64: 10 total, 9 up, 10 in 2024-09-19T06:47:23.610 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:23.610 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:23 smithi116 ceph-mon[35411]: from='osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:47:23.610 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:23 smithi116 ceph-mon[35411]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:47:23.610 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:23 smithi116 ceph-mon[35411]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:23 smithi118 ceph-mon[35527]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-19T06:47:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:23 smithi118 ceph-mon[35527]: osdmap e64: 10 total, 9 up, 10 in 2024-09-19T06:47:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:23 smithi118 ceph-mon[35527]: from='osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:47:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:23 smithi118 ceph-mon[35527]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-09-19T06:47:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:23 smithi118 ceph-mon[35527]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:23.922 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-09-19T06:47:23.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-09-19T06:47:24.166 INFO:journalctl@ceph.osd.9.smithi114.stdout:Sep 19 06:47:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-9[64528]: 2024-09-19T06:47:23.865+0000 7f8bb5600640 -1 osd.9 0 waiting for initial osdmap 2024-09-19T06:47:24.166 INFO:journalctl@ceph.osd.9.smithi114.stdout:Sep 19 06:47:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-9[64528]: 2024-09-19T06:47:23.900+0000 7f8bab600640 -1 osd.9 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:47:24.192 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.b/config 2024-09-19T06:47:24.639 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:24 smithi116 ceph-mon[35411]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:47:24.639 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:24 smithi116 ceph-mon[35411]: osdmap e65: 10 total, 9 up, 10 in 2024-09-19T06:47:24.639 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:24.639 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:24.639 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:24 smithi116 ceph-mon[35411]: from='osd.9 ' entity='osd.9' 2024-09-19T06:47:24.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:24 smithi114 ceph-mon[30820]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:47:24.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:24 smithi114 ceph-mon[30820]: osdmap e65: 10 total, 9 up, 10 in 2024-09-19T06:47:24.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:24.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:24.660 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:24 smithi114 ceph-mon[30820]: from='osd.9 ' entity='osd.9' 2024-09-19T06:47:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.637+0000 7f576a8d1640 1 -- 172.21.15.116:0/942895614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641017a0 msgr2=0x7f5764109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.637+0000 7f576a8d1640 1 --2- 172.21.15.116:0/942895614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641017a0 0x7f5764109040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f575c009ca0 tx=0x7f575c02a8c0 comp rx=0 tx=0).stop 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.637+0000 7f576a8d1640 1 -- 172.21.15.116:0/942895614 shutdown_connections 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.637+0000 7f576a8d1640 1 --2- 172.21.15.116:0/942895614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641017a0 0x7f5764109040 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.637+0000 7f576a8d1640 1 --2- 172.21.15.116:0/942895614 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5764100de0 0x7f5764101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.637+0000 7f576a8d1640 1 --2- 172.21.15.116:0/942895614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57641020a0 0x7f57641024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.637+0000 7f576a8d1640 1 -- 172.21.15.116:0/942895614 >> 172.21.15.116:0/942895614 conn(0x7f57640fba40 msgr2=0x7f57640fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.638+0000 7f576a8d1640 1 -- 172.21.15.116:0/942895614 shutdown_connections 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.638+0000 7f576a8d1640 1 -- 172.21.15.116:0/942895614 wait complete. 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.638+0000 7f576a8d1640 1 Processor -- start 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.638+0000 7f576a8d1640 1 -- start start 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f576a8d1640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5764100de0 0x7f5764197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f576a8d1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57641017a0 0x7f57641984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:47:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f576a8d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641020a0 0x7f576419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f576a8d1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f576410c780 con 0x7f57641017a0 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f576a8d1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f576410c600 con 0x7f57641020a0 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f576a8d1640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f576410c900 con 0x7f5764100de0 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641020a0 0x7f576419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641020a0 0x7f576419f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46366/0 (socket says 172.21.15.116:46366) 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 -- 172.21.15.116:0/4082570724 learned_addr learned my addr 172.21.15.116:0/4082570724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5763fff640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5764100de0 0x7f5764197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f57637fe640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57641017a0 0x7f57641984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 -- 172.21.15.116:0/4082570724 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5764100de0 msgr2=0x7f5764197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5764100de0 0x7f5764197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 -- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57641017a0 msgr2=0x7f57641984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57641017a0 0x7f57641984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 -- 172.21.15.116:0/4082570724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576419fbc0 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f57637fe640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57641017a0 0x7f57641984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f5768e6e640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641020a0 0x7f576419f550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f575c031ee0 tx=0x7f575c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.639+0000 7f57617fa640 1 -- 172.21.15.116:0/4082570724 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c003c30 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.640+0000 7f57617fa640 1 -- 172.21.15.116:0/4082570724 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575c003dd0 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.640+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576419fe50 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.640+0000 7f57617fa640 1 -- 172.21.15.116:0/4082570724 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575c02fb70 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.640+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57641a0300 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.641+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57641a0560 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.642+0000 7f57617fa640 1 -- 172.21.15.116:0/4082570724 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f575c02fd30 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.642+0000 7f57617fa640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f574c0a3270 0x7f574c0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.642+0000 7f57617fa640 1 -- 172.21.15.116:0/4082570724 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5981+0+0 (secure 0 0 0) 0x7f575c0f0270 con 0x7f57641020a0 2024-09-19T06:47:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.642+0000 7f5763fff640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f574c0a3270 0x7f574c0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:47:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.643+0000 7f5763fff640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f574c0a3270 0x7f574c0a5730 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f576406a510 tx=0x7f575400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:47:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.646+0000 7f57617fa640 1 -- 172.21.15.116:0/4082570724 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f575c0bd8b0 con 0x7f57641020a0 2024-09-19T06:47:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:24 smithi118 ceph-mon[35527]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-09-19T06:47:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:24 smithi118 ceph-mon[35527]: osdmap e65: 10 total, 9 up, 10 in 2024-09-19T06:47:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:24 smithi118 ceph-mon[35527]: from='osd.9 ' entity='osd.9' 2024-09-19T06:47:24.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:24.823+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f57641a0730 con 0x7f574c0a3270 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] boot 2024-09-19T06:47:25.501 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: osdmap e66: 10 total, 10 up, 10 in 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi114 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 4272M 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:25.502 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] boot 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: osdmap e66: 10 total, 10 up, 10 in 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi114 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 4272M 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:25.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:47:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:47:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: osd.9 [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] boot 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: osdmap e66: 10 total, 10 up, 10 in 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi114 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 804 GiB / 805 GiB avail 2024-09-19T06:47:25.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:25.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi114 to 4272M 2024-09-19T06:47:25.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:25.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:25.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:25.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:25 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:27.577 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:27 smithi116 ceph-mon[35411]: osdmap e67: 10 total, 10 up, 10 in 2024-09-19T06:47:27.578 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:27 smithi116 ceph-mon[35411]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:27.578 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:27 smithi116 ceph-mon[35411]: from='client.? 172.21.15.116:0/2400260567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]: dispatch 2024-09-19T06:47:27.578 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:27 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]: dispatch 2024-09-19T06:47:27.578 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:27 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]': finished 2024-09-19T06:47:27.578 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:27 smithi116 ceph-mon[35411]: osdmap e68: 11 total, 10 up, 11 in 2024-09-19T06:47:27.578 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:27 smithi114 ceph-mon[30820]: osdmap e67: 10 total, 10 up, 10 in 2024-09-19T06:47:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:27 smithi114 ceph-mon[30820]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:27 smithi114 ceph-mon[30820]: from='client.? 172.21.15.116:0/2400260567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]: dispatch 2024-09-19T06:47:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:27 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]: dispatch 2024-09-19T06:47:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:27 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]': finished 2024-09-19T06:47:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:27 smithi114 ceph-mon[30820]: osdmap e68: 11 total, 10 up, 11 in 2024-09-19T06:47:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:27.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:27 smithi118 ceph-mon[35527]: osdmap e67: 10 total, 10 up, 10 in 2024-09-19T06:47:27.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:27 smithi118 ceph-mon[35527]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:27.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:27 smithi118 ceph-mon[35527]: from='client.? 172.21.15.116:0/2400260567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]: dispatch 2024-09-19T06:47:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:27 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]: dispatch 2024-09-19T06:47:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:27 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12da1b00-07fc-49c3-96ec-450fd6568732"}]': finished 2024-09-19T06:47:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:27 smithi118 ceph-mon[35527]: osdmap e68: 11 total, 10 up, 11 in 2024-09-19T06:47:27.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:28 smithi114 ceph-mon[30820]: from='client.? 172.21.15.116:0/3059122591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:28 smithi116 ceph-mon[35411]: from='client.? 172.21.15.116:0/3059122591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:28.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:28 smithi118 ceph-mon[35527]: from='client.? 172.21.15.116:0/3059122591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:29 smithi114 ceph-mon[30820]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:29 smithi116 ceph-mon[35411]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:29.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:29 smithi118 ceph-mon[35527]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:32.395 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:32 smithi116 ceph-mon[35411]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:32 smithi114 ceph-mon[30820]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:32 smithi118 ceph-mon[35527]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:34 smithi114 ceph-mon[30820]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:34 smithi116 ceph-mon[35411]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:34 smithi118 ceph-mon[35527]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:36 smithi114 ceph-mon[30820]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-19T06:47:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:36 smithi116 ceph-mon[35411]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-19T06:47:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:36 smithi118 ceph-mon[35527]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:36 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-19T06:47:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:36 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:37.389 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:37 smithi116 ceph-mon[35411]: Deploying daemon osd.10 on smithi116 2024-09-19T06:47:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:37 smithi114 ceph-mon[30820]: Deploying daemon osd.10 on smithi116 2024-09-19T06:47:37.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:37 smithi118 ceph-mon[35527]: Deploying daemon osd.10 on smithi116 2024-09-19T06:47:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:38 smithi114 ceph-mon[30820]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:38 smithi116 ceph-mon[35411]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:38.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:38 smithi118 ceph-mon[35527]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:40 smithi114 ceph-mon[30820]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:40.426 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:40 smithi116 ceph-mon[35411]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:40.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:40 smithi118 ceph-mon[35527]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:42.410 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:42 smithi116 ceph-mon[35411]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:42.410 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:42.410 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:42.410 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:42 smithi114 ceph-mon[30820]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:42 smithi118 ceph-mon[35527]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:42.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:42 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:44 smithi114 ceph-mon[30820]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:44 smithi116 ceph-mon[35411]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:44.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:44 smithi118 ceph-mon[35527]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:44.892 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 10 on host 'smithi116' 2024-09-19T06:47:44.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.890+0000 7f57617fa640 1 -- 172.21.15.116:0/4082570724 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f57641a0730 con 0x7f574c0a3270 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.892+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f574c0a3270 msgr2=0x7f574c0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.892+0000 7f576a8d1640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f574c0a3270 0x7f574c0a5730 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f576406a510 tx=0x7f575400b390 comp rx=0 tx=0).stop 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.892+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641020a0 msgr2=0x7f576419f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.892+0000 7f576a8d1640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641020a0 0x7f576419f550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f575c031ee0 tx=0x7f575c004ab0 comp rx=0 tx=0).stop 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.893+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 shutdown_connections 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.893+0000 7f576a8d1640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f574c0a3270 0x7f574c0a5730 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.893+0000 7f576a8d1640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57641020a0 0x7f576419f550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.893+0000 7f576a8d1640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57641017a0 0x7f57641984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:44.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.893+0000 7f576a8d1640 1 --2- 172.21.15.116:0/4082570724 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5764100de0 0x7f5764197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.893+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 >> 172.21.15.116:0/4082570724 conn(0x7f57640fba40 msgr2=0x7f57640ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:47:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.893+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 shutdown_connections 2024-09-19T06:47:44.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-09-19T06:47:44.893+0000 7f576a8d1640 1 -- 172.21.15.116:0/4082570724 wait complete. 2024-09-19T06:47:45.538 DEBUG:teuthology.orchestra.run.smithi116:osd.10> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.10.service 2024-09-19T06:47:45.540 INFO:tasks.cephadm:Deploying osd.11 on smithi118 with /dev/vg_nvme/lv_1... 2024-09-19T06:47:45.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-19T06:47:45.796 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:47:46.163 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.164 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.164 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:45 smithi118 ceph-mon[35527]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:46.164 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.164 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:45 smithi114 ceph-mon[30820]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:45 smithi116 ceph-mon[35411]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:47.929 INFO:journalctl@ceph.osd.10.smithi116.stdout:Sep 19 06:47:47 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-10[59958]: 2024-09-19T06:47:47.480+0000 7f9694698640 -1 osd.10 0 log_to_monitors true 2024-09-19T06:47:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:48 smithi114 ceph-mon[30820]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:48 smithi114 ceph-mon[30820]: from='osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T06:47:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:48 smithi114 ceph-mon[30820]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T06:47:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:48 smithi116 ceph-mon[35411]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:48 smithi116 ceph-mon[35411]: from='osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T06:47:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:48 smithi116 ceph-mon[35411]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T06:47:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:48 smithi118 ceph-mon[35527]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:48 smithi118 ceph-mon[35527]: from='osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T06:47:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:48 smithi118 ceph-mon[35527]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-19T06:47:48.588 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-19T06:47:48.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi118:vg_nvme/lv_1 2024-09-19T06:47:48.860 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:47:49.254 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: osdmap e69: 11 total, 10 up, 11 in 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.255 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:49.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:49.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:49.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:47:49.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: osdmap e70: 11 total, 10 up, 11 in 2024-09-19T06:47:49.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:49.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 -- 172.21.15.118:0/2576848287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac101d10 msgr2=0x7f88ac102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:49.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 --2- 172.21.15.118:0/2576848287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac101d10 0x7f88ac102190 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f889c009900 tx=0x7f889c02a410 comp rx=0 tx=0).stop 2024-09-19T06:47:49.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 -- 172.21.15.118:0/2576848287 shutdown_connections 2024-09-19T06:47:49.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 --2- 172.21.15.118:0/2576848287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88ac1026d0 0x7f88ac108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:49.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 --2- 172.21.15.118:0/2576848287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac101d10 0x7f88ac102190 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 --2- 172.21.15.118:0/2576848287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88ac100b10 0x7f88ac100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 -- 172.21.15.118:0/2576848287 >> 172.21.15.118:0/2576848287 conn(0x7f88ac0fc2a0 msgr2=0x7f88ac0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 -- 172.21.15.118:0/2576848287 shutdown_connections 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.338+0000 7f88b41ce640 1 -- 172.21.15.118:0/2576848287 wait complete. 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b41ce640 1 Processor -- start 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b41ce640 1 -- start start 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b41ce640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac100b10 0x7f88ac197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b41ce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88ac101d10 0x7f88ac198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b41ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88ac1026d0 0x7f88ac19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b41ce640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10c790 con 0x7f88ac101d10 2024-09-19T06:47:49.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b41ce640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10c610 con 0x7f88ac1026d0 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b41ce640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac10c910 con 0x7f88ac100b10 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b1f6a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac100b10 0x7f88ac197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b1f6a640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac100b10 0x7f88ac197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60154/0 (socket says 172.21.15.118:60154) 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b1f6a640 1 -- 172.21.15.118:0/1929873012 learned_addr learned my addr 172.21.15.118:0/1929873012 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b1769640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88ac101d10 0x7f88ac198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.339+0000 7f88b276b640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88ac1026d0 0x7f88ac19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88b1f6a640 1 -- 172.21.15.118:0/1929873012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88ac1026d0 msgr2=0x7f88ac19f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88b1f6a640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88ac1026d0 0x7f88ac19f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:49.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88b1f6a640 1 -- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88ac101d10 msgr2=0x7f88ac198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88b1f6a640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88ac101d10 0x7f88ac198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88b1f6a640 1 -- 172.21.15.118:0/1929873012 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ac19fb60 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88b1f6a640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac100b10 0x7f88ac197f30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c970 tx=0x7f88a800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88a2ffd640 1 -- 172.21.15.118:0/1929873012 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8004220 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88a2ffd640 1 -- 172.21.15.118:0/1929873012 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88a80043c0 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ac19fe50 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88a2ffd640 1 -- 172.21.15.118:0/1929873012 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8004a00 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.340+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ac1a0250 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.341+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88ac1a04e0 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.342+0000 7f88a2ffd640 1 -- 172.21.15.118:0/1929873012 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f88a80079b0 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.343+0000 7f88a2ffd640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88980a3270 0x7f88980a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.343+0000 7f88a2ffd640 1 -- 172.21.15.118:0/1929873012 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6727+0+0 (secure 0 0 0) 0x7f88a80cf370 con 0x7f88ac100b10 2024-09-19T06:47:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.343+0000 7f88b1769640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88980a3270 0x7f88980a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:47:49.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.344+0000 7f88b1769640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88980a3270 0x7f88980a5730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f889c0314e0 tx=0x7f889c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:47:49.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.346+0000 7f88a2ffd640 1 -- 172.21.15.118:0/1929873012 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f88a8014030 con 0x7f88ac100b10 2024-09-19T06:47:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-19T06:47:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: osdmap e69: 11 total, 10 up, 11 in 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:47:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: osdmap e70: 11 total, 10 up, 11 in 2024-09-19T06:47:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:49.428 INFO:journalctl@ceph.osd.10.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-10[59958]: 2024-09-19T06:47:49.127+0000 7f969061b640 -1 osd.10 0 waiting for initial osdmap 2024-09-19T06:47:49.428 INFO:journalctl@ceph.osd.10.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-10[59958]: 2024-09-19T06:47:49.143+0000 7f968c443640 -1 osd.10 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: osdmap e69: 11 total, 10 up, 11 in 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: osdmap e70: 11 total, 10 up, 11 in 2024-09-19T06:47:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:49.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:47:49.522+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f88ac1a06b0 con 0x7f88980a3270 2024-09-19T06:47:50.200 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi116 2024-09-19T06:47:50.200 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi116 to 4272M 2024-09-19T06:47:50.200 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:50.200 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:50.200 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] boot 2024-09-19T06:47:50.200 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: osdmap e71: 11 total, 11 up, 11 in 2024-09-19T06:47:50.201 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:50.201 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:47:50.201 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:47:50.201 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:50.201 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:50.201 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:50.201 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:50 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi116 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 4272M 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] boot 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: osdmap e71: 11 total, 11 up, 11 in 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi116 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 4272M 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 894 GiB / 894 GiB avail 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: osd.10 [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] boot 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: osdmap e71: 11 total, 11 up, 11 in 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:47:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:47:51.410 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:51 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:47:51.410 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:51 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:47:51.410 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:51 smithi118 ceph-mon[35527]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:47:51.410 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:51 smithi118 ceph-mon[35527]: osdmap e72: 11 total, 11 up, 11 in 2024-09-19T06:47:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:51 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:47:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:51 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:47:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:51 smithi114 ceph-mon[30820]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:47:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:51 smithi114 ceph-mon[30820]: osdmap e72: 11 total, 11 up, 11 in 2024-09-19T06:47:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:51 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:47:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:51 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:47:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:51 smithi116 ceph-mon[35411]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:47:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:51 smithi116 ceph-mon[35411]: osdmap e72: 11 total, 11 up, 11 in 2024-09-19T06:47:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:52 smithi114 ceph-mon[30820]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:47:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:52 smithi114 ceph-mon[30820]: osdmap e73: 11 total, 11 up, 11 in 2024-09-19T06:47:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:52 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/3816316326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]: dispatch 2024-09-19T06:47:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:52 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]: dispatch 2024-09-19T06:47:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:52 smithi114 ceph-mon[30820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]': finished 2024-09-19T06:47:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:52 smithi114 ceph-mon[30820]: osdmap e74: 12 total, 11 up, 12 in 2024-09-19T06:47:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:47:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:52 smithi116 ceph-mon[35411]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:47:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:52 smithi116 ceph-mon[35411]: osdmap e73: 11 total, 11 up, 11 in 2024-09-19T06:47:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:52 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/3816316326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]: dispatch 2024-09-19T06:47:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:52 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]: dispatch 2024-09-19T06:47:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:52 smithi116 ceph-mon[35411]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]': finished 2024-09-19T06:47:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:52 smithi116 ceph-mon[35411]: osdmap e74: 12 total, 11 up, 12 in 2024-09-19T06:47:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:47:52.715 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:52 smithi118 ceph-mon[35527]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:47:52.715 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:52 smithi118 ceph-mon[35527]: osdmap e73: 11 total, 11 up, 11 in 2024-09-19T06:47:52.715 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:52 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/3816316326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]: dispatch 2024-09-19T06:47:52.715 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:52 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]: dispatch 2024-09-19T06:47:52.715 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:52 smithi118 ceph-mon[35527]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71"}]': finished 2024-09-19T06:47:52.715 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:52 smithi118 ceph-mon[35527]: osdmap e74: 12 total, 11 up, 12 in 2024-09-19T06:47:52.715 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:47:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:53 smithi114 ceph-mon[30820]: from='client.? 172.21.15.118:0/3433889040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:53 smithi114 ceph-mon[30820]: osdmap e75: 12 total, 11 up, 12 in 2024-09-19T06:47:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:47:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:53 smithi116 ceph-mon[35411]: from='client.? 172.21.15.118:0/3433889040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:53 smithi116 ceph-mon[35411]: osdmap e75: 12 total, 11 up, 12 in 2024-09-19T06:47:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:47:53.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:53 smithi118 ceph-mon[35527]: from='client.? 172.21.15.118:0/3433889040' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-19T06:47:53.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:53 smithi118 ceph-mon[35527]: osdmap e75: 12 total, 11 up, 12 in 2024-09-19T06:47:53.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:47:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:54 smithi114 ceph-mon[30820]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:47:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:54 smithi116 ceph-mon[35411]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:47:54.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:54 smithi118 ceph-mon[35527]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 306 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:47:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:56 smithi114 ceph-mon[30820]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-19T06:47:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:56 smithi116 ceph-mon[35411]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-19T06:47:56.697 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:56 smithi118 ceph-mon[35527]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-09-19T06:47:58.559 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:47:58 smithi118 ceph-mon[35527]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T06:47:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:47:58 smithi114 ceph-mon[30820]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T06:47:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:47:58 smithi116 ceph-mon[35411]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-19T06:48:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:00 smithi114 ceph-mon[30820]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-19T06:48:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:00 smithi116 ceph-mon[35411]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-19T06:48:00.679 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:00 smithi118 ceph-mon[35527]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-19T06:48:01.407 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:01 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-19T06:48:01.408 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:01 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-19T06:48:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-19T06:48:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:02.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:02 smithi118 ceph-mon[35527]: Deploying daemon osd.11 on smithi118 2024-09-19T06:48:02.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:02 smithi118 ceph-mon[35527]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T06:48:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:02 smithi114 ceph-mon[30820]: Deploying daemon osd.11 on smithi118 2024-09-19T06:48:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:02 smithi114 ceph-mon[30820]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T06:48:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:02 smithi116 ceph-mon[35411]: Deploying daemon osd.11 on smithi118 2024-09-19T06:48:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:02 smithi116 ceph-mon[35411]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-19T06:48:03.445 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:03 smithi118 ceph-mon[35527]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T06:48:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:03 smithi114 ceph-mon[30820]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T06:48:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:03 smithi116 ceph-mon[35411]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-19T06:48:06.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:06 smithi118 ceph-mon[35527]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:48:06.382 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:06 smithi116 ceph-mon[35411]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:48:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:06 smithi114 ceph-mon[30820]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-19T06:48:07.352 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:07.353 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:07.353 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:08.387 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:08 smithi118 ceph-mon[35527]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:08 smithi114 ceph-mon[30820]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:08 smithi116 ceph-mon[35411]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:10.235 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:10 smithi118 ceph-mon[35527]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:10.235 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:10 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.235 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:10 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.236 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:10 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:10.236 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:10 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:10.236 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:10 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.276+0000 7f88a2ffd640 1 -- 172.21.15.118:0/1929873012 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f88ac1a06b0 con 0x7f88980a3270 2024-09-19T06:48:10.279 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 11 on host 'smithi118' 2024-09-19T06:48:10.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88980a3270 msgr2=0x7f88980a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:10.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88980a3270 0x7f88980a5730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f889c0314e0 tx=0x7f889c000bc0 comp rx=0 tx=0).stop 2024-09-19T06:48:10.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac100b10 msgr2=0x7f88ac197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:10.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac100b10 0x7f88ac197f30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f88a800c970 tx=0x7f88a800ce30 comp rx=0 tx=0).stop 2024-09-19T06:48:10.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 shutdown_connections 2024-09-19T06:48:10.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88980a3270 0x7f88980a5730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:10.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88ac1026d0 0x7f88ac19f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:10.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88ac101d10 0x7f88ac198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:10.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 --2- 172.21.15.118:0/1929873012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f88ac100b10 0x7f88ac197f30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:10.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 >> 172.21.15.118:0/1929873012 conn(0x7f88ac0fc2a0 msgr2=0x7f88ac06a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:10.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 shutdown_connections 2024-09-19T06:48:10.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:10.278+0000 7f88b41ce640 1 -- 172.21.15.118:0/1929873012 wait complete. 2024-09-19T06:48:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:10 smithi114 ceph-mon[30820]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:10 smithi116 ceph-mon[35411]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:10.989 DEBUG:teuthology.orchestra.run.smithi118:osd.11> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@osd.11.service 2024-09-19T06:48:10.991 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-19T06:48:10.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-19T06:48:11.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:11.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:11.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:11.377 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.928+0000 7f9348800640 1 -- 172.21.15.114:0/2455467767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340076220 msgr2=0x7f9340074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.928+0000 7f9348800640 1 --2- 172.21.15.114:0/2455467767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340076220 0x7f9340074680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9334009900 tx=0x7f933402a410 comp rx=0 tx=0).stop 2024-09-19T06:48:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.928+0000 7f9348800640 1 -- 172.21.15.114:0/2455467767 shutdown_connections 2024-09-19T06:48:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.928+0000 7f9348800640 1 --2- 172.21.15.114:0/2455467767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934010e1e0 0x7f93401105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.928+0000 7f9348800640 1 --2- 172.21.15.114:0/2455467767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9340074c50 0x7f93400750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.928+0000 7f9348800640 1 --2- 172.21.15.114:0/2455467767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340076220 0x7f9340074680 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.928+0000 7f9348800640 1 -- 172.21.15.114:0/2455467767 >> 172.21.15.114:0/2455467767 conn(0x7f93400fd390 msgr2=0x7f93400ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.929+0000 7f9348800640 1 -- 172.21.15.114:0/2455467767 shutdown_connections 2024-09-19T06:48:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.929+0000 7f9348800640 1 -- 172.21.15.114:0/2455467767 wait complete. 2024-09-19T06:48:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.929+0000 7f9348800640 1 Processor -- start 2024-09-19T06:48:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9348800640 1 -- start start 2024-09-19T06:48:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9348800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340074c50 0x7f934019e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9348800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9340076220 0x7f934019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9348800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934010e1e0 0x7f93401a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9348800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9340113ef0 con 0x7f9340074c50 2024-09-19T06:48:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9348800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9340113d70 con 0x7f934010e1e0 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9348800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9340114070 con 0x7f9340076220 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9345e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340074c50 0x7f934019e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9345e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340074c50 0x7f934019e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49508/0 (socket says 172.21.15.114:49508) 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9345e00640 1 -- 172.21.15.114:0/1481595884 learned_addr learned my addr 172.21.15.114:0/1481595884 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9345400640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9340076220 0x7f934019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.930+0000 7f9346800640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934010e1e0 0x7f93401a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f9345e00640 1 -- 172.21.15.114:0/1481595884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9340076220 msgr2=0x7f934019ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f9345e00640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9340076220 0x7f934019ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f9345e00640 1 -- 172.21.15.114:0/1481595884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934010e1e0 msgr2=0x7f93401a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f9345e00640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934010e1e0 0x7f93401a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f9345e00640 1 -- 172.21.15.114:0/1481595884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93401a60f0 con 0x7f9340074c50 2024-09-19T06:48:11.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f9346800640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934010e1e0 0x7f93401a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:48:11.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f9345e00640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340074c50 0x7f934019e4c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9334031440 tx=0x7f9334032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:11.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f933aa00640 1 -- 172.21.15.114:0/1481595884 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933403f030 con 0x7f9340074c50 2024-09-19T06:48:11.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f933aa00640 1 -- 172.21.15.114:0/1481595884 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9334003ce0 con 0x7f9340074c50 2024-09-19T06:48:11.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93401a6380 con 0x7f9340074c50 2024-09-19T06:48:11.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.931+0000 7f933aa00640 1 -- 172.21.15.114:0/1481595884 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9334035070 con 0x7f9340074c50 2024-09-19T06:48:11.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.932+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93401a68b0 con 0x7f9340074c50 2024-09-19T06:48:11.935 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.933+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9340104290 con 0x7f9340074c50 2024-09-19T06:48:11.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.934+0000 7f933aa00640 1 -- 172.21.15.114:0/1481595884 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f9334042030 con 0x7f9340074c50 2024-09-19T06:48:11.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.936+0000 7f933aa00640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f93280a32f0 0x7f93280a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:11.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.936+0000 7f933aa00640 1 -- 172.21.15.114:0/1481595884 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6809+0+0 (secure 0 0 0) 0x7f93340ef4f0 con 0x7f9340074c50 2024-09-19T06:48:11.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.936+0000 7f9345400640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f93280a32f0 0x7f93280a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:11.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.937+0000 7f9345400640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f93280a32f0 0x7f93280a57b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f934019f9e0 tx=0x7f933000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:11.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:11.940+0000 7f933aa00640 1 -- 172.21.15.114:0/1481595884 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f93340bc7f0 con 0x7f9340074c50 2024-09-19T06:48:12.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.119+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f9340062df0 con 0x7f9340074c50 2024-09-19T06:48:12.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.119+0000 7f933aa00640 1 -- 172.21.15.114:0/1481595884 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) ==== 74+0+133 (secure 0 0 0) 0x7f93340bc0f0 con 0x7f9340074c50 2024-09-19T06:48:12.121 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:12.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.122+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f93280a32f0 msgr2=0x7f93280a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:12.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.122+0000 7f9348800640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f93280a32f0 0x7f93280a57b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f934019f9e0 tx=0x7f933000b410 comp rx=0 tx=0).stop 2024-09-19T06:48:12.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.122+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340074c50 msgr2=0x7f934019e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:12.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.122+0000 7f9348800640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340074c50 0x7f934019e4c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9334031440 tx=0x7f9334032ce0 comp rx=0 tx=0).stop 2024-09-19T06:48:12.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.122+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 shutdown_connections 2024-09-19T06:48:12.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.123+0000 7f9348800640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f93280a32f0 0x7f93280a57b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:12.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.123+0000 7f9348800640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f934010e1e0 0x7f93401a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:12.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.123+0000 7f9348800640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9340076220 0x7f934019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:12.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.123+0000 7f9348800640 1 --2- 172.21.15.114:0/1481595884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9340074c50 0x7f934019e4c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:12.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.123+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 >> 172.21.15.114:0/1481595884 conn(0x7f93400fd390 msgr2=0x7f9340101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:12.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.123+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 shutdown_connections 2024-09-19T06:48:12.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:12.123+0000 7f9348800640 1 -- 172.21.15.114:0/1481595884 wait complete. 2024-09-19T06:48:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:12 smithi114 ceph-mon[30820]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:12 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1481595884' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T06:48:12.474 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:12 smithi118 ceph-mon[35527]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:12.474 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:12 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1481595884' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T06:48:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:12 smithi116 ceph-mon[35411]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:12 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1481595884' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T06:48:12.756 INFO:journalctl@ceph.osd.11.smithi118.stdout:Sep 19 06:48:12 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-11[60612]: 2024-09-19T06:48:12.685+0000 7fe82b797640 -1 osd.11 0 log_to_monitors true 2024-09-19T06:48:13.016 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1726728469,"num_in_osds":12,"osd_in_since":1726728472,"num_remapped_pgs":0} 2024-09-19T06:48:13.486 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:13 smithi118 ceph-mon[35527]: from='osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T06:48:13.487 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:13 smithi118 ceph-mon[35527]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T06:48:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:13 smithi114 ceph-mon[30820]: from='osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T06:48:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:13 smithi114 ceph-mon[30820]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T06:48:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:13 smithi116 ceph-mon[35411]: from='osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T06:48:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:13 smithi116 ceph-mon[35411]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-19T06:48:14.017 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-19T06:48:14.398 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: osdmap e76: 12 total, 11 up, 12 in 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: osdmap e76: 12 total, 11 up, 12 in 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.756 INFO:journalctl@ceph.osd.11.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-11[60612]: 2024-09-19T06:48:14.288+0000 7fe82771a640 -1 osd.11 0 waiting for initial osdmap 2024-09-19T06:48:14.756 INFO:journalctl@ceph.osd.11.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-11[60612]: 2024-09-19T06:48:14.301+0000 7fe823542640 -1 osd.11 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: osdmap e76: 12 total, 11 up, 12 in 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:48:14.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:14.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:14.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:14.758 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:14 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:15.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.117+0000 7f9f2ec00640 1 -- 172.21.15.114:0/164222490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 msgr2=0x7f9f28074fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:15.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.117+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/164222490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 0x7f9f28074fa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009900 tx=0x7f9f1402a410 comp rx=0 tx=0).stop 2024-09-19T06:48:15.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.117+0000 7f9f2ec00640 1 -- 172.21.15.114:0/164222490 shutdown_connections 2024-09-19T06:48:15.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.117+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/164222490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f281098b0 0x7f9f2810bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.117+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/164222490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 0x7f9f28074fa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.117+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/164222490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f280761a0 0x7f9f28074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.117+0000 7f9f2ec00640 1 -- 172.21.15.114:0/164222490 >> 172.21.15.114:0/164222490 conn(0x7f9f280fd1f0 msgr2=0x7f9f280ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:15.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.117+0000 7f9f2ec00640 1 -- 172.21.15.114:0/164222490 shutdown_connections 2024-09-19T06:48:15.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.118+0000 7f9f2ec00640 1 -- 172.21.15.114:0/164222490 wait complete. 2024-09-19T06:48:15.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.118+0000 7f9f2ec00640 1 Processor -- start 2024-09-19T06:48:15.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2ec00640 1 -- start start 2024-09-19T06:48:15.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 0x7f9f28199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:15.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f280761a0 0x7f9f2819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:15.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2ec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f281098b0 0x7f9f281a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:15.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2810f8b0 con 0x7f9f28074b40 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2810f730 con 0x7f9f280761a0 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2ec00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2810fa30 con 0x7f9f281098b0 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 0x7f9f28199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 0x7f9f28199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49526/0 (socket says 172.21.15.114:49526) 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2d800640 1 -- 172.21.15.114:0/4057984139 learned_addr learned my addr 172.21.15.114:0/4057984139 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2ce00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f280761a0 0x7f9f2819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.119+0000 7f9f2e200640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f281098b0 0x7f9f281a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f2d800640 1 -- 172.21.15.114:0/4057984139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f281098b0 msgr2=0x7f9f281a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f2d800640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f281098b0 0x7f9f281a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f2d800640 1 -- 172.21.15.114:0/4057984139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f280761a0 msgr2=0x7f9f2819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f2d800640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f280761a0 0x7f9f2819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f2d800640 1 -- 172.21.15.114:0/4057984139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f281a1b40 con 0x7f9f28074b40 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f2d800640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 0x7f9f28199f10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00ea60 tx=0x7f9f1c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:15.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f26000640 1 -- 172.21.15.114:0/4057984139 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c00cde0 con 0x7f9f28074b40 2024-09-19T06:48:15.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f26000640 1 -- 172.21.15.114:0/4057984139 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f1c0035f0 con 0x7f9f28074b40 2024-09-19T06:48:15.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f281a1e30 con 0x7f9f28074b40 2024-09-19T06:48:15.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f26000640 1 -- 172.21.15.114:0/4057984139 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c010670 con 0x7f9f28074b40 2024-09-19T06:48:15.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.120+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f281a2290 con 0x7f9f28074b40 2024-09-19T06:48:15.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.122+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f281086c0 con 0x7f9f28074b40 2024-09-19T06:48:15.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.123+0000 7f9f26000640 1 -- 172.21.15.114:0/4057984139 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f9f1c0037a0 con 0x7f9f28074b40 2024-09-19T06:48:15.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.125+0000 7f9f26000640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9f100a32f0 0x7f9f100a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:15.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.125+0000 7f9f26000640 1 -- 172.21.15.114:0/4057984139 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6847+0+0 (secure 0 0 0) 0x7f9f1c0cf640 con 0x7f9f28074b40 2024-09-19T06:48:15.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.125+0000 7f9f2ce00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9f100a32f0 0x7f9f100a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:15.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.126+0000 7f9f2ce00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9f100a32f0 0x7f9f100a57b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f140314e0 tx=0x7f9f14000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:15.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.129+0000 7f9f26000640 1 -- 172.21.15.114:0/4057984139 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9f1c09c880 con 0x7f9f28074b40 2024-09-19T06:48:15.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.315+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f9f28062e90 con 0x7f9f28074b40 2024-09-19T06:48:15.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.316+0000 7f9f26000640 1 -- 172.21.15.114:0/4057984139 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) ==== 74+0+133 (secure 0 0 0) 0x7f9f1c09c180 con 0x7f9f28074b40 2024-09-19T06:48:15.317 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:15.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9f100a32f0 msgr2=0x7f9f100a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9f100a32f0 0x7f9f100a57b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9f140314e0 tx=0x7f9f14000bc0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 msgr2=0x7f9f28199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 0x7f9f28199f10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00ea60 tx=0x7f9f1c00ef20 comp rx=0 tx=0).stop 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 shutdown_connections 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9f100a32f0 0x7f9f100a57b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f281098b0 0x7f9f281a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f280761a0 0x7f9f2819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 --2- 172.21.15.114:0/4057984139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f28074b40 0x7f9f28199f10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 >> 172.21.15.114:0/4057984139 conn(0x7f9f280fd1f0 msgr2=0x7f9f28105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 shutdown_connections 2024-09-19T06:48:15.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:15.318+0000 7f9f2ec00640 1 -- 172.21.15.114:0/4057984139 wait complete. 2024-09-19T06:48:15.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:15 smithi114 ceph-mon[30820]: Detected new or changed devices on smithi118 2024-09-19T06:48:15.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:15 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi118 to 4272M 2024-09-19T06:48:15.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:15 smithi114 ceph-mon[30820]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:48:15.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:15 smithi114 ceph-mon[30820]: osdmap e77: 12 total, 11 up, 12 in 2024-09-19T06:48:15.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:15.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:15.549 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:15 smithi114 ceph-mon[30820]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:15 smithi116 ceph-mon[35411]: Detected new or changed devices on smithi118 2024-09-19T06:48:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:15 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi118 to 4272M 2024-09-19T06:48:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:15 smithi116 ceph-mon[35411]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:48:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:15 smithi116 ceph-mon[35411]: osdmap e77: 12 total, 11 up, 12 in 2024-09-19T06:48:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:15 smithi116 ceph-mon[35411]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:15 smithi118 ceph-mon[35527]: Detected new or changed devices on smithi118 2024-09-19T06:48:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:15 smithi118 ceph-mon[35527]: Adjusting osd_memory_target on smithi118 to 4272M 2024-09-19T06:48:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:15 smithi118 ceph-mon[35527]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-09-19T06:48:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:15 smithi118 ceph-mon[35527]: osdmap e77: 12 total, 11 up, 12 in 2024-09-19T06:48:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:15.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:15.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:15 smithi118 ceph-mon[35527]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 983 GiB / 983 GiB avail 2024-09-19T06:48:16.234 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1726728495,"num_in_osds":12,"osd_in_since":1726728472,"num_remapped_pgs":0} 2024-09-19T06:48:16.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:48:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:16 smithi118 ceph-mon[35527]: purged_snaps scrub starts 2024-09-19T06:48:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:16 smithi118 ceph-mon[35527]: purged_snaps scrub ok 2024-09-19T06:48:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:16 smithi118 ceph-mon[35527]: osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] boot 2024-09-19T06:48:16.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:16 smithi118 ceph-mon[35527]: osdmap e78: 12 total, 12 up, 12 in 2024-09-19T06:48:16.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:16.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:16 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/4057984139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T06:48:16.596 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:16 smithi114 ceph-mon[30820]: purged_snaps scrub starts 2024-09-19T06:48:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:16 smithi114 ceph-mon[30820]: purged_snaps scrub ok 2024-09-19T06:48:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:16 smithi114 ceph-mon[30820]: osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] boot 2024-09-19T06:48:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:16 smithi114 ceph-mon[30820]: osdmap e78: 12 total, 12 up, 12 in 2024-09-19T06:48:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:16 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/4057984139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T06:48:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:16 smithi116 ceph-mon[35411]: purged_snaps scrub starts 2024-09-19T06:48:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:16 smithi116 ceph-mon[35411]: purged_snaps scrub ok 2024-09-19T06:48:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:16 smithi116 ceph-mon[35411]: osd.11 [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] boot 2024-09-19T06:48:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:16 smithi116 ceph-mon[35411]: osdmap e78: 12 total, 12 up, 12 in 2024-09-19T06:48:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-19T06:48:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:16 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/4057984139' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 -- 172.21.15.114:0/103234821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34106f10 msgr2=0x7fad3410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 --2- 172.21.15.114:0/103234821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34106f10 0x7fad3410ea90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009ca0 tx=0x7fad2c02a8c0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 -- 172.21.15.114:0/103234821 shutdown_connections 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 --2- 172.21.15.114:0/103234821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34106f10 0x7fad3410ea90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 --2- 172.21.15.114:0/103234821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad34102790 0x7fad341067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 --2- 172.21.15.114:0/103234821 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fad34101de0 0x7fad341021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 -- 172.21.15.114:0/103234821 >> 172.21.15.114:0/103234821 conn(0x7fad340fd8c0 msgr2=0x7fad340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 -- 172.21.15.114:0/103234821 shutdown_connections 2024-09-19T06:48:17.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.165+0000 7fad38c00640 1 -- 172.21.15.114:0/103234821 wait complete. 2024-09-19T06:48:17.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.166+0000 7fad38c00640 1 Processor -- start 2024-09-19T06:48:17.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.166+0000 7fad38c00640 1 -- start start 2024-09-19T06:48:17.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.166+0000 7fad38c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fad34101de0 0x7fad3419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:17.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.166+0000 7fad38c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34102790 0x7fad3419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:17.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad38c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad34106f10 0x7fad3419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:17.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad38c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fad34111f70 con 0x7fad34102790 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad38c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad34111df0 con 0x7fad34106f10 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad38c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fad341120f0 con 0x7fad34101de0 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34102790 0x7fad3419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34102790 0x7fad3419ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45686/0 (socket says 172.21.15.114:45686) 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31400640 1 -- 172.21.15.114:0/3542960732 learned_addr learned my addr 172.21.15.114:0/3542960732 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31e00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fad34101de0 0x7fad3419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad32800640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad34106f10 0x7fad3419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31400640 1 -- 172.21.15.114:0/3542960732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fad34101de0 msgr2=0x7fad3419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31400640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fad34101de0 0x7fad3419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31400640 1 -- 172.21.15.114:0/3542960732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad34106f10 msgr2=0x7fad3419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31400640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad34106f10 0x7fad3419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.167+0000 7fad31400640 1 -- 172.21.15.114:0/3542960732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad3419f5d0 con 0x7fad34102790 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.168+0000 7fad31400640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34102790 0x7fad3419ab60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fad2000e9f0 tx=0x7fad2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:17.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.168+0000 7fad2aa00640 1 -- 172.21.15.114:0/3542960732 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2000cd80 con 0x7fad34102790 2024-09-19T06:48:17.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.168+0000 7fad2aa00640 1 -- 172.21.15.114:0/3542960732 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad20010910 con 0x7fad34102790 2024-09-19T06:48:17.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.168+0000 7fad2aa00640 1 -- 172.21.15.114:0/3542960732 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20018990 con 0x7fad34102790 2024-09-19T06:48:17.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.168+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad3419f8c0 con 0x7fad34102790 2024-09-19T06:48:17.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.168+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad341a71a0 con 0x7fad34102790 2024-09-19T06:48:17.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.169+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad341021c0 con 0x7fad34102790 2024-09-19T06:48:17.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.171+0000 7fad2aa00640 1 -- 172.21.15.114:0/3542960732 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fad20010ac0 con 0x7fad34102790 2024-09-19T06:48:17.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.173+0000 7fad2aa00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fad180a32c0 0x7fad180a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:17.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.173+0000 7fad2aa00640 1 -- 172.21.15.114:0/3542960732 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7fad200cfa90 con 0x7fad34102790 2024-09-19T06:48:17.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.173+0000 7fad31e00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fad180a32c0 0x7fad180a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:17.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.174+0000 7fad31e00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fad180a32c0 0x7fad180a5780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fad24009bd0 tx=0x7fad24005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:17.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.174+0000 7fad2aa00640 1 -- 172.21.15.114:0/3542960732 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fad2009c560 con 0x7fad34102790 2024-09-19T06:48:17.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.352+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad341a7370 con 0x7fad34102790 2024-09-19T06:48:17.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:17 smithi114 ceph-mon[30820]: osdmap e79: 12 total, 12 up, 12 in 2024-09-19T06:48:17.353 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:17 smithi114 ceph-mon[30820]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:17.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.353+0000 7fad2aa00640 1 -- 172.21.15.114:0/3542960732 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+18527 (secure 0 0 0) 0x7fad200a0860 con 0x7fad34102790 2024-09-19T06:48:17.354 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:17.355 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":79,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:48:16.294724+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fad180a32c0 msgr2=0x7fad180a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fad180a32c0 0x7fad180a5780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fad24009bd0 tx=0x7fad24005f90 comp rx=0 tx=0).stop 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34102790 msgr2=0x7fad3419ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34102790 0x7fad3419ab60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fad2000e9f0 tx=0x7fad2000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 shutdown_connections 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fad180a32c0 0x7fad180a5780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad34106f10 0x7fad3419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fad34102790 0x7fad3419ab60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 --2- 172.21.15.114:0/3542960732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fad34101de0 0x7fad3419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:17.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.355+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 >> 172.21.15.114:0/3542960732 conn(0x7fad340fd8c0 msgr2=0x7fad340febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:17.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.356+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 shutdown_connections 2024-09-19T06:48:17.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:17.356+0000 7fad38c00640 1 -- 172.21.15.114:0/3542960732 wait complete. 2024-09-19T06:48:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:17 smithi116 ceph-mon[35411]: osdmap e79: 12 total, 12 up, 12 in 2024-09-19T06:48:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:17 smithi116 ceph-mon[35411]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:17 smithi118 ceph-mon[35527]: osdmap e79: 12 total, 12 up, 12 in 2024-09-19T06:48:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:17 smithi118 ceph-mon[35527]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:18.219 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-19T06:44:13.612575+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-19T06:48:18.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-19T06:48:18.583 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:18 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3542960732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:48:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:18 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3542960732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:48:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:18 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3542960732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:48:19.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 -- 172.21.15.114:0/1247044483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a81049a0 msgr2=0x7f97a81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:19.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 --2- 172.21.15.114:0/1247044483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a81049a0 0x7f97a81089f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9794009900 tx=0x7f979402a410 comp rx=0 tx=0).stop 2024-09-19T06:48:19.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 -- 172.21.15.114:0/1247044483 shutdown_connections 2024-09-19T06:48:19.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 --2- 172.21.15.114:0/1247044483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97a8109120 0x7f97a8110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 --2- 172.21.15.114:0/1247044483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a81049a0 0x7f97a81089f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 --2- 172.21.15.114:0/1247044483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97a8103ff0 0x7f97a81043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 -- 172.21.15.114:0/1247044483 >> 172.21.15.114:0/1247044483 conn(0x7f97a80ffa90 msgr2=0x7f97a8101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:19.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 -- 172.21.15.114:0/1247044483 shutdown_connections 2024-09-19T06:48:19.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.151+0000 7f97adc00640 1 -- 172.21.15.114:0/1247044483 wait complete. 2024-09-19T06:48:19.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.152+0000 7f97adc00640 1 Processor -- start 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.152+0000 7f97adc00640 1 -- start start 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97adc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97a8103ff0 0x7f97a819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97adc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97a81049a0 0x7f97a819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97adc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a8109120 0x7f97a819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97adc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f97a8114170 con 0x7f97a8109120 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97adc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97a8113ff0 con 0x7f97a8103ff0 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97adc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81142f0 con 0x7f97a81049a0 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97a7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a8109120 0x7f97a819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97a7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a8109120 0x7f97a819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45712/0 (socket says 172.21.15.114:45712) 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97a7400640 1 -- 172.21.15.114:0/3416060281 learned_addr learned my addr 172.21.15.114:0/3416060281 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:19.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97a6a00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97a8103ff0 0x7f97a819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.153+0000 7f97a6000640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97a81049a0 0x7f97a819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97a7400640 1 -- 172.21.15.114:0/3416060281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97a81049a0 msgr2=0x7f97a819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97a7400640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97a81049a0 0x7f97a819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97a7400640 1 -- 172.21.15.114:0/3416060281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97a8103ff0 msgr2=0x7f97a819a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97a7400640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97a8103ff0 0x7f97a819a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97a7400640 1 -- 172.21.15.114:0/3416060281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a819f600 con 0x7f97a8109120 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97a6000640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97a81049a0 0x7f97a819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97a6a00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97a8103ff0 0x7f97a819a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97a7400640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a8109120 0x7f97a819ef20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f97a000eb10 tx=0x7f97a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f979f400640 1 -- 172.21.15.114:0/3416060281 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0004b20 con 0x7f97a8109120 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f979f400640 1 -- 172.21.15.114:0/3416060281 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a0004cc0 con 0x7f97a8109120 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f979f400640 1 -- 172.21.15.114:0/3416060281 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0010760 con 0x7f97a8109120 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a819f8f0 con 0x7f97a8109120 2024-09-19T06:48:19.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.154+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a81a71d0 con 0x7f97a8109120 2024-09-19T06:48:19.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.155+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a81043d0 con 0x7f97a8109120 2024-09-19T06:48:19.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.157+0000 7f979f400640 1 -- 172.21.15.114:0/3416060281 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f97a0005020 con 0x7f97a8109120 2024-09-19T06:48:19.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.159+0000 7f979f400640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f978c0a32c0 0x7f978c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:19.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.160+0000 7f979f400640 1 -- 172.21.15.114:0/3416060281 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7f97a00cfa50 con 0x7f97a8109120 2024-09-19T06:48:19.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.160+0000 7f97a6a00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f978c0a32c0 0x7f978c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:19.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.160+0000 7f97a6a00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f978c0a32c0 0x7f978c0a5780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9798005d00 tx=0x7f979800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:19.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.163+0000 7f979f400640 1 -- 172.21.15.114:0/3416060281 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f97a009cb70 con 0x7f97a8109120 2024-09-19T06:48:19.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.340+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97a802d7c0 con 0x7f97a8109120 2024-09-19T06:48:19.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.340+0000 7f979f400640 1 -- 172.21.15.114:0/3416060281 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) ==== 93+0+10 (secure 0 0 0) 0x7f97a009c470 con 0x7f97a8109120 2024-09-19T06:48:19.342 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-09-19T06:48:19.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f978c0a32c0 msgr2=0x7f978c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:19.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f978c0a32c0 0x7f978c0a5780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9798005d00 tx=0x7f979800b410 comp rx=0 tx=0).stop 2024-09-19T06:48:19.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a8109120 msgr2=0x7f97a819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a8109120 0x7f97a819ef20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f97a000eb10 tx=0x7f97a000c2d0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 shutdown_connections 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f978c0a32c0 0x7f978c0a5780 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97a8109120 0x7f97a819ef20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97a81049a0 0x7f97a819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 --2- 172.21.15.114:0/3416060281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97a8103ff0 0x7f97a819a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 >> 172.21.15.114:0/3416060281 conn(0x7f97a80ffa90 msgr2=0x7f97a8100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 shutdown_connections 2024-09-19T06:48:19.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:19.343+0000 7f97adc00640 1 -- 172.21.15.114:0/3416060281 wait complete. 2024-09-19T06:48:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:19 smithi114 ceph-mon[30820]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:19 smithi116 ceph-mon[35411]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:19.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:19 smithi118 ceph-mon[35527]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.a on smithi114 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.d on smithi114 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.g on smithi114 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.j on smithi114 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.b on smithi116 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.e on smithi116 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.h on smithi116 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.k on smithi116 2024-09-19T06:48:20.215 INFO:tasks.cephadm:Adding mds.c on smithi118 2024-09-19T06:48:20.216 INFO:tasks.cephadm:Adding mds.f on smithi118 2024-09-19T06:48:20.216 INFO:tasks.cephadm:Adding mds.i on smithi118 2024-09-19T06:48:20.216 INFO:tasks.cephadm:Adding mds.l on smithi118 2024-09-19T06:48:20.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph orch apply mds all '12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l' 2024-09-19T06:48:20.470 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.c/config 2024-09-19T06:48:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:20 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3416060281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T06:48:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:20 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3416060281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T06:48:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:20 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3416060281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-19T06:48:20.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 -- 172.21.15.118:0/703357575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40141037c0 msgr2=0x7f401410a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 --2- 172.21.15.118:0/703357575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40141037c0 0x7f401410a050 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f400c009ca0 tx=0x7f400c02a8c0 comp rx=0 tx=0).stop 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 -- 172.21.15.118:0/703357575 shutdown_connections 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 --2- 172.21.15.118:0/703357575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40141037c0 0x7f401410a050 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 --2- 172.21.15.118:0/703357575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4014102e00 0x7f4014103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 --2- 172.21.15.118:0/703357575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4014101c00 0x7f4014102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 -- 172.21.15.118:0/703357575 >> 172.21.15.118:0/703357575 conn(0x7f40140fd390 msgr2=0x7f40140ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 -- 172.21.15.118:0/703357575 shutdown_connections 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.891+0000 7f40191ea640 1 -- 172.21.15.118:0/703357575 wait complete. 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f40191ea640 1 Processor -- start 2024-09-19T06:48:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f40191ea640 1 -- start start 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f40191ea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4014101c00 0x7f4014075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f40191ea640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4014102e00 0x7f40140745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f40191ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40141037c0 0x7f4014074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f40191ea640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f401410d860 con 0x7f4014101c00 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f40191ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f401410d6e0 con 0x7f40141037c0 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f40191ea640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f401410d9e0 con 0x7f4014102e00 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f401259c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4014102e00 0x7f40140745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f401259c640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4014102e00 0x7f40140745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35864/0 (socket says 172.21.15.118:35864) 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f401259c640 1 -- 172.21.15.118:0/1213288759 learned_addr learned my addr 172.21.15.118:0/1213288759 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.892+0000 7f4012d9d640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4014101c00 0x7f4014075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f401259c640 1 -- 172.21.15.118:0/1213288759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40141037c0 msgr2=0x7f4014074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f401359e640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40141037c0 0x7f4014074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f401259c640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40141037c0 0x7f4014074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f401259c640 1 -- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4014101c00 msgr2=0x7f4014075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f401259c640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4014101c00 0x7f4014075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f401259c640 1 -- 172.21.15.118:0/1213288759 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40140753e0 con 0x7f4014102e00 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f4012d9d640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4014101c00 0x7f4014075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f401259c640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4014102e00 0x7f40140745e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00c920 tx=0x7f3ffc00cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f400bfff640 1 -- 172.21.15.118:0/1213288759 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc007c70 con 0x7f4014102e00 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f400bfff640 1 -- 172.21.15.118:0/1213288759 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ffc007e10 con 0x7f4014102e00 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40141a1f90 con 0x7f4014102e00 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f400bfff640 1 -- 172.21.15.118:0/1213288759 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc018890 con 0x7f4014102e00 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.893+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40141a23f0 con 0x7f4014102e00 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.894+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4014108f30 con 0x7f4014102e00 2024-09-19T06:48:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.895+0000 7f400bfff640 1 -- 172.21.15.118:0/1213288759 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f3ffc018a50 con 0x7f4014102e00 2024-09-19T06:48:20.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.896+0000 7f400bfff640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40000a3240 0x7f40000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:20.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.897+0000 7f400bfff640 1 -- 172.21.15.118:0/1213288759 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7f3ffc0cf690 con 0x7f4014102e00 2024-09-19T06:48:20.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.897+0000 7f4012d9d640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40000a3240 0x7f40000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:20.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.897+0000 7f4012d9d640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40000a3240 0x7f40000a5700 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4014102c60 tx=0x7f400400a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:20.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:20.901+0000 7f400bfff640 1 -- 172.21.15.118:0/1213288759 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f3ffc014030 con 0x7f4014102e00 2024-09-19T06:48:21.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.075+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l", "target": ["mon-mgr", ""]}) -- 0x7f4014062730 con 0x7f40000a3240 2024-09-19T06:48:21.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.082+0000 7f400bfff640 1 -- 172.21.15.118:0/1213288759 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f4014062730 con 0x7f40000a3240 2024-09-19T06:48:21.083 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled mds.all update... 2024-09-19T06:48:21.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.084+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40000a3240 msgr2=0x7f40000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.084+0000 7f40191ea640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40000a3240 0x7f40000a5700 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4014102c60 tx=0x7f400400a560 comp rx=0 tx=0).stop 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.084+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4014102e00 msgr2=0x7f40140745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.084+0000 7f40191ea640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4014102e00 0x7f40140745e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc00c920 tx=0x7f3ffc00cde0 comp rx=0 tx=0).stop 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.085+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 shutdown_connections 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.085+0000 7f40191ea640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40000a3240 0x7f40000a5700 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.085+0000 7f40191ea640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40141037c0 0x7f4014074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.085+0000 7f40191ea640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4014102e00 0x7f40140745e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.085+0000 7f40191ea640 1 --2- 172.21.15.118:0/1213288759 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4014101c00 0x7f4014075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.085+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 >> 172.21.15.118:0/1213288759 conn(0x7f40140fd390 msgr2=0x7f4014106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.085+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 shutdown_connections 2024-09-19T06:48:21.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-19T06:48:21.085+0000 7f40191ea640 1 -- 172.21.15.118:0/1213288759 wait complete. 2024-09-19T06:48:21.838 DEBUG:teuthology.orchestra.run.smithi114:mds.a> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.a.service 2024-09-19T06:48:21.842 DEBUG:teuthology.orchestra.run.smithi114:mds.d> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.d.service 2024-09-19T06:48:21.888 DEBUG:teuthology.orchestra.run.smithi114:mds.g> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.g.service 2024-09-19T06:48:21.933 DEBUG:teuthology.orchestra.run.smithi114:mds.j> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.j.service 2024-09-19T06:48:21.978 DEBUG:teuthology.orchestra.run.smithi116:mds.b> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.b.service 2024-09-19T06:48:21.981 DEBUG:teuthology.orchestra.run.smithi116:mds.e> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.e.service 2024-09-19T06:48:22.024 DEBUG:teuthology.orchestra.run.smithi116:mds.h> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.h.service 2024-09-19T06:48:22.068 DEBUG:teuthology.orchestra.run.smithi116:mds.k> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.k.service 2024-09-19T06:48:22.111 DEBUG:teuthology.orchestra.run.smithi118:mds.c> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.c.service 2024-09-19T06:48:22.113 DEBUG:teuthology.orchestra.run.smithi118:mds.f> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.f.service 2024-09-19T06:48:22.156 DEBUG:teuthology.orchestra.run.smithi118:mds.i> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.i.service 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l;count:12 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:22.199 DEBUG:teuthology.orchestra.run.smithi118:mds.l> sudo journalctl -f -n 0 -u ceph-fface640-7651-11ef-bceb-c7b262605968@mds.l.service 2024-09-19T06:48:22.244 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-19T06:48:22.244 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:22.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:22.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:48:22.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l;count:12 2024-09-19T06:48:22.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:22.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:22.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:22.369 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:22.370 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:22.370 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:22.370 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:22.370 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l;count:12 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:22.601 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 -- 172.21.15.114:0/1971144316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 msgr2=0x7fe6141124f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 --2- 172.21.15.114:0/1971144316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 0x7fe6141124f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe600009900 tx=0x7fe60002a410 comp rx=0 tx=0).stop 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 -- 172.21.15.114:0/1971144316 shutdown_connections 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 --2- 172.21.15.114:0/1971144316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe614112a30 0x7fe614114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 --2- 172.21.15.114:0/1971144316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 0x7fe6141124f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 --2- 172.21.15.114:0/1971144316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe614076720 0x7fe614076b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 -- 172.21.15.114:0/1971144316 >> 172.21.15.114:0/1971144316 conn(0x7fe6140ffa90 msgr2=0x7fe614101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 -- 172.21.15.114:0/1971144316 shutdown_connections 2024-09-19T06:48:23.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.194+0000 7fe61aa00640 1 -- 172.21.15.114:0/1971144316 wait complete. 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.195+0000 7fe61aa00640 1 Processor -- start 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.195+0000 7fe61aa00640 1 -- start start 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe61aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe614076720 0x7fe6141a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe61aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 0x7fe6141a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe61aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe614112a30 0x7fe61419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe61aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6141185f0 con 0x7fe614077040 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe61aa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe614118470 con 0x7fe614112a30 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe61aa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe614118770 con 0x7fe614076720 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe613e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe614076720 0x7fe6141a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe613400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 0x7fe6141a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:23.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe613400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 0x7fe6141a4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45740/0 (socket says 172.21.15.114:45740) 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe613400640 1 -- 172.21.15.114:0/919198528 learned_addr learned my addr 172.21.15.114:0/919198528 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe613e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe614076720 0x7fe6141a4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:41818/0 (socket says 172.21.15.114:41818) 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.196+0000 7fe618a00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe614112a30 0x7fe61419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe613e00640 1 -- 172.21.15.114:0/919198528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe614112a30 msgr2=0x7fe61419e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe613e00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe614112a30 0x7fe61419e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe613e00640 1 -- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 msgr2=0x7fe6141a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe613e00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 0x7fe6141a4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe613e00640 1 -- 172.21.15.114:0/919198528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61419f010 con 0x7fe614076720 2024-09-19T06:48:23.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe613e00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe614076720 0x7fe6141a4720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe60400e9f0 tx=0x7fe60400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:23.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe610c00640 1 -- 172.21.15.114:0/919198528 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60400cd80 con 0x7fe614076720 2024-09-19T06:48:23.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe610c00640 1 -- 172.21.15.114:0/919198528 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6040035f0 con 0x7fe614076720 2024-09-19T06:48:23.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe61419f300 con 0x7fe614076720 2024-09-19T06:48:23.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe610c00640 1 -- 172.21.15.114:0/919198528 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604010690 con 0x7fe614076720 2024-09-19T06:48:23.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.197+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6141ab5f0 con 0x7fe614076720 2024-09-19T06:48:23.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.198+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe614076b00 con 0x7fe614076720 2024-09-19T06:48:23.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.200+0000 7fe610c00640 1 -- 172.21.15.114:0/919198528 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fe604003b30 con 0x7fe614076720 2024-09-19T06:48:23.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.201+0000 7fe610c00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe5f80a3240 0x7fe5f80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:23.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.201+0000 7fe610c00640 1 -- 172.21.15.114:0/919198528 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7fe6040cf8a0 con 0x7fe614076720 2024-09-19T06:48:23.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.202+0000 7fe613400640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe5f80a3240 0x7fe5f80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:23.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.202+0000 7fe613400640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe5f80a3240 0x7fe5f80a5700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe614079300 tx=0x7fe6000046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:23.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.205+0000 7fe610c00640 1 -- 172.21.15.114:0/919198528 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe60409c9c0 con 0x7fe614076720 2024-09-19T06:48:23.240 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:23 smithi118 ceph-mon[35527]: Deploying daemon mds.l on smithi118 2024-09-19T06:48:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:23 smithi114 ceph-mon[30820]: Deploying daemon mds.l on smithi118 2024-09-19T06:48:23.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.416+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe61402d7c0 con 0x7fe614076720 2024-09-19T06:48:23.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.417+0000 7fe610c00640 1 -- 172.21.15.114:0/919198528 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7fe60409c2c0 con 0x7fe614076720 2024-09-19T06:48:23.418 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:23.418 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"btime":"2024-09-19T06:40:58:566495+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-19T06:48:23.419 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 1 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe5f80a3240 msgr2=0x7fe5f80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe5f80a3240 0x7fe5f80a5700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe614079300 tx=0x7fe6000046c0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe614076720 msgr2=0x7fe6141a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe614076720 0x7fe6141a4720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe60400e9f0 tx=0x7fe60400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 shutdown_connections 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe5f80a3240 0x7fe5f80a5700 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe614112a30 0x7fe61419e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe614077040 0x7fe6141a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 --2- 172.21.15.114:0/919198528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe614076720 0x7fe6141a4720 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 >> 172.21.15.114:0/919198528 conn(0x7fe6140ffa90 msgr2=0x7fe614101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:23.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.419+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 shutdown_connections 2024-09-19T06:48:23.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:23.420+0000 7fe61aa00640 1 -- 172.21.15.114:0/919198528 wait complete. 2024-09-19T06:48:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:23 smithi116 ceph-mon[35411]: Deploying daemon mds.l on smithi118 2024-09-19T06:48:23.506 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 systemd[1]: Starting Ceph mds.l for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:48:23.903 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 podman[65085]: 2024-09-19 06:48:23.5408515 +0000 UTC m=+0.015518767 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:48:23.903 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 podman[65085]: 2024-09-19 06:48:23.640611983 +0000 UTC m=+0.115279237 container create cbb3da9be1cf464b4174248c7cafcec7b20a7fdefb2306e0abad30bfda50a297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-l, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-2f599a4) 2024-09-19T06:48:23.903 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 podman[65085]: 2024-09-19 06:48:23.876604957 +0000 UTC m=+0.351272214 container init cbb3da9be1cf464b4174248c7cafcec7b20a7fdefb2306e0abad30bfda50a297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-l, GIT_CLEAN=True, RELEASE=squid-2f599a4, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-19T06:48:23.903 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 podman[65085]: 2024-09-19 06:48:23.879411063 +0000 UTC m=+0.354078318 container start cbb3da9be1cf464b4174248c7cafcec7b20a7fdefb2306e0abad30bfda50a297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-l, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-19T06:48:24.256 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 ceph-mds[65104]: -- 172.21.15.118:0/2609259158 <== mon.1 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55736cc17860 con 0x55736cbf5000 2024-09-19T06:48:24.256 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-l[65100]: starting mds.l at 2024-09-19T06:48:24.256 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 bash[65085]: cbb3da9be1cf464b4174248c7cafcec7b20a7fdefb2306e0abad30bfda50a297 2024-09-19T06:48:24.256 INFO:journalctl@ceph.mds.l.smithi118.stdout:Sep 19 06:48:23 smithi118 systemd[1]: Started Ceph mds.l for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/919198528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.118:6832/3230062684,v1:172.21.15.118:6833/3230062684] up:boot 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: fsmap 1 up:standby 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:24.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/919198528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.118:6832/3230062684,v1:172.21.15.118:6833/3230062684] up:boot 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: fsmap 1 up:standby 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/919198528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.118:6832/3230062684,v1:172.21.15.118:6833/3230062684] up:boot 2024-09-19T06:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: fsmap 1 up:standby 2024-09-19T06:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-19T06:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:25 smithi114 ceph-mon[30820]: Deploying daemon mds.k on smithi116 2024-09-19T06:48:25.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:25 smithi116 ceph-mon[35411]: Deploying daemon mds.k on smithi116 2024-09-19T06:48:25.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:25 smithi118 ceph-mon[35527]: Deploying daemon mds.k on smithi116 2024-09-19T06:48:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:26 smithi116 ceph-mon[35411]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:26.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:26.367 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:26 smithi114 ceph-mon[30820]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:26.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:26 smithi118 ceph-mon[35527]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:26.683 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:26.928 INFO:journalctl@ceph.mds.k.smithi116.stdout:Sep 19 06:48:26 smithi116 systemd[1]: Started Ceph mds.k for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:27.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.273+0000 7fb043200640 1 -- 172.21.15.114:0/3139148033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c106f10 msgr2=0x7fb03c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.273+0000 7fb043200640 1 --2- 172.21.15.114:0/3139148033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c106f10 0x7fb03c10ea90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb034009ca0 tx=0x7fb03402a8c0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.273+0000 7fb043200640 1 -- 172.21.15.114:0/3139148033 shutdown_connections 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.273+0000 7fb043200640 1 --2- 172.21.15.114:0/3139148033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c106f10 0x7fb03c10ea90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.273+0000 7fb043200640 1 --2- 172.21.15.114:0/3139148033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb03c102790 0x7fb03c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.273+0000 7fb043200640 1 --2- 172.21.15.114:0/3139148033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb03c101de0 0x7fb03c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.273+0000 7fb043200640 1 -- 172.21.15.114:0/3139148033 >> 172.21.15.114:0/3139148033 conn(0x7fb03c0fd8c0 msgr2=0x7fb03c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.274+0000 7fb043200640 1 -- 172.21.15.114:0/3139148033 shutdown_connections 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.274+0000 7fb043200640 1 -- 172.21.15.114:0/3139148033 wait complete. 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.274+0000 7fb043200640 1 Processor -- start 2024-09-19T06:48:27.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.274+0000 7fb043200640 1 -- start start 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb043200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c101de0 0x7fb03c1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb043200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb03c102790 0x7fb03c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb043200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb03c106f10 0x7fb03c19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb043200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c111f70 con 0x7fb03c101de0 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb043200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c111df0 con 0x7fb03c106f10 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb043200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb03c1120f0 con 0x7fb03c102790 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb03be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c101de0 0x7fb03c1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb03be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c101de0 0x7fb03c1983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38012/0 (socket says 172.21.15.114:38012) 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb03be00640 1 -- 172.21.15.114:0/771457446 learned_addr learned my addr 172.21.15.114:0/771457446 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb041200640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb03c106f10 0x7fb03c19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:27.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.275+0000 7fb03b400640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb03c102790 0x7fb03c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb03be00640 1 -- 172.21.15.114:0/771457446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb03c102790 msgr2=0x7fb03c198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb03be00640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb03c102790 0x7fb03c198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb03be00640 1 -- 172.21.15.114:0/771457446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb03c106f10 msgr2=0x7fb03c19cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb03be00640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb03c106f10 0x7fb03c19cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb03be00640 1 -- 172.21.15.114:0/771457446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03c19d370 con 0x7fb03c101de0 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb041200640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb03c106f10 0x7fb03c19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb03be00640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c101de0 0x7fb03c1983c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00e9f0 tx=0x7fb02c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb038c00640 1 -- 172.21.15.114:0/771457446 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00cd80 con 0x7fb03c101de0 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb03c19d660 con 0x7fb03c101de0 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb038c00640 1 -- 172.21.15.114:0/771457446 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb02c010910 con 0x7fb03c101de0 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb038c00640 1 -- 172.21.15.114:0/771457446 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0188f0 con 0x7fb03c101de0 2024-09-19T06:48:27.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.276+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb03c1a4f40 con 0x7fb03c101de0 2024-09-19T06:48:27.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.278+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb03c1021c0 con 0x7fb03c101de0 2024-09-19T06:48:27.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.280+0000 7fb038c00640 1 -- 172.21.15.114:0/771457446 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fb02c010ac0 con 0x7fb03c101de0 2024-09-19T06:48:27.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.281+0000 7fb038c00640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb0200a3240 0x7fb0200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:27.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.281+0000 7fb038c00640 1 -- 172.21.15.114:0/771457446 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7fb02c0cfc20 con 0x7fb03c101de0 2024-09-19T06:48:27.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.281+0000 7fb03b400640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb0200a3240 0x7fb0200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:27.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.282+0000 7fb03b400640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb0200a3240 0x7fb0200a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0fdce0 tx=0x7fb028005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:27.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.284+0000 7fb038c00640 1 -- 172.21.15.114:0/771457446 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb02c09c6f0 con 0x7fb03c101de0 2024-09-19T06:48:27.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.515+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb03c10a180 con 0x7fb03c101de0 2024-09-19T06:48:27.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.516+0000 7fb038c00640 1 -- 172.21.15.114:0/771457446 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2266 (secure 0 0 0) 0x7fb02c0a09f0 con 0x7fb03c101de0 2024-09-19T06:48:27.517 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:27.517 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":3,"btime":"2024-09-19T06:48:27:101611+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-19T06:48:27.518 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 3 2024-09-19T06:48:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb0200a3240 msgr2=0x7fb0200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:27.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb0200a3240 0x7fb0200a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0fdce0 tx=0x7fb028005f90 comp rx=0 tx=0).stop 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c101de0 msgr2=0x7fb03c1983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c101de0 0x7fb03c1983c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00e9f0 tx=0x7fb02c00eeb0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 shutdown_connections 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb0200a3240 0x7fb0200a5700 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb03c106f10 0x7fb03c19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb03c102790 0x7fb03c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 --2- 172.21.15.114:0/771457446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb03c101de0 0x7fb03c1983c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.518+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 >> 172.21.15.114:0/771457446 conn(0x7fb03c0fd8c0 msgr2=0x7fb03c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.519+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 shutdown_connections 2024-09-19T06:48:27.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:27.519+0000 7fb043200640 1 -- 172.21.15.114:0/771457446 wait complete. 2024-09-19T06:48:27.910 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: Deploying daemon mds.j on smithi114 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:boot 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: fsmap 2 up:standby 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T06:48:27.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:27 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/771457446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:28.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:28.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: Deploying daemon mds.j on smithi114 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:boot 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: fsmap 2 up:standby 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T06:48:28.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:27 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/771457446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:28.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: Deploying daemon mds.j on smithi114 2024-09-19T06:48:28.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:28.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:boot 2024-09-19T06:48:28.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: fsmap 2 up:standby 2024-09-19T06:48:28.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-19T06:48:28.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:27 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/771457446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:30 smithi114 ceph-mon[30820]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:30 smithi116 ceph-mon[35411]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:30.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:30.495 INFO:journalctl@ceph.mds.j.smithi114.stdout:Sep 19 06:48:30 smithi114 podman[69991]: 2024-09-19 06:48:30.225182524 +0000 UTC m=+0.147106881 container create 00ed245ba6693e2e9b5c2ea8ed4a99befd168f259f7458cb828ab3c0af3e130a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-j, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-19T06:48:30.495 INFO:journalctl@ceph.mds.j.smithi114.stdout:Sep 19 06:48:30 smithi114 podman[69991]: 2024-09-19 06:48:30.494805847 +0000 UTC m=+0.416730205 container init 00ed245ba6693e2e9b5c2ea8ed4a99befd168f259f7458cb828ab3c0af3e130a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-19T06:48:30.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:30 smithi118 ceph-mon[35527]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:30.887 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:30.916 INFO:journalctl@ceph.mds.j.smithi114.stdout:Sep 19 06:48:30 smithi114 podman[69991]: 2024-09-19 06:48:30.501352655 +0000 UTC m=+0.423277009 container start 00ed245ba6693e2e9b5c2ea8ed4a99befd168f259f7458cb828ab3c0af3e130a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-j, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-19T06:48:30.916 INFO:journalctl@ceph.mds.j.smithi114.stdout:Sep 19 06:48:30 smithi114 ceph-mds[70018]: -- 172.21.15.114:0/48628126 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5634d0e7d860 con 0x5634d0e5a800 2024-09-19T06:48:30.916 INFO:journalctl@ceph.mds.j.smithi114.stdout:Sep 19 06:48:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-j[70006]: starting mds.j at 2024-09-19T06:48:30.916 INFO:journalctl@ceph.mds.j.smithi114.stdout:Sep 19 06:48:30 smithi114 bash[69991]: 00ed245ba6693e2e9b5c2ea8ed4a99befd168f259f7458cb828ab3c0af3e130a 2024-09-19T06:48:30.916 INFO:journalctl@ceph.mds.j.smithi114.stdout:Sep 19 06:48:30 smithi114 systemd[1]: Started Ceph mds.j for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:31.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.510+0000 7f3ca1200640 1 -- 172.21.15.114:0/2207387629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c101de0 msgr2=0x7f3c9c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:31.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.510+0000 7f3ca1200640 1 --2- 172.21.15.114:0/2207387629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c101de0 0x7f3c9c1021c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009900 tx=0x7f3c8c02a3f0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.511+0000 7f3ca1200640 1 -- 172.21.15.114:0/2207387629 shutdown_connections 2024-09-19T06:48:31.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.511+0000 7f3ca1200640 1 --2- 172.21.15.114:0/2207387629 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c9c106f10 0x7f3c9c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.511+0000 7f3ca1200640 1 --2- 172.21.15.114:0/2207387629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9c102790 0x7f3c9c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.511+0000 7f3ca1200640 1 --2- 172.21.15.114:0/2207387629 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c101de0 0x7f3c9c1021c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.511+0000 7f3ca1200640 1 -- 172.21.15.114:0/2207387629 >> 172.21.15.114:0/2207387629 conn(0x7f3c9c0fd8c0 msgr2=0x7f3c9c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.511+0000 7f3ca1200640 1 -- 172.21.15.114:0/2207387629 shutdown_connections 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.511+0000 7f3ca1200640 1 -- 172.21.15.114:0/2207387629 wait complete. 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3ca1200640 1 Processor -- start 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3ca1200640 1 -- start start 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3ca1200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9c101de0 0x7f3c9c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3ca1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c102790 0x7f3c9c19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3ca1200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c9c106f10 0x7f3c9c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3ca1200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c111f30 con 0x7f3c9c102790 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3ca1200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c111db0 con 0x7f3c9c101de0 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3ca1200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c1120b0 con 0x7f3c9c106f10 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3c99400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c102790 0x7f3c9c19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3c99400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c102790 0x7f3c9c19ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38060/0 (socket says 172.21.15.114:38060) 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3c99e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9c101de0 0x7f3c9c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.512+0000 7f3c99400640 1 -- 172.21.15.114:0/758189518 learned_addr learned my addr 172.21.15.114:0/758189518 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c9a800640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c9c106f10 0x7f3c9c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c99400640 1 -- 172.21.15.114:0/758189518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c9c106f10 msgr2=0x7f3c9c19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c99400640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c9c106f10 0x7f3c9c19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c99400640 1 -- 172.21.15.114:0/758189518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9c101de0 msgr2=0x7f3c9c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c99400640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9c101de0 0x7f3c9c19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c99400640 1 -- 172.21.15.114:0/758189518 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9c19f5c0 con 0x7f3c9c102790 2024-09-19T06:48:31.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c99400640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c102790 0x7f3c9c19ab50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800e9f0 tx=0x7f3c8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:31.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c92a00640 1 -- 172.21.15.114:0/758189518 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8800cd80 con 0x7f3c9c102790 2024-09-19T06:48:31.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9c19f8b0 con 0x7f3c9c102790 2024-09-19T06:48:31.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c92a00640 1 -- 172.21.15.114:0/758189518 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c88010910 con 0x7f3c9c102790 2024-09-19T06:48:31.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.513+0000 7f3c92a00640 1 -- 172.21.15.114:0/758189518 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c88018990 con 0x7f3c9c102790 2024-09-19T06:48:31.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.514+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c9c1a7190 con 0x7f3c9c102790 2024-09-19T06:48:31.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.515+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c9c1021c0 con 0x7f3c9c102790 2024-09-19T06:48:31.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.517+0000 7f3c92a00640 1 -- 172.21.15.114:0/758189518 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f3c88010ac0 con 0x7f3c9c102790 2024-09-19T06:48:31.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.518+0000 7f3c92a00640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f3c800a3240 0x7f3c800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:31.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.519+0000 7f3c92a00640 1 -- 172.21.15.114:0/758189518 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7f3c880cf820 con 0x7f3c9c102790 2024-09-19T06:48:31.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.519+0000 7f3c99e00640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f3c800a3240 0x7f3c800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:31.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.519+0000 7f3c99e00640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f3c800a3240 0x7f3c800a5700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009230 tx=0x7f3c8c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:31.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.523+0000 7f3c92a00640 1 -- 172.21.15.114:0/758189518 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f3c8809c2f0 con 0x7f3c9c102790 2024-09-19T06:48:31.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.745+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c9c1a74c0 con 0x7f3c9c102790 2024-09-19T06:48:31.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.746+0000 7f3c92a00640 1 -- 172.21.15.114:0/758189518 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3094 (secure 0 0 0) 0x7f3c880a05f0 con 0x7f3c9c102790 2024-09-19T06:48:31.749 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:31.749 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":4,"btime":"2024-09-19T06:48:31:127492+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-19T06:48:31.749 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 4 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f3c800a3240 msgr2=0x7f3c800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f3c800a3240 0x7f3c800a5700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009230 tx=0x7f3c8c0058d0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c102790 msgr2=0x7f3c9c19ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c102790 0x7f3c9c19ab50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800e9f0 tx=0x7f3c8800eeb0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 shutdown_connections 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f3c800a3240 0x7f3c800a5700 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c9c106f10 0x7f3c9c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9c102790 0x7f3c9c19ab50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 --2- 172.21.15.114:0/758189518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c9c101de0 0x7f3c9c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 >> 172.21.15.114:0/758189518 conn(0x7f3c9c0fd8c0 msgr2=0x7f3c9c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 shutdown_connections 2024-09-19T06:48:31.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:31.748+0000 7f3ca1200640 1 -- 172.21.15.114:0/758189518 wait complete. 2024-09-19T06:48:32.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:32.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:32.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:32.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: Deploying daemon mds.d on smithi114 2024-09-19T06:48:32.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:32.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:boot 2024-09-19T06:48:32.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: fsmap 3 up:standby 2024-09-19T06:48:32.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:31 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: Deploying daemon mds.d on smithi114 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:32.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:boot 2024-09-19T06:48:32.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: fsmap 3 up:standby 2024-09-19T06:48:32.131 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: Deploying daemon mds.d on smithi114 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:boot 2024-09-19T06:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: fsmap 3 up:standby 2024-09-19T06:48:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-19T06:48:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:32 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/758189518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:33.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:32 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/758189518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:32 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/758189518' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:33.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:33 smithi114 ceph-mon[30820]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:34.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:33 smithi118 ceph-mon[35527]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:33 smithi116 ceph-mon[35411]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:34.496 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 systemd[1]: Starting Ceph mds.d for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:48:34.768 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 podman[70397]: 2024-09-19 06:48:34.399265346 +0000 UTC m=+0.027026874 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:48:34.769 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 podman[70397]: 2024-09-19 06:48:34.540009399 +0000 UTC m=+0.167770917 container create b0213f5b7fabb4c569859de495d15e7fa617abacb9cecfc1a86c88ae054e2abe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-d, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-19T06:48:34.769 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 podman[70397]: 2024-09-19 06:48:34.767863503 +0000 UTC m=+0.395625029 container init b0213f5b7fabb4c569859de495d15e7fa617abacb9cecfc1a86c88ae054e2abe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-d, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-19T06:48:34.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:35.046 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 podman[70397]: 2024-09-19 06:48:34.77446158 +0000 UTC m=+0.402223106 container start b0213f5b7fabb4c569859de495d15e7fa617abacb9cecfc1a86c88ae054e2abe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-d, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-19T06:48:35.046 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 ceph-mds[70415]: -- 172.21.15.114:0/3002141705 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b893a73860 con 0x55b893a50800 2024-09-19T06:48:35.046 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-d[70411]: starting mds.d at 2024-09-19T06:48:35.046 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 bash[70397]: b0213f5b7fabb4c569859de495d15e7fa617abacb9cecfc1a86c88ae054e2abe 2024-09-19T06:48:35.046 INFO:journalctl@ceph.mds.d.smithi114.stdout:Sep 19 06:48:34 smithi114 systemd[1]: Started Ceph mds.d for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:35.366 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6836/1558253171,v1:172.21.15.114:6837/1558253171] up:boot 2024-09-19T06:48:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: fsmap 4 up:standby 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: Deploying daemon mds.e on smithi116 2024-09-19T06:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:35 smithi116 ceph-mon[35411]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:35.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 -- 172.21.15.114:0/721296730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f60102b90 msgr2=0x7f7f60102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 --2- 172.21.15.114:0/721296730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f60102b90 0x7f7f60102f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009900 tx=0x7f7f5402a480 comp rx=0 tx=0).stop 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 -- 172.21.15.114:0/721296730 shutdown_connections 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 --2- 172.21.15.114:0/721296730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f6010bfc0 0x7f7f6010e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 --2- 172.21.15.114:0/721296730 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f601034b0 0x7f7f6010b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 --2- 172.21.15.114:0/721296730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f60102b90 0x7f7f60102f70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 -- 172.21.15.114:0/721296730 >> 172.21.15.114:0/721296730 conn(0x7f7f600fc610 msgr2=0x7f7f600fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 -- 172.21.15.114:0/721296730 shutdown_connections 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.941+0000 7f7f66200640 1 -- 172.21.15.114:0/721296730 wait complete. 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.942+0000 7f7f66200640 1 Processor -- start 2024-09-19T06:48:35.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.942+0000 7f7f66200640 1 -- start start 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f66200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60102b90 0x7f7f6019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f66200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f601034b0 0x7f7f6019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f66200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f6010bfc0 0x7f7f6019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f66200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60111e10 con 0x7f7f60102b90 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f66200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60111c90 con 0x7f7f601034b0 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f66200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60111f90 con 0x7f7f6010bfc0 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f64e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60102b90 0x7f7f6019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f64e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60102b90 0x7f7f6019a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38104/0 (socket says 172.21.15.114:38104) 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f64e00640 1 -- 172.21.15.114:0/626737268 learned_addr learned my addr 172.21.15.114:0/626737268 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f5fe00640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f601034b0 0x7f7f6019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:35.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f65800640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f6010bfc0 0x7f7f6019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f64e00640 1 -- 172.21.15.114:0/626737268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f6010bfc0 msgr2=0x7f7f6019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f64e00640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f6010bfc0 0x7f7f6019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f64e00640 1 -- 172.21.15.114:0/626737268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f601034b0 msgr2=0x7f7f6019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f64e00640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f601034b0 0x7f7f6019aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f64e00640 1 -- 172.21.15.114:0/626737268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f6019f480 con 0x7f7f60102b90 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.943+0000 7f7f5fe00640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f601034b0 0x7f7f6019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.944+0000 7f7f64e00640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60102b90 0x7f7f6019a4d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f54004770 tx=0x7f7f54003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.944+0000 7f7f5d600640 1 -- 172.21.15.114:0/626737268 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5403f070 con 0x7f7f60102b90 2024-09-19T06:48:35.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.944+0000 7f7f5d600640 1 -- 172.21.15.114:0/626737268 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f54031ca0 con 0x7f7f60102b90 2024-09-19T06:48:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.944+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f6019f710 con 0x7f7f60102b90 2024-09-19T06:48:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.944+0000 7f7f5d600640 1 -- 172.21.15.114:0/626737268 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54035030 con 0x7f7f60102b90 2024-09-19T06:48:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.944+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f601a7050 con 0x7f7f60102b90 2024-09-19T06:48:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.945+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f60102f70 con 0x7f7f60102b90 2024-09-19T06:48:35.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.947+0000 7f7f5d600640 1 -- 172.21.15.114:0/626737268 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f7f54042070 con 0x7f7f60102b90 2024-09-19T06:48:35.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.949+0000 7f7f5d600640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7f480a32c0 0x7f7f480a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:35.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.949+0000 7f7f5d600640 1 -- 172.21.15.114:0/626737268 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7f7f540efb90 con 0x7f7f60102b90 2024-09-19T06:48:35.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.950+0000 7f7f5fe00640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7f480a32c0 0x7f7f480a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:35.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.950+0000 7f7f5fe00640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7f480a32c0 0x7f7f480a5780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7f6010e910 tx=0x7f7f4c00b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:35.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:35.954+0000 7f7f5d600640 1 -- 172.21.15.114:0/626737268 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f7f540bccb0 con 0x7f7f60102b90 2024-09-19T06:48:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6836/1558253171,v1:172.21.15.114:6837/1558253171] up:boot 2024-09-19T06:48:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: fsmap 4 up:standby 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: Deploying daemon mds.e on smithi116 2024-09-19T06:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:35 smithi114 ceph-mon[30820]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:36.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.167+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f601a7220 con 0x7f7f60102b90 2024-09-19T06:48:36.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.168+0000 7f7f5d600640 1 -- 172.21.15.114:0/626737268 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3922 (secure 0 0 0) 0x7f7f540bc5b0 con 0x7f7f60102b90 2024-09-19T06:48:36.169 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:36.169 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":5,"btime":"2024-09-19T06:48:34:871110+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-19T06:48:36.170 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 5 2024-09-19T06:48:36.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7f480a32c0 msgr2=0x7f7f480a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7f480a32c0 0x7f7f480a5780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7f6010e910 tx=0x7f7f4c00b2e0 comp rx=0 tx=0).stop 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60102b90 msgr2=0x7f7f6019a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60102b90 0x7f7f6019a4d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f54004770 tx=0x7f7f54003c10 comp rx=0 tx=0).stop 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 shutdown_connections 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f7f480a32c0 0x7f7f480a5780 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f6010bfc0 0x7f7f6019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f601034b0 0x7f7f6019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 --2- 172.21.15.114:0/626737268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f60102b90 0x7f7f6019a4d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.170+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 >> 172.21.15.114:0/626737268 conn(0x7f7f600fc610 msgr2=0x7f7f600fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.171+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 shutdown_connections 2024-09-19T06:48:36.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:36.171+0000 7f7f66200640 1 -- 172.21.15.114:0/626737268 wait complete. 2024-09-19T06:48:36.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.114:6836/1558253171,v1:172.21.15.114:6837/1558253171] up:boot 2024-09-19T06:48:36.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: fsmap 4 up:standby 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: Deploying daemon mds.e on smithi116 2024-09-19T06:48:36.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:35 smithi118 ceph-mon[35527]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:37.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:36 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/626737268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:37.309 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:36 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/626737268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:36 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/626737268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:37.678 INFO:journalctl@ceph.mds.e.smithi116.stdout:Sep 19 06:48:37 smithi116 systemd[1]: Started Ceph mds.e for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:38.238 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:37 smithi118 ceph-mon[35527]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:38.239 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.239 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.239 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.239 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:38.239 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:38.239 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:37 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:38.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:37 smithi114 ceph-mon[30820]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:38.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:38.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:38.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:37 smithi116 ceph-mon[35411]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:38.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:39.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:38 smithi118 ceph-mon[35527]: Deploying daemon mds.f on smithi118 2024-09-19T06:48:39.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:38 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.116:6834/2147208941,v1:172.21.15.116:6835/2147208941] up:boot 2024-09-19T06:48:39.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:38 smithi118 ceph-mon[35527]: fsmap 5 up:standby 2024-09-19T06:48:39.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:38 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T06:48:39.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:38 smithi114 ceph-mon[30820]: Deploying daemon mds.f on smithi118 2024-09-19T06:48:39.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:38 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6834/2147208941,v1:172.21.15.116:6835/2147208941] up:boot 2024-09-19T06:48:39.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:38 smithi114 ceph-mon[30820]: fsmap 5 up:standby 2024-09-19T06:48:39.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T06:48:39.401 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:38 smithi116 ceph-mon[35411]: Deploying daemon mds.f on smithi118 2024-09-19T06:48:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:38 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6834/2147208941,v1:172.21.15.116:6835/2147208941] up:boot 2024-09-19T06:48:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:38 smithi116 ceph-mon[35411]: fsmap 5 up:standby 2024-09-19T06:48:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-19T06:48:39.685 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:39 smithi118 systemd[1]: Starting Ceph mds.f for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:48:39.938 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:39 smithi118 podman[65397]: 2024-09-19 06:48:39.587592239 +0000 UTC m=+0.015603006 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:48:39.938 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:39 smithi118 podman[65397]: 2024-09-19 06:48:39.709308159 +0000 UTC m=+0.137318921 container create 1f75ea9ff44ca9c7df90a965df21c90cac0afe9e1c90459485652908d7383835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-f, maintainer=Guillaume Abrioux , RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T06:48:39.938 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:39 smithi118 podman[65397]: 2024-09-19 06:48:39.937152975 +0000 UTC m=+0.365163741 container init 1f75ea9ff44ca9c7df90a965df21c90cac0afe9e1c90459485652908d7383835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-f, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-19T06:48:40.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 -- 172.21.15.114:0/2474854295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 msgr2=0x7f978810dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:40.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 --2- 172.21.15.114:0/2474854295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 0x7f978810dd20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9770009900 tx=0x7f977002a480 comp rx=0 tx=0).stop 2024-09-19T06:48:40.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 -- 172.21.15.114:0/2474854295 shutdown_connections 2024-09-19T06:48:40.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 --2- 172.21.15.114:0/2474854295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f978810e260 0x7f9788110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 --2- 172.21.15.114:0/2474854295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 0x7f978810dd20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 --2- 172.21.15.114:0/2474854295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97880745c0 0x7f97880749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 -- 172.21.15.114:0/2474854295 >> 172.21.15.114:0/2474854295 conn(0x7f97880fd740 msgr2=0x7f97880ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:40.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 -- 172.21.15.114:0/2474854295 shutdown_connections 2024-09-19T06:48:40.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.014+0000 7f978e400640 1 -- 172.21.15.114:0/2474854295 wait complete. 2024-09-19T06:48:40.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.015+0000 7f978e400640 1 Processor -- start 2024-09-19T06:48:40.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.015+0000 7f978e400640 1 -- start start 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.015+0000 7f978e400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97880745c0 0x7f978819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.015+0000 7f978e400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 0x7f978819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.015+0000 7f978e400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978810e260 0x7f978819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.015+0000 7f978e400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9788114000 con 0x7f9788074ee0 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.015+0000 7f978e400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9788113e80 con 0x7f978810e260 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.015+0000 7f978e400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9788114180 con 0x7f97880745c0 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f978d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97880745c0 0x7f978819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f977fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 0x7f978819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:40.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f977fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 0x7f978819a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33674/0 (socket says 172.21.15.114:33674) 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f978da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978810e260 0x7f978819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f977fe00640 1 -- 172.21.15.114:0/2180600039 learned_addr learned my addr 172.21.15.114:0/2180600039 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f978d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97880745c0 0x7f978819a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:60902/0 (socket says 172.21.15.114:60902) 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f978da00640 1 -- 172.21.15.114:0/2180600039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97880745c0 msgr2=0x7f978819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f978da00640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97880745c0 0x7f978819a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f978da00640 1 -- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 msgr2=0x7f978819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f978da00640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 0x7f978819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f978da00640 1 -- 172.21.15.114:0/2180600039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978819f450 con 0x7f978810e260 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.016+0000 7f977fe00640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 0x7f978819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.017+0000 7f978da00640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978810e260 0x7f978819ed70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9780005fd0 tx=0x7f978000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.017+0000 7f9786000640 1 -- 172.21.15.114:0/2180600039 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978000cdf0 con 0x7f978810e260 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.017+0000 7f9786000640 1 -- 172.21.15.114:0/2180600039 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9780005020 con 0x7f978810e260 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.017+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978819f740 con 0x7f978810e260 2024-09-19T06:48:40.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.017+0000 7f9786000640 1 -- 172.21.15.114:0/2180600039 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97800106d0 con 0x7f978810e260 2024-09-19T06:48:40.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.017+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97881a7020 con 0x7f978810e260 2024-09-19T06:48:40.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.018+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97880749a0 con 0x7f978810e260 2024-09-19T06:48:40.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.020+0000 7f9786000640 1 -- 172.21.15.114:0/2180600039 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f97800054d0 con 0x7f978810e260 2024-09-19T06:48:40.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.021+0000 7f9786000640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f976c0a32c0 0x7f976c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:40.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.021+0000 7f9786000640 1 -- 172.21.15.114:0/2180600039 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7f97800cfad0 con 0x7f978810e260 2024-09-19T06:48:40.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.022+0000 7f978d000640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f976c0a32c0 0x7f976c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:40.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.022+0000 7f978d000640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f976c0a32c0 0x7f976c0a5780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9778009bd0 tx=0x7f9778005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:40.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.025+0000 7f9786000640 1 -- 172.21.15.114:0/2180600039 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f978009cbf0 con 0x7f978810e260 2024-09-19T06:48:40.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:39 smithi118 ceph-mon[35527]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:40.256 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:39 smithi118 podman[65397]: 2024-09-19 06:48:39.940202873 +0000 UTC m=+0.368213637 container start 1f75ea9ff44ca9c7df90a965df21c90cac0afe9e1c90459485652908d7383835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-f, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-19T06:48:40.256 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:39 smithi118 ceph-mds[65416]: -- 172.21.15.118:0/193857418 <== mon.1 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f03d4d860 con 0x559f03d2b400 2024-09-19T06:48:40.257 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:39 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-f[65412]: starting mds.f at 2024-09-19T06:48:40.257 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:40 smithi118 bash[65397]: 1f75ea9ff44ca9c7df90a965df21c90cac0afe9e1c90459485652908d7383835 2024-09-19T06:48:40.257 INFO:journalctl@ceph.mds.f.smithi118.stdout:Sep 19 06:48:40 smithi118 systemd[1]: Started Ceph mds.f for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:40.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.258+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f978802d7c0 con 0x7f978810e260 2024-09-19T06:48:40.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:39 smithi114 ceph-mon[30820]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:40.260 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.259+0000 7f9786000640 1 -- 172.21.15.114:0/2180600039 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5575 (secure 0 0 0) 0x7f978009c4f0 con 0x7f978810e260 2024-09-19T06:48:40.261 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:40.261 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":7,"btime":"2024-09-19T06:48:39:970444+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-09-19T06:48:40.261 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 7 2024-09-19T06:48:40.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.261+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f976c0a32c0 msgr2=0x7f976c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:40.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.261+0000 7f978e400640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f976c0a32c0 0x7f976c0a5780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9778009bd0 tx=0x7f9778005f90 comp rx=0 tx=0).stop 2024-09-19T06:48:40.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.261+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978810e260 msgr2=0x7f978819ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:40.263 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.261+0000 7f978e400640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978810e260 0x7f978819ed70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9780005fd0 tx=0x7f978000ef10 comp rx=0 tx=0).stop 2024-09-19T06:48:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.262+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 shutdown_connections 2024-09-19T06:48:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.262+0000 7f978e400640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f976c0a32c0 0x7f976c0a5780 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.262+0000 7f978e400640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f978810e260 0x7f978819ed70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.262+0000 7f978e400640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9788074ee0 0x7f978819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.262+0000 7f978e400640 1 --2- 172.21.15.114:0/2180600039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97880745c0 0x7f978819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.262+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 >> 172.21.15.114:0/2180600039 conn(0x7f97880fd740 msgr2=0x7f97880fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.262+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 shutdown_connections 2024-09-19T06:48:40.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:40.262+0000 7f978e400640 1 -- 172.21.15.114:0/2180600039 wait complete. 2024-09-19T06:48:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:39 smithi116 ceph-mon[35411]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:41.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] up:boot 2024-09-19T06:48:41.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: fsmap 6 up:standby 2024-09-19T06:48:41.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T06:48:41.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:41.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:41.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:41.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: Deploying daemon mds.c on smithi118 2024-09-19T06:48:41.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:40 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2180600039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] up:boot 2024-09-19T06:48:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: fsmap 6 up:standby 2024-09-19T06:48:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T06:48:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: Deploying daemon mds.c on smithi118 2024-09-19T06:48:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:40 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2180600039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] up:boot 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: fsmap 6 up:standby 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: Deploying daemon mds.c on smithi118 2024-09-19T06:48:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:40 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2180600039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:42.240 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:41 smithi118 ceph-mon[35527]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:42.241 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 systemd[1]: Starting Ceph mds.c for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:48:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:41 smithi114 ceph-mon[30820]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:41 smithi116 ceph-mon[35411]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:42.506 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 podman[65697]: 2024-09-19 06:48:42.142038681 +0000 UTC m=+0.015451899 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:48:42.506 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 podman[65697]: 2024-09-19 06:48:42.264354271 +0000 UTC m=+0.137767476 container create 499bdf3469e0c4af238c6fea207cae922baf6b92edb41f1a997b3bfb7d9fea4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-c, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-19T06:48:43.006 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 podman[65697]: 2024-09-19 06:48:42.542022782 +0000 UTC m=+0.415435987 container init 499bdf3469e0c4af238c6fea207cae922baf6b92edb41f1a997b3bfb7d9fea4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-19T06:48:43.007 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 podman[65697]: 2024-09-19 06:48:42.544836975 +0000 UTC m=+0.418250180 container start 499bdf3469e0c4af238c6fea207cae922baf6b92edb41f1a997b3bfb7d9fea4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-c, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-19T06:48:43.007 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 ceph-mds[65716]: -- 172.21.15.118:0/2742311440 <== mon.1 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55efd34f5860 con 0x55efd34d3400 2024-09-19T06:48:43.007 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-c[65712]: starting mds.c at 2024-09-19T06:48:43.007 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 bash[65697]: 499bdf3469e0c4af238c6fea207cae922baf6b92edb41f1a997b3bfb7d9fea4c 2024-09-19T06:48:43.007 INFO:journalctl@ceph.mds.c.smithi118.stdout:Sep 19 06:48:42 smithi118 systemd[1]: Started Ceph mds.c for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:43.008 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:43.385 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:43.866 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:43.866 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:43.866 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:43.866 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:43.866 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:43.867 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:43.867 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: Deploying daemon mds.h on smithi116 2024-09-19T06:48:43.867 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] up:boot 2024-09-19T06:48:43.867 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: fsmap 7 up:standby 2024-09-19T06:48:43.867 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T06:48:43.867 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:43 smithi114 ceph-mon[30820]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:43.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 -- 172.21.15.114:0/1707743560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 msgr2=0x7fd7001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:43.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 --2- 172.21.15.114:0/1707743560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 0x7fd7001067e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009ca0 tx=0x7fd6e802a8c0 comp rx=0 tx=0).stop 2024-09-19T06:48:43.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 -- 172.21.15.114:0/1707743560 shutdown_connections 2024-09-19T06:48:43.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 --2- 172.21.15.114:0/1707743560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700106f10 0x7fd70010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:43.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 --2- 172.21.15.114:0/1707743560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 0x7fd7001067e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:43.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 --2- 172.21.15.114:0/1707743560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd700101de0 0x7fd7001021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:43.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 -- 172.21.15.114:0/1707743560 >> 172.21.15.114:0/1707743560 conn(0x7fd7000fd8c0 msgr2=0x7fd7000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:43.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 -- 172.21.15.114:0/1707743560 shutdown_connections 2024-09-19T06:48:43.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.976+0000 7fd705c00640 1 -- 172.21.15.114:0/1707743560 wait complete. 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.977+0000 7fd705c00640 1 Processor -- start 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.977+0000 7fd705c00640 1 -- start start 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.977+0000 7fd705c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd700101de0 0x7fd70019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.977+0000 7fd705c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 0x7fd70019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.977+0000 7fd705c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700106f10 0x7fd70019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.977+0000 7fd705c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd700111f70 con 0x7fd700102790 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.977+0000 7fd705c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd700111df0 con 0x7fd700101de0 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.977+0000 7fd705c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd7001120f0 con 0x7fd700106f10 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 0x7fd70019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 0x7fd70019ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33688/0 (socket says 172.21.15.114:33688) 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 -- 172.21.15.114:0/648150396 learned_addr learned my addr 172.21.15.114:0/648150396 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:43.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6fea00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd700101de0 0x7fd70019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:43.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6ff400640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700106f10 0x7fd70019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 -- 172.21.15.114:0/648150396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700106f10 msgr2=0x7fd70019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700106f10 0x7fd70019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 -- 172.21.15.114:0/648150396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd700101de0 msgr2=0x7fd70019a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd700101de0 0x7fd70019a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 -- 172.21.15.114:0/648150396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70019f5e0 con 0x7fd700102790 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6ff400640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700106f10 0x7fd70019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.978+0000 7fd6f6000640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 0x7fd70019ab70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6e802ade0 tx=0x7fd6e80319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.979+0000 7fd6f7e00640 1 -- 172.21.15.114:0/648150396 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e802fa20 con 0x7fd700102790 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.979+0000 7fd6f7e00640 1 -- 172.21.15.114:0/648150396 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6e802fbc0 con 0x7fd700102790 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.979+0000 7fd6f7e00640 1 -- 172.21.15.114:0/648150396 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e80466e0 con 0x7fd700102790 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.979+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70019f870 con 0x7fd700102790 2024-09-19T06:48:43.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.979+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7001a71b0 con 0x7fd700102790 2024-09-19T06:48:43.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.980+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7001032c0 con 0x7fd700102790 2024-09-19T06:48:43.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.982+0000 7fd6f7e00640 1 -- 172.21.15.114:0/648150396 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fd6e8046940 con 0x7fd700102790 2024-09-19T06:48:43.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.983+0000 7fd6f7e00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd6e40a3390 0x7fd6e40a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:43.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.983+0000 7fd6f7e00640 1 -- 172.21.15.114:0/648150396 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7fd6e80f0800 con 0x7fd700102790 2024-09-19T06:48:43.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.984+0000 7fd6fea00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd6e40a3390 0x7fd6e40a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:43.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.984+0000 7fd6fea00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd6e40a3390 0x7fd6e40a5850 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0005d00 tx=0x7fd6f000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:43.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:43.988+0000 7fd6f7e00640 1 -- 172.21.15.114:0/648150396 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fd6e80bd920 con 0x7fd700102790 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: Deploying daemon mds.h on smithi116 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] up:boot 2024-09-19T06:48:44.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: fsmap 7 up:standby 2024-09-19T06:48:44.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T06:48:44.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:43 smithi118 ceph-mon[35527]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:44.167 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:44.167 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:44.167 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:44.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:44.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:44.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:44.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: Deploying daemon mds.h on smithi116 2024-09-19T06:48:44.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] up:boot 2024-09-19T06:48:44.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: fsmap 7 up:standby 2024-09-19T06:48:44.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-19T06:48:44.168 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:43 smithi116 ceph-mon[35411]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:44.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.198+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7001a7380 con 0x7fd700102790 2024-09-19T06:48:44.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.199+0000 7fd6f7e00640 1 -- 172.21.15.114:0/648150396 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6403 (secure 0 0 0) 0x7fd6e80bd220 con 0x7fd700102790 2024-09-19T06:48:44.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:44.201 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"btime":"2024-09-19T06:48:42:982798+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-19T06:48:44.201 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-09-19T06:48:44.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.201+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd6e40a3390 msgr2=0x7fd6e40a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:44.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.201+0000 7fd705c00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd6e40a3390 0x7fd6e40a5850 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0005d00 tx=0x7fd6f000b410 comp rx=0 tx=0).stop 2024-09-19T06:48:44.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.201+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 msgr2=0x7fd70019ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:44.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.201+0000 7fd705c00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 0x7fd70019ab70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd6e802ade0 tx=0x7fd6e80319f0 comp rx=0 tx=0).stop 2024-09-19T06:48:44.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.202+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 shutdown_connections 2024-09-19T06:48:44.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.202+0000 7fd705c00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd6e40a3390 0x7fd6e40a5850 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:44.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.202+0000 7fd705c00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd700106f10 0x7fd70019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:44.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.202+0000 7fd705c00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd700102790 0x7fd70019ab70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:44.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.202+0000 7fd705c00640 1 --2- 172.21.15.114:0/648150396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd700101de0 0x7fd70019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:44.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.202+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 >> 172.21.15.114:0/648150396 conn(0x7fd7000fd8c0 msgr2=0x7fd7000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:44.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.202+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 shutdown_connections 2024-09-19T06:48:44.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:44.202+0000 7fd705c00640 1 -- 172.21.15.114:0/648150396 wait complete. 2024-09-19T06:48:44.928 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:44 smithi116 systemd[1]: Starting Ceph mds.h for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:48:45.251 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:44 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/648150396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:45.252 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:44 smithi116 podman[64827]: 2024-09-19 06:48:44.851001809 +0000 UTC m=+0.016095504 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:48:45.252 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:44 smithi116 podman[64827]: 2024-09-19 06:48:44.973966473 +0000 UTC m=+0.139060164 container create b3c95259708bb517f3c4984e034bda1fd4f48d06aaa4fc7ddac3e2e11d85781e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-h, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-2f599a4, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-19T06:48:45.252 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:45 smithi116 podman[64827]: 2024-09-19 06:48:45.251629959 +0000 UTC m=+0.416723651 container init b3c95259708bb517f3c4984e034bda1fd4f48d06aaa4fc7ddac3e2e11d85781e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-h, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731) 2024-09-19T06:48:45.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:44 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/648150396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:44 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/648150396' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:45.678 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:45 smithi116 podman[64827]: 2024-09-19 06:48:45.254504811 +0000 UTC m=+0.419598504 container start b3c95259708bb517f3c4984e034bda1fd4f48d06aaa4fc7ddac3e2e11d85781e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-19T06:48:45.679 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-mds[64846]: -- 172.21.15.116:0/3058656739 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df42d23860 con 0x55df42d01400 2024-09-19T06:48:45.679 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-h[64842]: starting mds.h at 2024-09-19T06:48:45.679 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:45 smithi116 bash[64827]: b3c95259708bb517f3c4984e034bda1fd4f48d06aaa4fc7ddac3e2e11d85781e 2024-09-19T06:48:45.679 INFO:journalctl@ceph.mds.h.smithi116.stdout:Sep 19 06:48:45 smithi116 systemd[1]: Started Ceph mds.h for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:46.154 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:45 smithi118 ceph-mon[35527]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:46.154 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.154 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.155 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.155 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:46.155 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:46.155 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:45 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:45 smithi114 ceph-mon[30820]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-mon[35411]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:46.983 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:47.158 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:47 smithi118 ceph-mon[35527]: Deploying daemon mds.i on smithi118 2024-09-19T06:48:47.158 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:47 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] up:boot 2024-09-19T06:48:47.158 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:47 smithi118 ceph-mon[35527]: fsmap 8 up:standby 2024-09-19T06:48:47.158 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:47 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T06:48:47.359 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:47 smithi116 ceph-mon[35411]: Deploying daemon mds.i on smithi118 2024-09-19T06:48:47.360 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:47 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] up:boot 2024-09-19T06:48:47.360 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:47 smithi116 ceph-mon[35411]: fsmap 8 up:standby 2024-09-19T06:48:47.360 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T06:48:47.361 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:47 smithi114 ceph-mon[30820]: Deploying daemon mds.i on smithi118 2024-09-19T06:48:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:47 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] up:boot 2024-09-19T06:48:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:47 smithi114 ceph-mon[30820]: fsmap 8 up:standby 2024-09-19T06:48:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-19T06:48:47.506 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 systemd[1]: Starting Ceph mds.i for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:48:47.881 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 podman[65995]: 2024-09-19 06:48:47.482622877 +0000 UTC m=+0.015319713 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:48:47.881 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 podman[65995]: 2024-09-19 06:48:47.594818334 +0000 UTC m=+0.127515159 container create 8f43fef95b920c1ef333df77088d8156e54bb221dc4167fd0b9027cd394d8b57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-i, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-2f599a4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-19T06:48:47.881 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 podman[65995]: 2024-09-19 06:48:47.880846384 +0000 UTC m=+0.413543226 container init 8f43fef95b920c1ef333df77088d8156e54bb221dc4167fd0b9027cd394d8b57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-2f599a4, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-19T06:48:47.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 -- 172.21.15.114:0/861287782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 msgr2=0x7fbbd4110ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 --2- 172.21.15.114:0/861287782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 0x7fbbd4110ca0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009ca0 tx=0x7fbbcc02a8c0 comp rx=0 tx=0).stop 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 -- 172.21.15.114:0/861287782 shutdown_connections 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 --2- 172.21.15.114:0/861287782 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 0x7fbbd4110ca0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 --2- 172.21.15.114:0/861287782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd41049a0 0x7fbbd41089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 --2- 172.21.15.114:0/861287782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbd4103ff0 0x7fbbd41043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 -- 172.21.15.114:0/861287782 >> 172.21.15.114:0/861287782 conn(0x7fbbd40ffa90 msgr2=0x7fbbd4101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 -- 172.21.15.114:0/861287782 shutdown_connections 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.968+0000 7fbbdc400640 1 -- 172.21.15.114:0/861287782 wait complete. 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.969+0000 7fbbdc400640 1 Processor -- start 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.969+0000 7fbbdc400640 1 -- start start 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbdc400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbd4103ff0 0x7fbbd419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbdc400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd41049a0 0x7fbbd419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbdc400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 0x7fbbd419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbdc400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd4114180 con 0x7fbbd4109120 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbdc400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd4114000 con 0x7fbbd41049a0 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbdc400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd4114300 con 0x7fbbd4103ff0 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbda400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 0x7fbbd419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbda400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 0x7fbbd419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47054/0 (socket says 172.21.15.114:47054) 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbda400640 1 -- 172.21.15.114:0/3097274936 learned_addr learned my addr 172.21.15.114:0/3097274936 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbd9000640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd41049a0 0x7fbbd419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbd9a00640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbd4103ff0 0x7fbbd419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbda400640 1 -- 172.21.15.114:0/3097274936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbd4103ff0 msgr2=0x7fbbd419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbda400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbd4103ff0 0x7fbbd419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbda400640 1 -- 172.21.15.114:0/3097274936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd41049a0 msgr2=0x7fbbd419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbda400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd41049a0 0x7fbbd419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.970+0000 7fbbda400640 1 -- 172.21.15.114:0/3097274936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd419f5f0 con 0x7fbbd4109120 2024-09-19T06:48:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.971+0000 7fbbda400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 0x7fbbd419ef10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc005e30 tx=0x7fbbcc003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.971+0000 7fbbd2000640 1 -- 172.21.15.114:0/3097274936 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc031650 con 0x7fbbd4109120 2024-09-19T06:48:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.971+0000 7fbbd2000640 1 -- 172.21.15.114:0/3097274936 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbcc0317f0 con 0x7fbbd4109120 2024-09-19T06:48:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.971+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd419f880 con 0x7fbbd4109120 2024-09-19T06:48:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.971+0000 7fbbd2000640 1 -- 172.21.15.114:0/3097274936 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc02f9d0 con 0x7fbbd4109120 2024-09-19T06:48:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.971+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd41a71c0 con 0x7fbbd4109120 2024-09-19T06:48:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.972+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbd41043d0 con 0x7fbbd4109120 2024-09-19T06:48:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.974+0000 7fbbd2000640 1 -- 172.21.15.114:0/3097274936 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fbbcc02fbf0 con 0x7fbbd4109120 2024-09-19T06:48:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.975+0000 7fbbd2000640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbbbc0a3240 0x7fbbbc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.976+0000 7fbbd2000640 1 -- 172.21.15.114:0/3097274936 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7fbbcc0f0960 con 0x7fbbd4109120 2024-09-19T06:48:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.976+0000 7fbbd9a00640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbbbc0a3240 0x7fbbbc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.976+0000 7fbbd9a00640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbbbc0a3240 0x7fbbbc0a5700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80095f0 tx=0x7fbbc8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:47.979+0000 7fbbd2000640 1 -- 172.21.15.114:0/3097274936 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbbcc0bdba0 con 0x7fbbd4109120 2024-09-19T06:48:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:48 smithi114 ceph-mon[30820]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.223+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbd41a7390 con 0x7fbbd4109120 2024-09-19T06:48:48.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.224+0000 7fbbd2000640 1 -- 172.21.15.114:0/3097274936 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8058 (secure 0 0 0) 0x7fbbcc0bd4a0 con 0x7fbbd4109120 2024-09-19T06:48:48.226 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:48.226 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":10,"btime":"2024-09-19T06:48:48:001045+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-19T06:48:48.226 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 10 2024-09-19T06:48:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbbbc0a3240 msgr2=0x7fbbbc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbbbc0a3240 0x7fbbbc0a5700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80095f0 tx=0x7fbbc8005f90 comp rx=0 tx=0).stop 2024-09-19T06:48:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 msgr2=0x7fbbd419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:48.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 0x7fbbd419ef10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc005e30 tx=0x7fbbcc003d00 comp rx=0 tx=0).stop 2024-09-19T06:48:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 shutdown_connections 2024-09-19T06:48:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbbbc0a3240 0x7fbbbc0a5700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbbd4109120 0x7fbbd419ef10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbd41049a0 0x7fbbd419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 --2- 172.21.15.114:0/3097274936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbbd4103ff0 0x7fbbd419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 >> 172.21.15.114:0/3097274936 conn(0x7fbbd40ffa90 msgr2=0x7fbbd4100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 shutdown_connections 2024-09-19T06:48:48.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:48.228+0000 7fbbdc400640 1 -- 172.21.15.114:0/3097274936 wait complete. 2024-09-19T06:48:48.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:48 smithi118 ceph-mon[35527]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:48.256 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 podman[65995]: 2024-09-19 06:48:47.883757412 +0000 UTC m=+0.416454232 container start 8f43fef95b920c1ef333df77088d8156e54bb221dc4167fd0b9027cd394d8b57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-19T06:48:48.256 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 ceph-mds[66014]: -- 172.21.15.118:0/516644284 <== mon.1 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d8b2421860 con 0x55d8b23ff000 2024-09-19T06:48:48.256 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-i[66010]: starting mds.i at 2024-09-19T06:48:48.257 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 bash[65995]: 8f43fef95b920c1ef333df77088d8156e54bb221dc4167fd0b9027cd394d8b57 2024-09-19T06:48:48.257 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:48:47 smithi118 systemd[1]: Started Ceph mds.i for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:48 smithi116 ceph-mon[35411]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:49.381 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] up:boot 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: fsmap 9 up:standby 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: Deploying daemon mds.g on smithi114 2024-09-19T06:48:49.382 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:49 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3097274936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] up:boot 2024-09-19T06:48:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: fsmap 9 up:standby 2024-09-19T06:48:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T06:48:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: Deploying daemon mds.g on smithi114 2024-09-19T06:48:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:49 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3097274936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:49.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] up:boot 2024-09-19T06:48:49.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: fsmap 9 up:standby 2024-09-19T06:48:49.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-19T06:48:49.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:49.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:49.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:49.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:49.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: Deploying daemon mds.g on smithi114 2024-09-19T06:48:49.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:49 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3097274936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:50.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:50 smithi114 ceph-mon[30820]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:50 smithi116 ceph-mon[35411]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:50 smithi118 ceph-mon[35527]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:51.066 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:51.619 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 06:48:51 smithi114 podman[71176]: 2024-09-19 06:48:51.373020038 +0000 UTC m=+0.153951645 container create 9d0acfebd10c9ec74bb9ab6d9461d010c3c72423e6261b1b62b161d0a8b4ec52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=squid-2f599a4, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-09-19T06:48:51.620 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 06:48:51 smithi114 podman[71176]: 2024-09-19 06:48:51.617813657 +0000 UTC m=+0.398745264 container init 9d0acfebd10c9ec74bb9ab6d9461d010c3c72423e6261b1b62b161d0a8b4ec52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-2f599a4, GIT_CLEAN=True) 2024-09-19T06:48:51.889 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:51.916 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 06:48:51 smithi114 podman[71176]: 2024-09-19 06:48:51.624029613 +0000 UTC m=+0.404961220 container start 9d0acfebd10c9ec74bb9ab6d9461d010c3c72423e6261b1b62b161d0a8b4ec52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-2f599a4) 2024-09-19T06:48:51.916 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 06:48:51 smithi114 ceph-mds[71208]: -- 172.21.15.114:0/2425052233 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5574ba2b5860 con 0x5574ba293000 2024-09-19T06:48:51.917 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 06:48:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g[71204]: starting mds.g at 2024-09-19T06:48:51.917 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 06:48:51 smithi114 bash[71176]: 9d0acfebd10c9ec74bb9ab6d9461d010c3c72423e6261b1b62b161d0a8b4ec52 2024-09-19T06:48:51.917 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 06:48:51 smithi114 systemd[1]: Started Ceph mds.g for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:52.364 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:52 smithi114 ceph-mon[30820]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:52.364 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.364 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.365 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.365 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:52.365 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:52.365 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:52 smithi116 ceph-mon[35411]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:52.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 -- 172.21.15.114:0/1618105218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c1047e0 msgr2=0x7f389c108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 --2- 172.21.15.114:0/1618105218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c1047e0 0x7f389c108830 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3888009860 tx=0x7f388802a340 comp rx=0 tx=0).stop 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 -- 172.21.15.114:0/1618105218 shutdown_connections 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 --2- 172.21.15.114:0/1618105218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f389c108f60 0x7f389c110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 --2- 172.21.15.114:0/1618105218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c1047e0 0x7f389c108830 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 --2- 172.21.15.114:0/1618105218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f389c103e30 0x7f389c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 -- 172.21.15.114:0/1618105218 >> 172.21.15.114:0/1618105218 conn(0x7f389c0ff8d0 msgr2=0x7f389c101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 -- 172.21.15.114:0/1618105218 shutdown_connections 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 -- 172.21.15.114:0/1618105218 wait complete. 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 Processor -- start 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 -- start start 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.474+0000 7f389b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c103e30 0x7f389c19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:52.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f389c1047e0 0x7f389c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f389c108f60 0x7f389c19ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f389c113fc0 con 0x7f389c108f60 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f389c113e40 con 0x7f389c1047e0 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f389c114140 con 0x7f389c103e30 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f389c108f60 0x7f389c19ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f389c108f60 0x7f389c19ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47076/0 (socket says 172.21.15.114:47076) 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389a800640 1 -- 172.21.15.114:0/2321738711 learned_addr learned my addr 172.21.15.114:0/2321738711 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f3899e00640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c103e30 0x7f389c19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f3899400640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f389c1047e0 0x7f389c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389a800640 1 -- 172.21.15.114:0/2321738711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c103e30 msgr2=0x7f389c19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389a800640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c103e30 0x7f389c19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389a800640 1 -- 172.21.15.114:0/2321738711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f389c1047e0 msgr2=0x7f389c19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:52.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389a800640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f389c1047e0 0x7f389c19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.475+0000 7f389a800640 1 -- 172.21.15.114:0/2321738711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389c19f450 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.476+0000 7f389a800640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f389c108f60 0x7f389c19ed70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f389400efc0 tx=0x7f389400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.476+0000 7f3892a00640 1 -- 172.21.15.114:0/2321738711 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894003970 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.476+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389c19f740 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.476+0000 7f3892a00640 1 -- 172.21.15.114:0/2321738711 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3894004020 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.476+0000 7f3892a00640 1 -- 172.21.15.114:0/2321738711 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389400b9f0 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.476+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389c1a7020 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.477+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f389c104210 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.479+0000 7f3892a00640 1 -- 172.21.15.114:0/2321738711 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f389401d020 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.480+0000 7f3892a00640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f38800a3240 0x7f38800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.480+0000 7f3892a00640 1 -- 172.21.15.114:0/2321738711 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7f38940cf320 con 0x7f389c108f60 2024-09-19T06:48:52.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.480+0000 7f3899e00640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f38800a3240 0x7f38800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:52.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.481+0000 7f3899e00640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f38800a3240 0x7f38800a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f388c009bd0 tx=0x7f388c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:52.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.484+0000 7f3892a00640 1 -- 172.21.15.114:0/2321738711 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f389409c440 con 0x7f389c108f60 2024-09-19T06:48:52.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:52 smithi118 ceph-mon[35527]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:52.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:52.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:52.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:52.509 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:52.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.706+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f389c10c1a0 con 0x7f389c108f60 2024-09-19T06:48:52.708 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.707+0000 7f3892a00640 1 -- 172.21.15.114:0/2321738711 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8887 (secure 0 0 0) 0x7f389409bd40 con 0x7f389c108f60 2024-09-19T06:48:52.709 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:52.712 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":11,"btime":"2024-09-19T06:48:52:130874+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-19T06:48:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 11 2024-09-19T06:48:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f38800a3240 msgr2=0x7f38800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:52.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f38800a3240 0x7f38800a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f388c009bd0 tx=0x7f388c005f90 comp rx=0 tx=0).stop 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f389c108f60 msgr2=0x7f389c19ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f389c108f60 0x7f389c19ed70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f389400efc0 tx=0x7f389400c800 comp rx=0 tx=0).stop 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 shutdown_connections 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f38800a3240 0x7f38800a5700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f389c108f60 0x7f389c19ed70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f389c1047e0 0x7f389c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 --2- 172.21.15.114:0/2321738711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f389c103e30 0x7f389c19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 >> 172.21.15.114:0/2321738711 conn(0x7f389c0ff8d0 msgr2=0x7f389c100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 shutdown_connections 2024-09-19T06:48:52.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:52.710+0000 7f389b200640 1 -- 172.21.15.114:0/2321738711 wait complete. 2024-09-19T06:48:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:53 smithi114 ceph-mon[30820]: Deploying daemon mds.a on smithi114 2024-09-19T06:48:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:53 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:boot 2024-09-19T06:48:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:53 smithi114 ceph-mon[30820]: fsmap 10 up:standby 2024-09-19T06:48:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T06:48:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:53 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2321738711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:53 smithi116 ceph-mon[35411]: Deploying daemon mds.a on smithi114 2024-09-19T06:48:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:53 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:boot 2024-09-19T06:48:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:53 smithi116 ceph-mon[35411]: fsmap 10 up:standby 2024-09-19T06:48:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T06:48:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:53 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2321738711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:53.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:53 smithi118 ceph-mon[35527]: Deploying daemon mds.a on smithi114 2024-09-19T06:48:53.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:53 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:boot 2024-09-19T06:48:53.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:53 smithi118 ceph-mon[35527]: fsmap 10 up:standby 2024-09-19T06:48:53.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:53 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-19T06:48:53.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:53 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2321738711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:54 smithi114 ceph-mon[30820]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:54 smithi116 ceph-mon[35411]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:54.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:54 smithi118 ceph-mon[35527]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:55.482 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:55.917 INFO:journalctl@ceph.mds.a.smithi114.stdout:Sep 19 06:48:55 smithi114 podman[71586]: 2024-09-19 06:48:55.678249036 +0000 UTC m=+0.144582314 container create 8f1ede13d215cf5a67442a087a05d45191bd6e6cd41addf74dc13c05961bf87f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-2f599a4, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-19T06:48:56.191 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:48:56.331 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:56 smithi114 ceph-mon[30820]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:56.331 INFO:journalctl@ceph.mds.a.smithi114.stdout:Sep 19 06:48:55 smithi114 podman[71586]: 2024-09-19 06:48:55.922881054 +0000 UTC m=+0.389214327 container init 8f1ede13d215cf5a67442a087a05d45191bd6e6cd41addf74dc13c05961bf87f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-2f599a4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-19T06:48:56.331 INFO:journalctl@ceph.mds.a.smithi114.stdout:Sep 19 06:48:55 smithi114 podman[71586]: 2024-09-19 06:48:55.929514221 +0000 UTC m=+0.395847502 container start 8f1ede13d215cf5a67442a087a05d45191bd6e6cd41addf74dc13c05961bf87f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-2f599a4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-19T06:48:56.331 INFO:journalctl@ceph.mds.a.smithi114.stdout:Sep 19 06:48:55 smithi114 ceph-mds[71629]: -- 172.21.15.114:0/3333831946 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55586baf3860 con 0x55586bad1400 2024-09-19T06:48:56.331 INFO:journalctl@ceph.mds.a.smithi114.stdout:Sep 19 06:48:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-a[71616]: starting mds.a at 2024-09-19T06:48:56.331 INFO:journalctl@ceph.mds.a.smithi114.stdout:Sep 19 06:48:56 smithi114 bash[71586]: 8f1ede13d215cf5a67442a087a05d45191bd6e6cd41addf74dc13c05961bf87f 2024-09-19T06:48:56.331 INFO:journalctl@ceph.mds.a.smithi114.stdout:Sep 19 06:48:56 smithi114 systemd[1]: Started Ceph mds.a for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:56 smithi116 ceph-mon[35411]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:56 smithi118 ceph-mon[35527]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:56.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.762+0000 7f9cea800640 1 -- 172.21.15.114:0/3300939129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 msgr2=0x7f9ce41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:56.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.762+0000 7f9cea800640 1 --2- 172.21.15.114:0/3300939129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 0x7f9ce41067e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009900 tx=0x7f9ccc02a410 comp rx=0 tx=0).stop 2024-09-19T06:48:56.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.762+0000 7f9cea800640 1 -- 172.21.15.114:0/3300939129 shutdown_connections 2024-09-19T06:48:56.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.762+0000 7f9cea800640 1 --2- 172.21.15.114:0/3300939129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ce4106f10 0x7f9ce410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.762+0000 7f9cea800640 1 --2- 172.21.15.114:0/3300939129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 0x7f9ce41067e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.762+0000 7f9cea800640 1 --2- 172.21.15.114:0/3300939129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ce4101de0 0x7f9ce41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.762+0000 7f9cea800640 1 -- 172.21.15.114:0/3300939129 >> 172.21.15.114:0/3300939129 conn(0x7f9ce40fd8c0 msgr2=0x7f9ce40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:56.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.762+0000 7f9cea800640 1 -- 172.21.15.114:0/3300939129 shutdown_connections 2024-09-19T06:48:56.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.763+0000 7f9cea800640 1 -- 172.21.15.114:0/3300939129 wait complete. 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.763+0000 7f9cea800640 1 Processor -- start 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cea800640 1 -- start start 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cea800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ce4101de0 0x7f9ce419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cea800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 0x7f9ce419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cea800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ce4106f10 0x7f9ce419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cea800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce4111f70 con 0x7f9ce4102790 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cea800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce4111df0 con 0x7f9ce4106f10 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cea800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce41120f0 con 0x7f9ce4101de0 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 0x7f9ce419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cdbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 0x7f9ce419abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47096/0 (socket says 172.21.15.114:47096) 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9cdbe00640 1 -- 172.21.15.114:0/3871008514 learned_addr learned my addr 172.21.15.114:0/3871008514 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:48:56.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9ce3400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ce4101de0 0x7f9ce419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.764+0000 7f9ce3e00640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ce4106f10 0x7f9ce419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9cdbe00640 1 -- 172.21.15.114:0/3871008514 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ce4101de0 msgr2=0x7f9ce419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9cdbe00640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ce4101de0 0x7f9ce419a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9cdbe00640 1 -- 172.21.15.114:0/3871008514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ce4106f10 msgr2=0x7f9ce419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9cdbe00640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ce4106f10 0x7f9ce419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9cdbe00640 1 -- 172.21.15.114:0/3871008514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce419f640 con 0x7f9ce4102790 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9ce3e00640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ce4106f10 0x7f9ce419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9cdbe00640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 0x7f9ce419abd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc006010 tx=0x7f9ccc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9ce0c00640 1 -- 172.21.15.114:0/3871008514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc03f030 con 0x7f9ce4102790 2024-09-19T06:48:56.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9ce0c00640 1 -- 172.21.15.114:0/3871008514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ccc003ce0 con 0x7f9ce4102790 2024-09-19T06:48:56.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.765+0000 7f9ce0c00640 1 -- 172.21.15.114:0/3871008514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc035070 con 0x7f9ce4102790 2024-09-19T06:48:56.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.766+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce419f8d0 con 0x7f9ce4102790 2024-09-19T06:48:56.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.766+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce41a7210 con 0x7f9ce4102790 2024-09-19T06:48:56.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.767+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ce41021c0 con 0x7f9ce4102790 2024-09-19T06:48:56.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.769+0000 7f9ce0c00640 1 -- 172.21.15.114:0/3871008514 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f9ccc042030 con 0x7f9ce4102790 2024-09-19T06:48:56.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.770+0000 7f9ce0c00640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cc80a32c0 0x7f9cc80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:48:56.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.770+0000 7f9ce0c00640 1 -- 172.21.15.114:0/3871008514 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7f9ccc0efc00 con 0x7f9ce4102790 2024-09-19T06:48:56.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.770+0000 7f9ce3400640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cc80a32c0 0x7f9cc80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:48:56.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.771+0000 7f9ce3400640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cc80a32c0 0x7f9cc80a5780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009bd0 tx=0x7f9cd4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:48:56.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.772+0000 7f9ce0c00640 1 -- 172.21.15.114:0/3871008514 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9ccc0bc790 con 0x7f9ce4102790 2024-09-19T06:48:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.984+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ce419b9f0 con 0x7f9ce4102790 2024-09-19T06:48:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.984+0000 7f9ce0c00640 1 -- 172.21.15.114:0/3871008514 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9716 (secure 0 0 0) 0x7f9ce419b9f0 con 0x7f9ce4102790 2024-09-19T06:48:56.986 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:48:56.986 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":12,"btime":"2024-09-19T06:48:56:145226+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14520,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-19T06:48:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 12 2024-09-19T06:48:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cc80a32c0 msgr2=0x7f9cc80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cc80a32c0 0x7f9cc80a5780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4009bd0 tx=0x7f9cd4005f90 comp rx=0 tx=0).stop 2024-09-19T06:48:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 msgr2=0x7f9ce419abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:48:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 0x7f9ce419abd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc006010 tx=0x7f9ccc0042f0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 shutdown_connections 2024-09-19T06:48:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cc80a32c0 0x7f9cc80a5780 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ce4106f10 0x7f9ce419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ce4102790 0x7f9ce419abd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 --2- 172.21.15.114:0/3871008514 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9ce4101de0 0x7f9ce419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:48:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 >> 172.21.15.114:0/3871008514 conn(0x7f9ce40fd8c0 msgr2=0x7f9ce40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:48:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 shutdown_connections 2024-09-19T06:48:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:48:56.987+0000 7f9cea800640 1 -- 172.21.15.114:0/3871008514 wait complete. 2024-09-19T06:48:57.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6840/2459666351,v1:172.21.15.114:6841/2459666351] up:boot 2024-09-19T06:48:57.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: fsmap 11 up:standby 2024-09-19T06:48:57.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T06:48:57.390 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:57.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:57.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:57.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: Deploying daemon mds.b on smithi116 2024-09-19T06:48:57.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:57 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3871008514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6840/2459666351,v1:172.21.15.114:6841/2459666351] up:boot 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: fsmap 11 up:standby 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: Deploying daemon mds.b on smithi116 2024-09-19T06:48:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:57 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3871008514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.114:6840/2459666351,v1:172.21.15.114:6841/2459666351] up:boot 2024-09-19T06:48:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: fsmap 11 up:standby 2024-09-19T06:48:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-19T06:48:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-19T06:48:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-19T06:48:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:48:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: Deploying daemon mds.b on smithi116 2024-09-19T06:48:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:57 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3871008514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:48:58.179 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 systemd[1]: Starting Ceph mds.b for fface640-7651-11ef-bceb-c7b262605968... 2024-09-19T06:48:58.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:58 smithi116 ceph-mon[35411]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:58.465 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 podman[65130]: 2024-09-19 06:48:58.219514128 +0000 UTC m=+0.016338756 image pull d2baa855c067ff6a7470a2dcec37621dc44864d3a0d23b0aa8bba9bd386dfe3d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e 2024-09-19T06:48:58.466 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 podman[65130]: 2024-09-19 06:48:58.354283728 +0000 UTC m=+0.151108347 container create fdafe126329975997fb4d42fcf86e161579db2b59dc9568769f9a89f8036869f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=squid-2f599a4, io.buildah.version=1.37.2) 2024-09-19T06:48:58.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:58 smithi118 ceph-mon[35527]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:58 smithi114 ceph-mon[30820]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:58.928 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 podman[65130]: 2024-09-19 06:48:58.597313665 +0000 UTC m=+0.394138283 container init fdafe126329975997fb4d42fcf86e161579db2b59dc9568769f9a89f8036869f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-b, RELEASE=squid-2f599a4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-19T06:48:58.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 podman[65130]: 2024-09-19 06:48:58.600319784 +0000 UTC m=+0.397144412 container start fdafe126329975997fb4d42fcf86e161579db2b59dc9568769f9a89f8036869f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ea3a79400324e9054be6f7f19ded2753beeca1d15cbe0b9cab2e816ae9b5fd3e, name=ceph-fface640-7651-11ef-bceb-c7b262605968-mds-b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-2f599a4, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-19T06:48:58.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 ceph-mds[65150]: -- 172.21.15.116:0/1354027990 <== mon.2 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb8a18f860 con 0x55fb8a16c800 2024-09-19T06:48:58.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-b[65146]: starting mds.b at 2024-09-19T06:48:58.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 bash[65130]: fdafe126329975997fb4d42fcf86e161579db2b59dc9568769f9a89f8036869f 2024-09-19T06:48:58.929 INFO:journalctl@ceph.mds.b.smithi116.stdout:Sep 19 06:48:58 smithi116 systemd[1]: Started Ceph mds.b for fface640-7651-11ef-bceb-c7b262605968. 2024-09-19T06:48:59.839 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] up:boot 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: fsmap 12 up:standby 2024-09-19T06:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:48:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T06:49:00.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:49:00.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:00.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] up:boot 2024-09-19T06:49:00.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: fsmap 12 up:standby 2024-09-19T06:49:00.007 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:48:59 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T06:49:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:49:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] up:boot 2024-09-19T06:49:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: fsmap 12 up:standby 2024-09-19T06:49:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:48:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-19T06:49:00.401 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:01.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 -- 172.21.15.114:0/1639169233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc1049c0 msgr2=0x7fcabc108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:01.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 --2- 172.21.15.114:0/1639169233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc1049c0 0x7fcabc108a10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009900 tx=0x7fcaa402a410 comp rx=0 tx=0).stop 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 -- 172.21.15.114:0/1639169233 shutdown_connections 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 --2- 172.21.15.114:0/1639169233 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcabc109140 0x7fcabc110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 --2- 172.21.15.114:0/1639169233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc1049c0 0x7fcabc108a10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 --2- 172.21.15.114:0/1639169233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcabc104010 0x7fcabc1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 -- 172.21.15.114:0/1639169233 >> 172.21.15.114:0/1639169233 conn(0x7fcabc0ffa50 msgr2=0x7fcabc101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 -- 172.21.15.114:0/1639169233 shutdown_connections 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.109+0000 7fcac0e00640 1 -- 172.21.15.114:0/1639169233 wait complete. 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.110+0000 7fcac0e00640 1 Processor -- start 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcac0e00640 1 -- start start 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcac0e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcabc104010 0x7fcabc19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:01.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcac0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcabc1049c0 0x7fcabc19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcac0e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc109140 0x7fcabc19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcac0e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc114160 con 0x7fcabc1049c0 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcac0e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc113fe0 con 0x7fcabc109140 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcac0e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc1142e0 con 0x7fcabc104010 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcab9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcabc1049c0 0x7fcabc19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcab9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcabc1049c0 0x7fcabc19aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52054/0 (socket says 172.21.15.114:52054) 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcab9400640 1 -- 172.21.15.114:0/4127334404 learned_addr learned my addr 172.21.15.114:0/4127334404 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcaba800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc109140 0x7fcabc19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.111+0000 7fcab9e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcabc104010 0x7fcabc19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcaba800640 1 -- 172.21.15.114:0/4127334404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcabc104010 msgr2=0x7fcabc19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcaba800640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcabc104010 0x7fcabc19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcaba800640 1 -- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcabc1049c0 msgr2=0x7fcabc19aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:01.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcaba800640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcabc1049c0 0x7fcabc19aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcaba800640 1 -- 172.21.15.114:0/4127334404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcabc19f610 con 0x7fcabc109140 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcab9400640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcabc1049c0 0x7fcabc19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcaba800640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc109140 0x7fcabc19ef30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e970 tx=0x7fcab400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcab2a00640 1 -- 172.21.15.114:0/4127334404 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab400ccc0 con 0x7fcabc109140 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcab2a00640 1 -- 172.21.15.114:0/4127334404 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcab4005020 con 0x7fcabc109140 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcab2a00640 1 -- 172.21.15.114:0/4127334404 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab4004a00 con 0x7fcabc109140 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcabc19f900 con 0x7fcabc109140 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.112+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcabc1a71e0 con 0x7fcabc109140 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.113+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcabc1043f0 con 0x7fcabc109140 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.115+0000 7fcab2a00640 1 -- 172.21.15.114:0/4127334404 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fcab40051d0 con 0x7fcabc109140 2024-09-19T06:49:01.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.116+0000 7fcab2a00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcaa00a32c0 0x7fcaa00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:01.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.116+0000 7fcab2a00640 1 -- 172.21.15.114:0/4127334404 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7fcab40cf330 con 0x7fcabc109140 2024-09-19T06:49:01.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.116+0000 7fcab9e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcaa00a32c0 0x7fcaa00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:01.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.117+0000 7fcab9e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcaa00a32c0 0x7fcaa00a5780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcaac005fd0 tx=0x7fcaac009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:01.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.120+0000 7fcab2a00640 1 -- 172.21.15.114:0/4127334404 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcab409c450 con 0x7fcabc109140 2024-09-19T06:49:01.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.347+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcabc10c3e0 con 0x7fcabc109140 2024-09-19T06:49:01.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.347+0000 7fcab2a00640 1 -- 172.21.15.114:0/4127334404 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10545 (secure 0 0 0) 0x7fcab409bd50 con 0x7fcabc109140 2024-09-19T06:49:01.349 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:01.350 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":13,"btime":"2024-09-19T06:48:59:179189+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14520,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24538,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-19T06:49:01.350 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 13 2024-09-19T06:49:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcaa00a32c0 msgr2=0x7fcaa00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcaa00a32c0 0x7fcaa00a5780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcaac005fd0 tx=0x7fcaac009500 comp rx=0 tx=0).stop 2024-09-19T06:49:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc109140 msgr2=0x7fcabc19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc109140 0x7fcabc19ef30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e970 tx=0x7fcab400ee30 comp rx=0 tx=0).stop 2024-09-19T06:49:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 shutdown_connections 2024-09-19T06:49:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcaa00a32c0 0x7fcaa00a5780 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcabc109140 0x7fcabc19ef30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcabc1049c0 0x7fcabc19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 --2- 172.21.15.114:0/4127334404 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcabc104010 0x7fcabc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:01.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 >> 172.21.15.114:0/4127334404 conn(0x7fcabc0ffa50 msgr2=0x7fcabc100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:01.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 shutdown_connections 2024-09-19T06:49:01.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:01.350+0000 7fcac0e00640 1 -- 172.21.15.114:0/4127334404 wait complete. 2024-09-19T06:49:01.365 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.365 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.365 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:01 smithi114 ceph-mon[30820]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:01 smithi116 ceph-mon[35411]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:01 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:01 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:01 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:01 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:01 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:01.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:01 smithi118 ceph-mon[35527]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:02.182 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-09-19T06:49:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:02 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/4127334404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:02.572 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:02 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/4127334404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:02 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/4127334404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.189+0000 7f33b1800640 1 -- 172.21.15.114:0/3348474667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33ac1019c0 msgr2=0x7f33ac101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.189+0000 7f33b1800640 1 --2- 172.21.15.114:0/3348474667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33ac1019c0 0x7f33ac101e20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3394009930 tx=0x7f339402a420 comp rx=0 tx=0).stop 2024-09-19T06:49:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.190+0000 7f33b1800640 1 -- 172.21.15.114:0/3348474667 shutdown_connections 2024-09-19T06:49:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.190+0000 7f33b1800640 1 --2- 172.21.15.114:0/3348474667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ac102360 0x7f33ac10e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.190+0000 7f33b1800640 1 --2- 172.21.15.114:0/3348474667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33ac1019c0 0x7f33ac101e20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.190+0000 7f33b1800640 1 --2- 172.21.15.114:0/3348474667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33ac1079c0 0x7f33ac107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.190+0000 7f33b1800640 1 -- 172.21.15.114:0/3348474667 >> 172.21.15.114:0/3348474667 conn(0x7f33ac0fd740 msgr2=0x7f33ac0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.190+0000 7f33b1800640 1 -- 172.21.15.114:0/3348474667 shutdown_connections 2024-09-19T06:49:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.190+0000 7f33b1800640 1 -- 172.21.15.114:0/3348474667 wait complete. 2024-09-19T06:49:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.191+0000 7f33b1800640 1 Processor -- start 2024-09-19T06:49:03.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.191+0000 7f33b1800640 1 -- start start 2024-09-19T06:49:03.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.191+0000 7f33b1800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ac1019c0 0x7f33ac0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:03.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.191+0000 7f33b1800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33ac102360 0x7f33ac074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:03.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.191+0000 7f33b1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33ac1079c0 0x7f33ac074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:03.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33b1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac111e10 con 0x7f33ac1079c0 2024-09-19T06:49:03.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33b1800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac111c90 con 0x7f33ac1019c0 2024-09-19T06:49:03.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33b1800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac111f90 con 0x7f33ac102360 2024-09-19T06:49:03.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33a3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33ac102360 0x7f33ac074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ac1019c0 0x7f33ac0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33a3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33ac102360 0x7f33ac074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:41044/0 (socket says 172.21.15.114:41044) 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33a3e00640 1 -- 172.21.15.114:0/1613091631 learned_addr learned my addr 172.21.15.114:0/1613091631 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33b0e00640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33ac1079c0 0x7f33ac074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33abe00640 1 -- 172.21.15.114:0/1613091631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33ac102360 msgr2=0x7f33ac074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33abe00640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33ac102360 0x7f33ac074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33abe00640 1 -- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33ac1079c0 msgr2=0x7f33ac074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33abe00640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33ac1079c0 0x7f33ac074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33abe00640 1 -- 172.21.15.114:0/1613091631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac1a6830 con 0x7f33ac1019c0 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.192+0000 7f33abe00640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ac1019c0 0x7f33ac0744f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f339c00e9f0 tx=0x7f339c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.193+0000 7f33a9600640 1 -- 172.21.15.114:0/1613091631 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c00ccc0 con 0x7f33ac1019c0 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.193+0000 7f33a9600640 1 -- 172.21.15.114:0/1613091631 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f339c005020 con 0x7f33ac1019c0 2024-09-19T06:49:03.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.193+0000 7f33a9600640 1 -- 172.21.15.114:0/1613091631 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c0049e0 con 0x7f33ac1019c0 2024-09-19T06:49:03.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.193+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33ac1a6b20 con 0x7f33ac1019c0 2024-09-19T06:49:03.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.193+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33ac1a7010 con 0x7f33ac1019c0 2024-09-19T06:49:03.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.194+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33ac101e20 con 0x7f33ac1019c0 2024-09-19T06:49:03.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.196+0000 7f33a9600640 1 -- 172.21.15.114:0/1613091631 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f339c005560 con 0x7f33ac1019c0 2024-09-19T06:49:03.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.197+0000 7f33a9600640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f33900a32c0 0x7f33900a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:03.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.198+0000 7f33a9600640 1 -- 172.21.15.114:0/1613091631 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7f339c0cf630 con 0x7f33ac1019c0 2024-09-19T06:49:03.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.198+0000 7f33a3e00640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f33900a32c0 0x7f33900a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:03.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.198+0000 7f33a3e00640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f33900a32c0 0x7f33900a5780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3394005c40 tx=0x7f3394005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:03.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.199+0000 7f33a9600640 1 -- 172.21.15.114:0/1613091631 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f339c0051d0 con 0x7f33ac1019c0 2024-09-19T06:49:03.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.388+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f33ac075a20 con 0x7f33ac1019c0 2024-09-19T06:49:03.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.388+0000 7f33a9600640 1 -- 172.21.15.114:0/1613091631 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) ==== 84+0+65 (secure 0 0 0) 0x7f339c09c750 con 0x7f33ac1019c0 2024-09-19T06:49:03.390 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:03.390 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-09-19T06:49:03.390 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-09-19T06:49:03.390 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-09-19T06:49:03.390 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-09-19T06:49:03.390 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-19T06:49:03.390 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-09-19T06:49:03.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f33900a32c0 msgr2=0x7f33900a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:03.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f33900a32c0 0x7f33900a5780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3394005c40 tx=0x7f3394005b50 comp rx=0 tx=0).stop 2024-09-19T06:49:03.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ac1019c0 msgr2=0x7f33ac0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ac1019c0 0x7f33ac0744f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f339c00e9f0 tx=0x7f339c00eeb0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 shutdown_connections 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f33900a32c0 0x7f33900a5780 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33ac1079c0 0x7f33ac074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f33ac102360 0x7f33ac074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 --2- 172.21.15.114:0/1613091631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33ac1019c0 0x7f33ac0744f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 >> 172.21.15.114:0/1613091631 conn(0x7f33ac0fd740 msgr2=0x7f33ac0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 shutdown_connections 2024-09-19T06:49:03.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:03.391+0000 7f33b1800640 1 -- 172.21.15.114:0/1613091631 wait complete. 2024-09-19T06:49:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:03 smithi114 ceph-mon[30820]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:49:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:49:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:03 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1613091631' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-19T06:49:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:49:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:03 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:03 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:03 smithi118 ceph-mon[35527]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:03 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:49:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:03 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:49:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:03 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:03 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1613091631' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-19T06:49:03.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:03 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:49:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:03 smithi116 ceph-mon[35411]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:49:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:49:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:03 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1613091631' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-19T06:49:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:49:04.157 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-19T06:49:04.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-19T06:49:05.167 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:05.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.747+0000 7ffa79000640 1 -- 172.21.15.114:0/4078840297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa74103e30 msgr2=0x7ffa74104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:05.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.747+0000 7ffa79000640 1 --2- 172.21.15.114:0/4078840297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa74103e30 0x7ffa74104210 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009860 tx=0x7ffa6402a340 comp rx=0 tx=0).stop 2024-09-19T06:49:05.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.747+0000 7ffa79000640 1 -- 172.21.15.114:0/4078840297 shutdown_connections 2024-09-19T06:49:05.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.747+0000 7ffa79000640 1 --2- 172.21.15.114:0/4078840297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa74108f60 0x7ffa74110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:05.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.747+0000 7ffa79000640 1 --2- 172.21.15.114:0/4078840297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa741047e0 0x7ffa74108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:05.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.747+0000 7ffa79000640 1 --2- 172.21.15.114:0/4078840297 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa74103e30 0x7ffa74104210 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:05.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.748+0000 7ffa79000640 1 -- 172.21.15.114:0/4078840297 >> 172.21.15.114:0/4078840297 conn(0x7ffa740ff8d0 msgr2=0x7ffa74101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:05.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.748+0000 7ffa79000640 1 -- 172.21.15.114:0/4078840297 shutdown_connections 2024-09-19T06:49:05.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.748+0000 7ffa79000640 1 -- 172.21.15.114:0/4078840297 wait complete. 2024-09-19T06:49:05.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa79000640 1 Processor -- start 2024-09-19T06:49:05.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa79000640 1 -- start start 2024-09-19T06:49:05.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa79000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa74103e30 0x7ffa7419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:05.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa79000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa741047e0 0x7ffa7419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:05.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa79000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa74108f60 0x7ffa7419edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:05.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa79000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74113f80 con 0x7ffa741047e0 2024-09-19T06:49:05.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa79000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74113e00 con 0x7ffa74103e30 2024-09-19T06:49:05.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa79000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ffa74114100 con 0x7ffa74108f60 2024-09-19T06:49:05.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa72a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa741047e0 0x7ffa7419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:05.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa72a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa741047e0 0x7ffa7419aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52110/0 (socket says 172.21.15.114:52110) 2024-09-19T06:49:05.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa72a00640 1 -- 172.21.15.114:0/993619625 learned_addr learned my addr 172.21.15.114:0/993619625 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:05.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.749+0000 7ffa73400640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa74103e30 0x7ffa7419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:05.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.750+0000 7ffa73e00640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa74108f60 0x7ffa7419edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.750+0000 7ffa73400640 1 -- 172.21.15.114:0/993619625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa74108f60 msgr2=0x7ffa7419edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.750+0000 7ffa73400640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa74108f60 0x7ffa7419edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.750+0000 7ffa73400640 1 -- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa741047e0 msgr2=0x7ffa7419aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.750+0000 7ffa73400640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa741047e0 0x7ffa7419aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.750+0000 7ffa73400640 1 -- 172.21.15.114:0/993619625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7419f4b0 con 0x7ffa74103e30 2024-09-19T06:49:05.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.750+0000 7ffa73400640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa74103e30 0x7ffa7419a500 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa64035040 tx=0x7ffa64033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.750+0000 7ffa6be00640 1 -- 172.21.15.114:0/993619625 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6403f020 con 0x7ffa74103e30 2024-09-19T06:49:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.751+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa7419f740 con 0x7ffa74103e30 2024-09-19T06:49:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.751+0000 7ffa6be00640 1 -- 172.21.15.114:0/993619625 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa64030550 con 0x7ffa74103e30 2024-09-19T06:49:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.751+0000 7ffa6be00640 1 -- 172.21.15.114:0/993619625 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64037040 con 0x7ffa74103e30 2024-09-19T06:49:05.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.751+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa741a7080 con 0x7ffa74103e30 2024-09-19T06:49:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.752+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa74104210 con 0x7ffa74103e30 2024-09-19T06:49:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.754+0000 7ffa6be00640 1 -- 172.21.15.114:0/993619625 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7ffa64042020 con 0x7ffa74103e30 2024-09-19T06:49:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.755+0000 7ffa6be00640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa580a3240 0x7ffa580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:05.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.755+0000 7ffa6be00640 1 -- 172.21.15.114:0/993619625 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7127+0+0 (secure 0 0 0) 0x7ffa640ef5f0 con 0x7ffa74103e30 2024-09-19T06:49:05.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.756+0000 7ffa72a00640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa580a3240 0x7ffa580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:05.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.756+0000 7ffa72a00640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa580a3240 0x7ffa580a5700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa740ffd80 tx=0x7ffa60009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:05.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.757+0000 7ffa6be00640 1 -- 172.21.15.114:0/993619625 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ffa64049020 con 0x7ffa74103e30 2024-09-19T06:49:05.879 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:05.879 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:05.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:05.947+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ffa7410c140 con 0x7ffa74103e30 2024-09-19T06:49:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:05 smithi114 ceph-mon[30820]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:05 smithi116 ceph-mon[35411]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:06.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:05 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:06.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:05 smithi118 ceph-mon[35527]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:06.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.888+0000 7ffa6be00640 1 -- 172.21.15.114:0/993619625 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) ==== 138+0+0 (secure 0 0 0) 0x7ffa640bc710 con 0x7ffa74103e30 2024-09-19T06:49:06.891 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-09-19T06:49:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa580a3240 msgr2=0x7ffa580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa580a3240 0x7ffa580a5700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa740ffd80 tx=0x7ffa60009260 comp rx=0 tx=0).stop 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa74103e30 msgr2=0x7ffa7419a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa74103e30 0x7ffa7419a500 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa64035040 tx=0x7ffa64033b60 comp rx=0 tx=0).stop 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 shutdown_connections 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ffa580a3240 0x7ffa580a5700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa74108f60 0x7ffa7419edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa741047e0 0x7ffa7419aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 --2- 172.21.15.114:0/993619625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa74103e30 0x7ffa7419a500 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 >> 172.21.15.114:0/993619625 conn(0x7ffa740ff8d0 msgr2=0x7ffa74100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 shutdown_connections 2024-09-19T06:49:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:06.893+0000 7ffa79000640 1 -- 172.21.15.114:0/993619625 wait complete. 2024-09-19T06:49:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:06 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/993619625' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T06:49:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:06 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T06:49:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:49:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:49:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:06 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/993619625' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T06:49:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:06 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T06:49:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:49:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:49:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:06 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/993619625' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T06:49:07.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:06 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-19T06:49:07.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:49:07.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:49:07.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:06 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:49:07.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-19T06:49:08.037 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:07 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-19T06:49:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:07 smithi114 ceph-mon[30820]: osdmap e80: 12 total, 12 up, 12 in 2024-09-19T06:49:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:07 smithi114 ceph-mon[30820]: pgmap v288: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:07 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-19T06:49:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:07 smithi116 ceph-mon[35411]: osdmap e80: 12 total, 12 up, 12 in 2024-09-19T06:49:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:07 smithi116 ceph-mon[35411]: pgmap v288: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:08.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:07 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-19T06:49:08.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:07 smithi118 ceph-mon[35527]: osdmap e80: 12 total, 12 up, 12 in 2024-09-19T06:49:08.257 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:07 smithi118 ceph-mon[35527]: pgmap v288: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 -- 172.21.15.114:0/4043210024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 msgr2=0x7f376c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 --2- 172.21.15.114:0/4043210024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 0x7f376c1067d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3754009900 tx=0x7f375402a410 comp rx=0 tx=0).stop 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 -- 172.21.15.114:0/4043210024 shutdown_connections 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 --2- 172.21.15.114:0/4043210024 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376c106f00 0x7f376c10ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 --2- 172.21.15.114:0/4043210024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 0x7f376c1067d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 --2- 172.21.15.114:0/4043210024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c101dd0 0x7f376c1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 -- 172.21.15.114:0/4043210024 >> 172.21.15.114:0/4043210024 conn(0x7f376c0fd8b0 msgr2=0x7f376c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 -- 172.21.15.114:0/4043210024 shutdown_connections 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.625+0000 7f3771600640 1 -- 172.21.15.114:0/4043210024 wait complete. 2024-09-19T06:49:08.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.626+0000 7f3771600640 1 Processor -- start 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.626+0000 7f3771600640 1 -- start start 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.626+0000 7f3771600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c101dd0 0x7f376c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.626+0000 7f3771600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 0x7f376c19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.626+0000 7f3771600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376c106f00 0x7f376c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.626+0000 7f3771600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f376c111f60 con 0x7f376c102780 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.626+0000 7f3771600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f376c111de0 con 0x7f376c101dd0 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.626+0000 7f3771600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f376c1120e0 con 0x7f376c106f00 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f3762000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 0x7f376c19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f3762000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 0x7f376c19ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51144/0 (socket says 172.21.15.114:51144) 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f3762000640 1 -- 172.21.15.114:0/3224516520 learned_addr learned my addr 172.21.15.114:0/3224516520 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f376b400640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376c106f00 0x7f376c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f376aa00640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c101dd0 0x7f376c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f376b400640 1 -- 172.21.15.114:0/3224516520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c101dd0 msgr2=0x7f376c19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f376b400640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c101dd0 0x7f376c19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f376b400640 1 -- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 msgr2=0x7f376c19ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f376b400640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 0x7f376c19ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f376b400640 1 -- 172.21.15.114:0/3224516520 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c19f670 con 0x7f376c106f00 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f3762000640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 0x7f376c19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.627+0000 7f376b400640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376c106f00 0x7f376c19ef90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f376400ec50 tx=0x7f376400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.628+0000 7f3763e00640 1 -- 172.21.15.114:0/3224516520 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764003b70 con 0x7f376c106f00 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.628+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c19f960 con 0x7f376c106f00 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.628+0000 7f3763e00640 1 -- 172.21.15.114:0/3224516520 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3764003d10 con 0x7f376c106f00 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.628+0000 7f3763e00640 1 -- 172.21.15.114:0/3224516520 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764010890 con 0x7f376c106f00 2024-09-19T06:49:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.628+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c1a7240 con 0x7f376c106f00 2024-09-19T06:49:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.629+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c1021b0 con 0x7f376c106f00 2024-09-19T06:49:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.631+0000 7f3763e00640 1 -- 172.21.15.114:0/3224516520 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f3764003fc0 con 0x7f376c106f00 2024-09-19T06:49:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.632+0000 7f3763e00640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37500a3240 0x7f37500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.632+0000 7f3763e00640 1 -- 172.21.15.114:0/3224516520 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7497+0+0 (secure 0 0 0) 0x7f37640cfbc0 con 0x7f376c106f00 2024-09-19T06:49:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.632+0000 7f376aa00640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37500a3240 0x7f37500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.633+0000 7f376aa00640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37500a3240 0x7f37500a5700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f375c009bd0 tx=0x7f375c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:08.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.637+0000 7f3763e00640 1 -- 172.21.15.114:0/3224516520 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f376409cb70 con 0x7f376c106f00 2024-09-19T06:49:08.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:08.818+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f376c1a7470 con 0x7f376c106f00 2024-09-19T06:49:09.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.355+0000 7f3763e00640 1 -- 172.21.15.114:0/3224516520 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v82) ==== 171+0+0 (secure 0 0 0) 0x7f376409c470 con 0x7f376c106f00 2024-09-19T06:49:09.357 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-09-19T06:49:09.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.360+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37500a3240 msgr2=0x7f37500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:09.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.360+0000 7f3771600640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37500a3240 0x7f37500a5700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f375c009bd0 tx=0x7f375c005f90 comp rx=0 tx=0).stop 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.360+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376c106f00 msgr2=0x7f376c19ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.360+0000 7f3771600640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376c106f00 0x7f376c19ef90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f376400ec50 tx=0x7f376400c6a0 comp rx=0 tx=0).stop 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.361+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 shutdown_connections 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.361+0000 7f3771600640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37500a3240 0x7f37500a5700 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.361+0000 7f3771600640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f376c106f00 0x7f376c19ef90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.361+0000 7f3771600640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f376c102780 0x7f376c19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.361+0000 7f3771600640 1 --2- 172.21.15.114:0/3224516520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c101dd0 0x7f376c19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.361+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 >> 172.21.15.114:0/3224516520 conn(0x7f376c0fd8b0 msgr2=0x7f376c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:09.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.361+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 shutdown_connections 2024-09-19T06:49:09.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:09.362+0000 7f3771600640 1 -- 172.21.15.114:0/3224516520 wait complete. 2024-09-19T06:49:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:09 smithi114 ceph-mon[30820]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T06:49:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:09 smithi114 ceph-mon[30820]: osdmap e81: 12 total, 12 up, 12 in 2024-09-19T06:49:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:09 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3224516520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-19T06:49:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:09 smithi116 ceph-mon[35411]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T06:49:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:09 smithi116 ceph-mon[35411]: osdmap e81: 12 total, 12 up, 12 in 2024-09-19T06:49:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:09 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3224516520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-19T06:49:09.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:09 smithi118 ceph-mon[35527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T06:49:09.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:09 smithi118 ceph-mon[35527]: osdmap e81: 12 total, 12 up, 12 in 2024-09-19T06:49:09.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:09 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3224516520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-19T06:49:10.268 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-19T06:49:10.655 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:10 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-19T06:49:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:10 smithi114 ceph-mon[30820]: pgmap v290: 33 pgs: 25 unknown, 8 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:10 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-19T06:49:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:10 smithi114 ceph-mon[30820]: osdmap e82: 12 total, 12 up, 12 in 2024-09-19T06:49:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:10 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-19T06:49:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:10 smithi116 ceph-mon[35411]: pgmap v290: 33 pgs: 25 unknown, 8 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:10 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-19T06:49:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:10 smithi116 ceph-mon[35411]: osdmap e82: 12 total, 12 up, 12 in 2024-09-19T06:49:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:10 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-19T06:49:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:10 smithi118 ceph-mon[35527]: pgmap v290: 33 pgs: 25 unknown, 8 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:10 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-19T06:49:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:10 smithi118 ceph-mon[35527]: osdmap e82: 12 total, 12 up, 12 in 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.224+0000 7fcbada00640 1 -- 172.21.15.114:0/2335507310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 msgr2=0x7fcba81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.224+0000 7fcbada00640 1 --2- 172.21.15.114:0/2335507310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 0x7fcba81067e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009900 tx=0x7fcb9402a3f0 comp rx=0 tx=0).stop 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.225+0000 7fcbada00640 1 -- 172.21.15.114:0/2335507310 shutdown_connections 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.225+0000 7fcbada00640 1 --2- 172.21.15.114:0/2335507310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8106f10 0x7fcba810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.225+0000 7fcbada00640 1 --2- 172.21.15.114:0/2335507310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 0x7fcba81067e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.225+0000 7fcbada00640 1 --2- 172.21.15.114:0/2335507310 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8101de0 0x7fcba81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.225+0000 7fcbada00640 1 -- 172.21.15.114:0/2335507310 >> 172.21.15.114:0/2335507310 conn(0x7fcba80fd8c0 msgr2=0x7fcba80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.225+0000 7fcbada00640 1 -- 172.21.15.114:0/2335507310 shutdown_connections 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.225+0000 7fcbada00640 1 -- 172.21.15.114:0/2335507310 wait complete. 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcbada00640 1 Processor -- start 2024-09-19T06:49:11.230 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcbada00640 1 -- start start 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcbada00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8101de0 0x7fcba819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcbada00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 0x7fcba819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcbada00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8106f10 0x7fcba819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcbada00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcba8111f70 con 0x7fcba8102790 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcbada00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcba8111df0 con 0x7fcba8101de0 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcbada00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fcba81120f0 con 0x7fcba8106f10 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcba6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 0x7fcba819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcba6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 0x7fcba819aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51160/0 (socket says 172.21.15.114:51160) 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcba6000640 1 -- 172.21.15.114:0/369728173 learned_addr learned my addr 172.21.15.114:0/369728173 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.226+0000 7fcba7400640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8106f10 0x7fcba819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcba6a00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8101de0 0x7fcba819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcba7400640 1 -- 172.21.15.114:0/369728173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8101de0 msgr2=0x7fcba819a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:11.231 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcba7400640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8101de0 0x7fcba819a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcba7400640 1 -- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 msgr2=0x7fcba819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcba7400640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 0x7fcba819aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcba7400640 1 -- 172.21.15.114:0/369728173 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba819f610 con 0x7fcba8106f10 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcba7400640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8106f10 0x7fcba819ef30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcba000e970 tx=0x7fcba000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcb9f400640 1 -- 172.21.15.114:0/369728173 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba000ccc0 con 0x7fcba8106f10 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba819f900 con 0x7fcba8106f10 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcb9f400640 1 -- 172.21.15.114:0/369728173 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcba0005020 con 0x7fcba8106f10 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.227+0000 7fcb9f400640 1 -- 172.21.15.114:0/369728173 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0004a00 con 0x7fcba8106f10 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.228+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba81a71e0 con 0x7fcba8106f10 2024-09-19T06:49:11.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.229+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba81021c0 con 0x7fcba8106f10 2024-09-19T06:49:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.231+0000 7fcb9f400640 1 -- 172.21.15.114:0/369728173 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fcba00051d0 con 0x7fcba8106f10 2024-09-19T06:49:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.232+0000 7fcb9f400640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcb8c0a32c0 0x7fcb8c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.232+0000 7fcb9f400640 1 -- 172.21.15.114:0/369728173 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7fcba00cf730 con 0x7fcba8106f10 2024-09-19T06:49:11.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.232+0000 7fcba6a00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcb8c0a32c0 0x7fcb8c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:11.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.233+0000 7fcba6a00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcb8c0a32c0 0x7fcb8c0a5780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb98005fd0 tx=0x7fcb98009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:11.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.234+0000 7fcb9f400640 1 -- 172.21.15.114:0/369728173 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fcba009c5f0 con 0x7fcba8106f10 2024-09-19T06:49:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:11 smithi114 ceph-mon[30820]: osdmap e83: 12 total, 12 up, 12 in 2024-09-19T06:49:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:11 smithi114 ceph-mon[30820]: pgmap v293: 97 pgs: 21 creating+peering, 45 unknown, 31 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:11.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:11.448+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fcba81a74f0 con 0x7fcba8106f10 2024-09-19T06:49:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:11 smithi116 ceph-mon[35411]: osdmap e83: 12 total, 12 up, 12 in 2024-09-19T06:49:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:11 smithi116 ceph-mon[35411]: pgmap v293: 97 pgs: 21 creating+peering, 45 unknown, 31 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:11 smithi118 ceph-mon[35527]: osdmap e83: 12 total, 12 up, 12 in 2024-09-19T06:49:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:11 smithi118 ceph-mon[35527]: pgmap v293: 97 pgs: 21 creating+peering, 45 unknown, 31 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.368+0000 7fcb9f400640 1 -- 172.21.15.114:0/369728173 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fcba009bef0 con 0x7fcba8106f10 2024-09-19T06:49:12.369 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 2 and data pool 3 2024-09-19T06:49:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.372+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcb8c0a32c0 msgr2=0x7fcb8c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.372+0000 7fcbada00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcb8c0a32c0 0x7fcb8c0a5780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcb98005fd0 tx=0x7fcb98009500 comp rx=0 tx=0).stop 2024-09-19T06:49:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.372+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8106f10 msgr2=0x7fcba819ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.372+0000 7fcbada00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8106f10 0x7fcba819ef30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcba000e970 tx=0x7fcba000ee30 comp rx=0 tx=0).stop 2024-09-19T06:49:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.373+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 shutdown_connections 2024-09-19T06:49:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.373+0000 7fcbada00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fcb8c0a32c0 0x7fcb8c0a5780 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:12.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.373+0000 7fcbada00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8106f10 0x7fcba819ef30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:12.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.373+0000 7fcbada00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcba8102790 0x7fcba819aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:12.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.373+0000 7fcbada00640 1 --2- 172.21.15.114:0/369728173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcba8101de0 0x7fcba819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:12.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.373+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 >> 172.21.15.114:0/369728173 conn(0x7fcba80fd8c0 msgr2=0x7fcba80ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:12.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.373+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 shutdown_connections 2024-09-19T06:49:12.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:12.374+0000 7fcbada00640 1 -- 172.21.15.114:0/369728173 wait complete. 2024-09-19T06:49:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:12 smithi114 ceph-mon[30820]: osdmap e84: 12 total, 12 up, 12 in 2024-09-19T06:49:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:12 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/369728173' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T06:49:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:12 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T06:49:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:12 smithi114 ceph-mon[30820]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-19T06:49:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:12 smithi116 ceph-mon[35411]: osdmap e84: 12 total, 12 up, 12 in 2024-09-19T06:49:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:12 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/369728173' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T06:49:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:12 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T06:49:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:12 smithi116 ceph-mon[35411]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-19T06:49:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:12 smithi118 ceph-mon[35527]: osdmap e84: 12 total, 12 up, 12 in 2024-09-19T06:49:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:12 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/369728173' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T06:49:12.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:12 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-19T06:49:12.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:12 smithi118 ceph-mon[35527]: daemon mds.i assigned to filesystem cephfs as rank 0 2024-09-19T06:49:13.293 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-19T06:49:13.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-19T06:49:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:13 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-19T06:49:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:13 smithi114 ceph-mon[30820]: osdmap e85: 12 total, 12 up, 12 in 2024-09-19T06:49:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:13 smithi114 ceph-mon[30820]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-19T06:49:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:13 smithi114 ceph-mon[30820]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-19T06:49:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:13 smithi114 ceph-mon[30820]: pgmap v296: 97 pgs: 21 creating+peering, 28 unknown, 48 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:13.670 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:13 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-19T06:49:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:13 smithi116 ceph-mon[35411]: osdmap e85: 12 total, 12 up, 12 in 2024-09-19T06:49:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:13 smithi116 ceph-mon[35411]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-19T06:49:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:13 smithi116 ceph-mon[35411]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-19T06:49:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:13 smithi116 ceph-mon[35411]: pgmap v296: 97 pgs: 21 creating+peering, 28 unknown, 48 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:13 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-19T06:49:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:13 smithi118 ceph-mon[35527]: osdmap e85: 12 total, 12 up, 12 in 2024-09-19T06:49:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:13 smithi118 ceph-mon[35527]: fsmap cephfs:1 {0=i=up:creating} 11 up:standby 2024-09-19T06:49:13.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:13 smithi118 ceph-mon[35527]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-09-19T06:49:13.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:13 smithi118 ceph-mon[35527]: pgmap v296: 97 pgs: 21 creating+peering, 28 unknown, 48 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 -- 172.21.15.114:0/1370759231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950103e30 msgr2=0x7fc950104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 --2- 172.21.15.114:0/1370759231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950103e30 0x7fc950104210 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc940009900 tx=0x7fc94002a410 comp rx=0 tx=0).stop 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 -- 172.21.15.114:0/1370759231 shutdown_connections 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 --2- 172.21.15.114:0/1370759231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc950108f60 0x7fc950110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 --2- 172.21.15.114:0/1370759231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501047e0 0x7fc950108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 --2- 172.21.15.114:0/1370759231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950103e30 0x7fc950104210 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 -- 172.21.15.114:0/1370759231 >> 172.21.15.114:0/1370759231 conn(0x7fc9500ff8d0 msgr2=0x7fc950101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 -- 172.21.15.114:0/1370759231 shutdown_connections 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.226+0000 7fc955800640 1 -- 172.21.15.114:0/1370759231 wait complete. 2024-09-19T06:49:14.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.227+0000 7fc955800640 1 Processor -- start 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.227+0000 7fc955800640 1 -- start start 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.227+0000 7fc955800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc950103e30 0x7fc95019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.227+0000 7fc955800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501047e0 0x7fc95019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.227+0000 7fc955800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950108f60 0x7fc95019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.227+0000 7fc955800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc950114010 con 0x7fc950108f60 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.227+0000 7fc955800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc950113e90 con 0x7fc9501047e0 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.227+0000 7fc955800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc950114190 con 0x7fc950103e30 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950108f60 0x7fc95019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc94f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501047e0 0x7fc95019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950108f60 0x7fc95019ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51178/0 (socket says 172.21.15.114:51178) 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc94f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501047e0 0x7fc95019a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:36526/0 (socket says 172.21.15.114:36526) 2024-09-19T06:49:14.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 -- 172.21.15.114:0/219759999 learned_addr learned my addr 172.21.15.114:0/219759999 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc94fe00640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc950103e30 0x7fc95019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 -- 172.21.15.114:0/219759999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc950103e30 msgr2=0x7fc95019a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc950103e30 0x7fc95019a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 -- 172.21.15.114:0/219759999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501047e0 msgr2=0x7fc95019a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501047e0 0x7fc95019a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 -- 172.21.15.114:0/219759999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95019f440 con 0x7fc950108f60 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc94f400640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501047e0 0x7fc95019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc94fe00640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc950103e30 0x7fc95019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.228+0000 7fc954e00640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950108f60 0x7fc95019ed60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc94800eb10 tx=0x7fc94800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.229+0000 7fc94cc00640 1 -- 172.21.15.114:0/219759999 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948004b20 con 0x7fc950108f60 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.229+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95019f730 con 0x7fc950108f60 2024-09-19T06:49:14.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.229+0000 7fc94cc00640 1 -- 172.21.15.114:0/219759999 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc948004cc0 con 0x7fc950108f60 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.229+0000 7fc94cc00640 1 -- 172.21.15.114:0/219759999 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948010760 con 0x7fc950108f60 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.229+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9501a7010 con 0x7fc950108f60 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.230+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc950104210 con 0x7fc950108f60 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.232+0000 7fc94cc00640 1 -- 172.21.15.114:0/219759999 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fc948005020 con 0x7fc950108f60 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.233+0000 7fc94cc00640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc9340a3240 0x7fc9340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.233+0000 7fc94cc00640 1 -- 172.21.15.114:0/219759999 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc9480cfea0 con 0x7fc950108f60 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.233+0000 7fc94fe00640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc9340a3240 0x7fc9340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.234+0000 7fc94fe00640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc9340a3240 0x7fc9340a5700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc940000c00 tx=0x7fc940031510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:14.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.238+0000 7fc94cc00640 1 -- 172.21.15.114:0/219759999 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fc94809cc70 con 0x7fc950108f60 2024-09-19T06:49:14.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:14.424+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fc95010bbb0 con 0x7fc950108f60 2024-09-19T06:49:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:14 smithi114 ceph-mon[30820]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-19T06:49:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:14 smithi114 ceph-mon[30820]: Cluster is now healthy 2024-09-19T06:49:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:14 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] up:active 2024-09-19T06:49:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:14 smithi114 ceph-mon[30820]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:14 smithi114 ceph-mon[30820]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:14 smithi116 ceph-mon[35411]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-19T06:49:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:14 smithi116 ceph-mon[35411]: Cluster is now healthy 2024-09-19T06:49:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:14 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] up:active 2024-09-19T06:49:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:14 smithi116 ceph-mon[35411]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:14 smithi116 ceph-mon[35411]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:14 smithi118 ceph-mon[35527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-19T06:49:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:14 smithi118 ceph-mon[35527]: Cluster is now healthy 2024-09-19T06:49:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:14 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] up:active 2024-09-19T06:49:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:14 smithi118 ceph-mon[35527]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:14 smithi118 ceph-mon[35527]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:15.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.384+0000 7fc94cc00640 1 -- 172.21.15.114:0/219759999 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7fc9480bc030 con 0x7fc950108f60 2024-09-19T06:49:15.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.387+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc9340a3240 msgr2=0x7fc9340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:15.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.387+0000 7fc955800640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc9340a3240 0x7fc9340a5700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc940000c00 tx=0x7fc940031510 comp rx=0 tx=0).stop 2024-09-19T06:49:15.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.387+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950108f60 msgr2=0x7fc95019ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:15.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.387+0000 7fc955800640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950108f60 0x7fc95019ed60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc94800eb10 tx=0x7fc94800c2d0 comp rx=0 tx=0).stop 2024-09-19T06:49:15.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.388+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 shutdown_connections 2024-09-19T06:49:15.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.388+0000 7fc955800640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc9340a3240 0x7fc9340a5700 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:15.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.388+0000 7fc955800640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc950108f60 0x7fc95019ed60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:15.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.388+0000 7fc955800640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9501047e0 0x7fc95019a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:15.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.388+0000 7fc955800640 1 --2- 172.21.15.114:0/219759999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc950103e30 0x7fc95019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:15.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.388+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 >> 172.21.15.114:0/219759999 conn(0x7fc9500ff8d0 msgr2=0x7fc950100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:15.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.388+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 shutdown_connections 2024-09-19T06:49:15.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:15.388+0000 7fc955800640 1 -- 172.21.15.114:0/219759999 wait complete. 2024-09-19T06:49:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:15 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/219759999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-19T06:49:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:15 smithi114 ceph-mon[30820]: pgmap v297: 97 pgs: 6 creating+peering, 91 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-19T06:49:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:15 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/219759999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-19T06:49:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:15 smithi116 ceph-mon[35411]: pgmap v297: 97 pgs: 6 creating+peering, 91 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-19T06:49:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:15 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/219759999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-19T06:49:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:15 smithi118 ceph-mon[35527]: pgmap v297: 97 pgs: 6 creating+peering, 91 active+clean; 577 KiB data, 335 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-19T06:49:16.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:16 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/219759999' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-19T06:49:16.436 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:16 smithi114 ceph-mon[30820]: osdmap e86: 12 total, 12 up, 12 in 2024-09-19T06:49:16.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-19T06:49:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:16 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/219759999' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-19T06:49:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:16 smithi116 ceph-mon[35411]: osdmap e86: 12 total, 12 up, 12 in 2024-09-19T06:49:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:16 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/219759999' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-19T06:49:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:16 smithi118 ceph-mon[35527]: osdmap e86: 12 total, 12 up, 12 in 2024-09-19T06:49:16.850 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:17.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 -- 172.21.15.114:0/1881037200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8106f10 msgr2=0x7f13d810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:17.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 --2- 172.21.15.114:0/1881037200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8106f10 0x7f13d810ea90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009ca0 tx=0x7f13d002a8c0 comp rx=0 tx=0).stop 2024-09-19T06:49:17.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 -- 172.21.15.114:0/1881037200 shutdown_connections 2024-09-19T06:49:17.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 --2- 172.21.15.114:0/1881037200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8106f10 0x7f13d810ea90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:17.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 --2- 172.21.15.114:0/1881037200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d8102790 0x7f13d81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:17.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 --2- 172.21.15.114:0/1881037200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13d8101de0 0x7f13d81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:17.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 -- 172.21.15.114:0/1881037200 >> 172.21.15.114:0/1881037200 conn(0x7f13d80fd8c0 msgr2=0x7f13d80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:17.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 -- 172.21.15.114:0/1881037200 shutdown_connections 2024-09-19T06:49:17.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.469+0000 7f13df400640 1 -- 172.21.15.114:0/1881037200 wait complete. 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.470+0000 7f13df400640 1 Processor -- start 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.470+0000 7f13df400640 1 -- start start 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13df400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d8101de0 0x7f13d819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13df400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8102790 0x7f13d819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13df400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13d8106f10 0x7f13d819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13df400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8111fc0 con 0x7f13d8102790 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13df400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8111e40 con 0x7f13d8101de0 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13df400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f13d8112140 con 0x7f13d8106f10 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8102790 0x7f13d819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:17.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13d7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8102790 0x7f13d819aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49800/0 (socket says 172.21.15.114:49800) 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13dca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d8101de0 0x7f13d819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13d7e00640 1 -- 172.21.15.114:0/2791787136 learned_addr learned my addr 172.21.15.114:0/2791787136 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.471+0000 7f13dd400640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13d8106f10 0x7f13d819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13dca00640 1 -- 172.21.15.114:0/2791787136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13d8106f10 msgr2=0x7f13d819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13dca00640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13d8106f10 0x7f13d819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13dca00640 1 -- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8102790 msgr2=0x7f13d819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13dca00640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8102790 0x7f13d819aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13dca00640 1 -- 172.21.15.114:0/2791787136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d819f610 con 0x7f13d8101de0 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13dca00640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d8101de0 0x7f13d819a660 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00e9f0 tx=0x7f13cc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13d5600640 1 -- 172.21.15.114:0/2791787136 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc00ccc0 con 0x7f13d8101de0 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d819f900 con 0x7f13d8101de0 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13d5600640 1 -- 172.21.15.114:0/2791787136 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13cc005020 con 0x7f13d8101de0 2024-09-19T06:49:17.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13d5600640 1 -- 172.21.15.114:0/2791787136 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0049e0 con 0x7f13d8101de0 2024-09-19T06:49:17.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.472+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d81a71e0 con 0x7f13d8101de0 2024-09-19T06:49:17.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.473+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d81021c0 con 0x7f13d8101de0 2024-09-19T06:49:17.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.476+0000 7f13d5600640 1 -- 172.21.15.114:0/2791787136 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f13cc005560 con 0x7f13d8101de0 2024-09-19T06:49:17.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.477+0000 7f13d5600640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f13c00a32c0 0x7f13c00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:17.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.477+0000 7f13d5600640 1 -- 172.21.15.114:0/2791787136 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7f13cc0cf870 con 0x7f13d8101de0 2024-09-19T06:49:17.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.477+0000 7f13d7e00640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f13c00a32c0 0x7f13c00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:17.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.478+0000 7f13d7e00640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f13c00a32c0 0x7f13c00a5780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f13d80fe0c0 tx=0x7f13c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:17.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.481+0000 7f13d5600640 1 -- 172.21.15.114:0/2791787136 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f13cc0051d0 con 0x7f13d8101de0 2024-09-19T06:49:17.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:17.656+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f13d81a73b0 con 0x7f13d8101de0 2024-09-19T06:49:17.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:17 smithi114 ceph-mon[30820]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-19T06:49:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:17 smithi116 ceph-mon[35411]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-19T06:49:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:17 smithi118 ceph-mon[35527]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 7 op/s 2024-09-19T06:49:18.395 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:49:18 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-i[66010]: 2024-09-19T06:49:18.016+0000 7f7e6a4f1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T06:49:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:18 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2791787136' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T06:49:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:18 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T06:49:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:18 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2791787136' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T06:49:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:18 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T06:49:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:18 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2791787136' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T06:49:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:18 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-19T06:49:19.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.404+0000 7f13d5600640 1 -- 172.21.15.114:0/2791787136 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7f13cc09c500 con 0x7f13d8101de0 2024-09-19T06:49:19.407 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data_ec' created 2024-09-19T06:49:19.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.409+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f13c00a32c0 msgr2=0x7f13c00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.409+0000 7f13df400640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f13c00a32c0 0x7f13c00a5780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f13d80fe0c0 tx=0x7f13c4005f90 comp rx=0 tx=0).stop 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.409+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d8101de0 msgr2=0x7f13d819a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.409+0000 7f13df400640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d8101de0 0x7f13d819a660 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00e9f0 tx=0x7f13cc00eeb0 comp rx=0 tx=0).stop 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.410+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 shutdown_connections 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.410+0000 7f13df400640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f13c00a32c0 0x7f13c00a5780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.410+0000 7f13df400640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13d8106f10 0x7f13d819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.410+0000 7f13df400640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d8102790 0x7f13d819aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.410+0000 7f13df400640 1 --2- 172.21.15.114:0/2791787136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13d8101de0 0x7f13d819a660 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:19.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.410+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 >> 172.21.15.114:0/2791787136 conn(0x7f13d80fd8c0 msgr2=0x7f13d80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:19.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.410+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 shutdown_connections 2024-09-19T06:49:19.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:19.411+0000 7f13df400640 1 -- 172.21.15.114:0/2791787136 wait complete. 2024-09-19T06:49:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:19 smithi114 ceph-mon[30820]: osdmap e87: 12 total, 12 up, 12 in 2024-09-19T06:49:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:19 smithi114 ceph-mon[30820]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-19T06:49:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:19 smithi116 ceph-mon[35411]: osdmap e87: 12 total, 12 up, 12 in 2024-09-19T06:49:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:19 smithi116 ceph-mon[35411]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-19T06:49:19.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:19 smithi118 ceph-mon[35527]: osdmap e87: 12 total, 12 up, 12 in 2024-09-19T06:49:19.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:19 smithi118 ceph-mon[35527]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 6 op/s 2024-09-19T06:49:20.181 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-19T06:49:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:20 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-19T06:49:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:20 smithi114 ceph-mon[30820]: osdmap e88: 12 total, 12 up, 12 in 2024-09-19T06:49:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:20 smithi114 ceph-mon[30820]: osdmap e89: 12 total, 12 up, 12 in 2024-09-19T06:49:20.567 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:20 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-19T06:49:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:20 smithi116 ceph-mon[35411]: osdmap e88: 12 total, 12 up, 12 in 2024-09-19T06:49:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:20 smithi116 ceph-mon[35411]: osdmap e89: 12 total, 12 up, 12 in 2024-09-19T06:49:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:20 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-19T06:49:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:20 smithi118 ceph-mon[35527]: osdmap e88: 12 total, 12 up, 12 in 2024-09-19T06:49:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:20 smithi118 ceph-mon[35527]: osdmap e89: 12 total, 12 up, 12 in 2024-09-19T06:49:21.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 -- 172.21.15.114:0/4009094991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848101de0 msgr2=0x7fe8481021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 --2- 172.21.15.114:0/4009094991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848101de0 0x7fe8481021c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe838009900 tx=0x7fe83802a3f0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 -- 172.21.15.114:0/4009094991 shutdown_connections 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 --2- 172.21.15.114:0/4009094991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe848106f10 0x7fe84810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 --2- 172.21.15.114:0/4009094991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe848102790 0x7fe8481067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 --2- 172.21.15.114:0/4009094991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848101de0 0x7fe8481021c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 -- 172.21.15.114:0/4009094991 >> 172.21.15.114:0/4009094991 conn(0x7fe8480fd8c0 msgr2=0x7fe8480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 -- 172.21.15.114:0/4009094991 shutdown_connections 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.102+0000 7fe84ca00640 1 -- 172.21.15.114:0/4009094991 wait complete. 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.103+0000 7fe84ca00640 1 Processor -- start 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.103+0000 7fe84ca00640 1 -- start start 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.103+0000 7fe84ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe848101de0 0x7fe84819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:21.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.103+0000 7fe84ca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe848102790 0x7fe84819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.103+0000 7fe84ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848106f10 0x7fe84819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.103+0000 7fe84ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe848111f80 con 0x7fe848106f10 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.103+0000 7fe84ca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe848111e00 con 0x7fe848101de0 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.103+0000 7fe84ca00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe848112100 con 0x7fe848102790 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848106f10 0x7fe84819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe845e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe848101de0 0x7fe84819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe83d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe848102790 0x7fe84819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848106f10 0x7fe84819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49804/0 (socket says 172.21.15.114:49804) 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 -- 172.21.15.114:0/2781772333 learned_addr learned my addr 172.21.15.114:0/2781772333 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 -- 172.21.15.114:0/2781772333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe848102790 msgr2=0x7fe84819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe848102790 0x7fe84819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 -- 172.21.15.114:0/2781772333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe848101de0 msgr2=0x7fe84819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:21.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe848101de0 0x7fe84819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 -- 172.21.15.114:0/2781772333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84819f620 con 0x7fe848106f10 2024-09-19T06:49:21.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe846800640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848106f10 0x7fe84819ef40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe84000efd0 tx=0x7fe84000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:21.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe83f400640 1 -- 172.21.15.114:0/2781772333 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840003b70 con 0x7fe848106f10 2024-09-19T06:49:21.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe83f400640 1 -- 172.21.15.114:0/2781772333 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe840003d10 con 0x7fe848106f10 2024-09-19T06:49:21.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84819f910 con 0x7fe848106f10 2024-09-19T06:49:21.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.104+0000 7fe83f400640 1 -- 172.21.15.114:0/2781772333 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8400109f0 con 0x7fe848106f10 2024-09-19T06:49:21.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.105+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8481a71f0 con 0x7fe848106f10 2024-09-19T06:49:21.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.106+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8481021c0 con 0x7fe848106f10 2024-09-19T06:49:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.108+0000 7fe83f400640 1 -- 172.21.15.114:0/2781772333 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fe840010b90 con 0x7fe848106f10 2024-09-19T06:49:21.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.109+0000 7fe83f400640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe82c0a3240 0x7fe82c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:21.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.109+0000 7fe83f400640 1 -- 172.21.15.114:0/2781772333 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7fe8400d0290 con 0x7fe848106f10 2024-09-19T06:49:21.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.109+0000 7fe845e00640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe82c0a3240 0x7fe82c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:21.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.110+0000 7fe845e00640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe82c0a3240 0x7fe82c0a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe838000c00 tx=0x7fe838005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:21.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.112+0000 7fe83f400640 1 -- 172.21.15.114:0/2781772333 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe84009cd90 con 0x7fe848106f10 2024-09-19T06:49:21.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.287+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fe84819b9d0 con 0x7fe848106f10 2024-09-19T06:49:21.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.419+0000 7fe83f400640 1 -- 172.21.15.114:0/2781772333 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v90) ==== 169+0+0 (secure 0 0 0) 0x7fe84819b9d0 con 0x7fe848106f10 2024-09-19T06:49:21.420 INFO:teuthology.orchestra.run.smithi114.stderr:set pool 4 allow_ec_overwrites to true 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe82c0a3240 msgr2=0x7fe82c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe82c0a3240 0x7fe82c0a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe838000c00 tx=0x7fe838005b50 comp rx=0 tx=0).stop 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848106f10 msgr2=0x7fe84819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848106f10 0x7fe84819ef40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe84000efd0 tx=0x7fe84000c6a0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 shutdown_connections 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe82c0a3240 0x7fe82c0a5700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe848106f10 0x7fe84819ef40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe848102790 0x7fe84819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 --2- 172.21.15.114:0/2781772333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe848101de0 0x7fe84819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 >> 172.21.15.114:0/2781772333 conn(0x7fe8480fd8c0 msgr2=0x7fe8480feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 shutdown_connections 2024-09-19T06:49:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:21.421+0000 7fe84ca00640 1 -- 172.21.15.114:0/2781772333 wait complete. 2024-09-19T06:49:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:21 smithi114 ceph-mon[30820]: pgmap v304: 129 pgs: 12 creating+peering, 20 unknown, 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:21 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2781772333' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-19T06:49:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:21 smithi116 ceph-mon[35411]: pgmap v304: 129 pgs: 12 creating+peering, 20 unknown, 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:21 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2781772333' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-19T06:49:21.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:21 smithi118 ceph-mon[35527]: pgmap v304: 129 pgs: 12 creating+peering, 20 unknown, 97 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:21.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:21 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2781772333' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-19T06:49:22.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-19T06:49:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:22 smithi114 ceph-mon[30820]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T06:49:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:22 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2781772333' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-19T06:49:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:22 smithi114 ceph-mon[30820]: osdmap e90: 12 total, 12 up, 12 in 2024-09-19T06:49:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:22 smithi116 ceph-mon[35411]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T06:49:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:22 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2781772333' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-19T06:49:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:22 smithi116 ceph-mon[35411]: osdmap e90: 12 total, 12 up, 12 in 2024-09-19T06:49:22.691 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:22 smithi118 ceph-mon[35527]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-19T06:49:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:22 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2781772333' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-19T06:49:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:22 smithi118 ceph-mon[35527]: osdmap e90: 12 total, 12 up, 12 in 2024-09-19T06:49:23.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 -- 172.21.15.114:0/3699756879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820077040 msgr2=0x7f2820112330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:23.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 --2- 172.21.15.114:0/3699756879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820077040 0x7f2820112330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f280c009900 tx=0x7f280c02a410 comp rx=0 tx=0).stop 2024-09-19T06:49:23.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 -- 172.21.15.114:0/3699756879 shutdown_connections 2024-09-19T06:49:23.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 --2- 172.21.15.114:0/3699756879 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2820112870 0x7f2820114c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:23.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 --2- 172.21.15.114:0/3699756879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820077040 0x7f2820112330 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:23.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 --2- 172.21.15.114:0/3699756879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2820076720 0x7f2820076b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 -- 172.21.15.114:0/3699756879 >> 172.21.15.114:0/3699756879 conn(0x7f28200ff8d0 msgr2=0x7f2820101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 -- 172.21.15.114:0/3699756879 shutdown_connections 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.268+0000 7f2826c00640 1 -- 172.21.15.114:0/3699756879 wait complete. 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.269+0000 7f2826c00640 1 Processor -- start 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.269+0000 7f2826c00640 1 -- start start 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2820076720 0x7f28201a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2820077040 0x7f28201a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820112870 0x7f282019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2820118430 con 0x7f2820112870 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f28201182b0 con 0x7f2820076720 2024-09-19T06:49:23.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f28201185b0 con 0x7f2820077040 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820112870 0x7f282019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2824e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2820077040 0x7f28201a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820112870 0x7f282019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49834/0 (socket says 172.21.15.114:49834) 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826200640 1 -- 172.21.15.114:0/2766875958 learned_addr learned my addr 172.21.15.114:0/2766875958 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2825800640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2820076720 0x7f28201a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826200640 1 -- 172.21.15.114:0/2766875958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2820077040 msgr2=0x7f28201a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826200640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2820077040 0x7f28201a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826200640 1 -- 172.21.15.114:0/2766875958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2820076720 msgr2=0x7f28201a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826200640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2820076720 0x7f28201a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.270+0000 7f2826200640 1 -- 172.21.15.114:0/2766875958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282019f050 con 0x7f2820112870 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.271+0000 7f2826200640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820112870 0x7f282019e820 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2818009fd0 tx=0x7f281800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:23.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.271+0000 7f281e000640 1 -- 172.21.15.114:0/2766875958 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2818004d20 con 0x7f2820112870 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.271+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f282019f340 con 0x7f2820112870 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.271+0000 7f281e000640 1 -- 172.21.15.114:0/2766875958 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2818005020 con 0x7f2820112870 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.271+0000 7f281e000640 1 -- 172.21.15.114:0/2766875958 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28180116d0 con 0x7f2820112870 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.271+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28201ab4e0 con 0x7f2820112870 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.272+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2820076b00 con 0x7f2820112870 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.274+0000 7f281e000640 1 -- 172.21.15.114:0/2766875958 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f28180051d0 con 0x7f2820112870 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.276+0000 7f281e000640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f28080a3240 0x7f28080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.276+0000 7f281e000640 1 -- 172.21.15.114:0/2766875958 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8770+0+0 (secure 0 0 0) 0x7f28180cffe0 con 0x7f2820112870 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.276+0000 7f2825800640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f28080a3240 0x7f28080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.277+0000 7f2825800640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f28080a3240 0x7f28080a5700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f281400bbe0 tx=0x7f281400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:23.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.280+0000 7f281e000640 1 -- 172.21.15.114:0/2766875958 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2818016030 con 0x7f2820112870 2024-09-19T06:49:23.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:23.495+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f282010da00 con 0x7f2820112870 2024-09-19T06:49:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:23 smithi118 ceph-mon[35527]: pgmap v306: 129 pgs: 12 creating+peering, 15 unknown, 102 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:23 smithi114 ceph-mon[30820]: pgmap v306: 129 pgs: 12 creating+peering, 15 unknown, 102 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:23 smithi116 ceph-mon[35411]: pgmap v306: 129 pgs: 12 creating+peering, 15 unknown, 102 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:24.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.433+0000 7f281e000640 1 -- 172.21.15.114:0/2766875958 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f281809c490 con 0x7f2820112870 2024-09-19T06:49:24.435 INFO:teuthology.orchestra.run.smithi114.stderr:added data pool 4 to fsmap 2024-09-19T06:49:24.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f28080a3240 msgr2=0x7f28080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:24.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f28080a3240 0x7f28080a5700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f281400bbe0 tx=0x7f281400b390 comp rx=0 tx=0).stop 2024-09-19T06:49:24.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820112870 msgr2=0x7f282019e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:24.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820112870 0x7f282019e820 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2818009fd0 tx=0x7f281800ee60 comp rx=0 tx=0).stop 2024-09-19T06:49:24.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 shutdown_connections 2024-09-19T06:49:24.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f28080a3240 0x7f28080a5700 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:24.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2820112870 0x7f282019e820 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:24.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2820077040 0x7f28201a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:24.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 --2- 172.21.15.114:0/2766875958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2820076720 0x7f28201a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:24.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.436+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 >> 172.21.15.114:0/2766875958 conn(0x7f28200ff8d0 msgr2=0x7f2820101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:24.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.437+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 shutdown_connections 2024-09-19T06:49:24.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:24.437+0000 7f2826c00640 1 -- 172.21.15.114:0/2766875958 wait complete. 2024-09-19T06:49:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:24 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2766875958' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-19T06:49:24.795 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:24 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2766875958' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-19T06:49:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:24 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2766875958' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-19T06:49:25.282 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:49:25.649 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:25 smithi114 ceph-mon[30820]: osdmap e91: 12 total, 12 up, 12 in 2024-09-19T06:49:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:25 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2766875958' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-19T06:49:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:25 smithi114 ceph-mon[30820]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:25 smithi114 ceph-mon[30820]: pgmap v308: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:25 smithi118 ceph-mon[35527]: osdmap e91: 12 total, 12 up, 12 in 2024-09-19T06:49:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:25 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2766875958' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-19T06:49:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:25 smithi118 ceph-mon[35527]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:25 smithi118 ceph-mon[35527]: pgmap v308: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:25 smithi116 ceph-mon[35411]: osdmap e91: 12 total, 12 up, 12 in 2024-09-19T06:49:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:25 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2766875958' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-19T06:49:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:25 smithi116 ceph-mon[35411]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:25 smithi116 ceph-mon[35411]: pgmap v308: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:26.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.215+0000 7fee9f000640 1 -- 172.21.15.114:0/3762613130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98103ff0 msgr2=0x7fee981043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:26.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.215+0000 7fee9f000640 1 --2- 172.21.15.114:0/3762613130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98103ff0 0x7fee981043d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fee88009900 tx=0x7fee8802a410 comp rx=0 tx=0).stop 2024-09-19T06:49:26.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.215+0000 7fee9f000640 1 -- 172.21.15.114:0/3762613130 shutdown_connections 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.215+0000 7fee9f000640 1 --2- 172.21.15.114:0/3762613130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee98109120 0x7fee98110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.215+0000 7fee9f000640 1 --2- 172.21.15.114:0/3762613130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee981049a0 0x7fee981089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.215+0000 7fee9f000640 1 --2- 172.21.15.114:0/3762613130 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98103ff0 0x7fee981043d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.215+0000 7fee9f000640 1 -- 172.21.15.114:0/3762613130 >> 172.21.15.114:0/3762613130 conn(0x7fee980ffa90 msgr2=0x7fee98101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.215+0000 7fee9f000640 1 -- 172.21.15.114:0/3762613130 shutdown_connections 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.216+0000 7fee9f000640 1 -- 172.21.15.114:0/3762613130 wait complete. 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.216+0000 7fee9f000640 1 Processor -- start 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.216+0000 7fee9f000640 1 -- start start 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9f000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee98103ff0 0x7fee9819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9f000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee981049a0 0x7fee9819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9f000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98109120 0x7fee9819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:26.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9f000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee98114120 con 0x7fee98109120 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9f000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee98113fa0 con 0x7fee981049a0 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9f000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fee981142a0 con 0x7fee98103ff0 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98109120 0x7fee9819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9d000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98109120 0x7fee9819eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49854/0 (socket says 172.21.15.114:49854) 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee9d000640 1 -- 172.21.15.114:0/1847728268 learned_addr learned my addr 172.21.15.114:0/1847728268 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee97400640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee981049a0 0x7fee9819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.217+0000 7fee97e00640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee98103ff0 0x7fee9819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee9d000640 1 -- 172.21.15.114:0/1847728268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee98103ff0 msgr2=0x7fee9819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee9d000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee98103ff0 0x7fee9819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee9d000640 1 -- 172.21.15.114:0/1847728268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee981049a0 msgr2=0x7fee9819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee9d000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee981049a0 0x7fee9819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee9d000640 1 -- 172.21.15.114:0/1847728268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9819f590 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee9d000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98109120 0x7fee9819eeb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fee9000eaa0 tx=0x7fee9000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee94c00640 1 -- 172.21.15.114:0/1847728268 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9000cdb0 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9819f880 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee94c00640 1 -- 172.21.15.114:0/1847728268 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee90005020 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee94c00640 1 -- 172.21.15.114:0/1847728268 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee900116b0 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.218+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee981a7160 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.219+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee981043d0 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.221+0000 7fee94c00640 1 -- 172.21.15.114:0/1847728268 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fee90005560 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.222+0000 7fee94c00640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fee7c0a3240 0x7fee7c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.223+0000 7fee94c00640 1 -- 172.21.15.114:0/1847728268 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fee900d00c0 con 0x7fee98109120 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.223+0000 7fee97e00640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fee7c0a3240 0x7fee7c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.223+0000 7fee97e00640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fee7c0a3240 0x7fee7c0a5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fee88009230 tx=0x7fee880058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:26.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.225+0000 7fee94c00640 1 -- 172.21.15.114:0/1847728268 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fee90016030 con 0x7fee98109120 2024-09-19T06:49:26.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.443+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee9802d7c0 con 0x7fee98109120 2024-09-19T06:49:26.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.444+0000 7fee94c00640 1 -- 172.21.15.114:0/1847728268 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11863 (secure 0 0 0) 0x7fee9009c4f0 con 0x7fee98109120 2024-09-19T06:49:26.445 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:26.446 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"btime":"2024-09-19T06:49:24:427295+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14520,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T06:49:11.450651+0000","modified":"2024-09-19T06:49:23.496663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24538},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24538,"qdb_cluster":[24538]},"id":1}]} 2024-09-19T06:49:26.446 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-09-19T06:49:26.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.446+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fee7c0a3240 msgr2=0x7fee7c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:26.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.446+0000 7fee9f000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fee7c0a3240 0x7fee7c0a5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fee88009230 tx=0x7fee880058d0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.446+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98109120 msgr2=0x7fee9819eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:26.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.446+0000 7fee9f000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98109120 0x7fee9819eeb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fee9000eaa0 tx=0x7fee9000ef60 comp rx=0 tx=0).stop 2024-09-19T06:49:26.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.447+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 shutdown_connections 2024-09-19T06:49:26.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.447+0000 7fee9f000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fee7c0a3240 0x7fee7c0a5700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.447+0000 7fee9f000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee98109120 0x7fee9819eeb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.447+0000 7fee9f000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee981049a0 0x7fee9819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.447+0000 7fee9f000640 1 --2- 172.21.15.114:0/1847728268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee98103ff0 0x7fee9819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:26.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.447+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 >> 172.21.15.114:0/1847728268 conn(0x7fee980ffa90 msgr2=0x7fee98100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:26.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.447+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 shutdown_connections 2024-09-19T06:49:26.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:26.447+0000 7fee9f000640 1 -- 172.21.15.114:0/1847728268 wait complete. 2024-09-19T06:49:26.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:26 smithi118 ceph-mon[35527]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-19T06:49:26.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:26 smithi118 ceph-mon[35527]: Cluster is now healthy 2024-09-19T06:49:26.846 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:26 smithi114 ceph-mon[30820]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-19T06:49:26.846 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:26 smithi114 ceph-mon[30820]: Cluster is now healthy 2024-09-19T06:49:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:26 smithi116 ceph-mon[35411]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-19T06:49:26.934 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:26 smithi116 ceph-mon[35411]: Cluster is now healthy 2024-09-19T06:49:27.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:49:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:27 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1847728268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:27 smithi114 ceph-mon[30820]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:27.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:27 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1847728268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:27.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:27 smithi118 ceph-mon[35527]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:27.766 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:27 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1847728268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:27 smithi116 ceph-mon[35411]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:28.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.334+0000 7f840dc00640 1 -- 172.21.15.114:0/547087786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84081019c0 msgr2=0x7f8408101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:28.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.334+0000 7f840dc00640 1 --2- 172.21.15.114:0/547087786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84081019c0 0x7f8408101e20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f83f4009930 tx=0x7f83f402a420 comp rx=0 tx=0).stop 2024-09-19T06:49:28.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.334+0000 7f840dc00640 1 -- 172.21.15.114:0/547087786 shutdown_connections 2024-09-19T06:49:28.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.334+0000 7f840dc00640 1 --2- 172.21.15.114:0/547087786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8408102360 0x7f840810e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.334+0000 7f840dc00640 1 --2- 172.21.15.114:0/547087786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84081019c0 0x7f8408101e20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.334+0000 7f840dc00640 1 --2- 172.21.15.114:0/547087786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081079c0 0x7f8408107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.334+0000 7f840dc00640 1 -- 172.21.15.114:0/547087786 >> 172.21.15.114:0/547087786 conn(0x7f84080fd6e0 msgr2=0x7f84080ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.335+0000 7f840dc00640 1 -- 172.21.15.114:0/547087786 shutdown_connections 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.335+0000 7f840dc00640 1 -- 172.21.15.114:0/547087786 wait complete. 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.335+0000 7f840dc00640 1 Processor -- start 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f840dc00640 1 -- start start 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f840dc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84081019c0 0x7f840819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f840dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8408102360 0x7f840819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f840dc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081079c0 0x7f840819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f840dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8408111d90 con 0x7f8408102360 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f840dc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8408111c10 con 0x7f84081079c0 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f840dc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8408111f10 con 0x7f84081019c0 2024-09-19T06:49:28.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f8407400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8408102360 0x7f840819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f8407400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8408102360 0x7f840819aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59968/0 (socket says 172.21.15.114:59968) 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f8407400640 1 -- 172.21.15.114:0/67582990 learned_addr learned my addr 172.21.15.114:0/67582990 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f8407e00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84081019c0 0x7f840819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.336+0000 7f840d200640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081079c0 0x7f840819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8407400640 1 -- 172.21.15.114:0/67582990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84081019c0 msgr2=0x7f840819a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8407400640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84081019c0 0x7f840819a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8407400640 1 -- 172.21.15.114:0/67582990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081079c0 msgr2=0x7f840819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8407400640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081079c0 0x7f840819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8407400640 1 -- 172.21.15.114:0/67582990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840819f4b0 con 0x7f8408102360 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8407e00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84081019c0 0x7f840819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8407400640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8408102360 0x7f840819aa40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f83f4009230 tx=0x7f83f4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8404c00640 1 -- 172.21.15.114:0/67582990 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f403f030 con 0x7f8408102360 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8404c00640 1 -- 172.21.15.114:0/67582990 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83f4003ad0 con 0x7f8408102360 2024-09-19T06:49:28.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840819f740 con 0x7f8408102360 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f8404c00640 1 -- 172.21.15.114:0/67582990 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f4035070 con 0x7f8408102360 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.337+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84081a7080 con 0x7f8408102360 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.338+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8408101e20 con 0x7f8408102360 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.340+0000 7f8404c00640 1 -- 172.21.15.114:0/67582990 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f83f4042030 con 0x7f8408102360 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.341+0000 7f8404c00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f83ec0a32c0 0x7f83ec0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.342+0000 7f8404c00640 1 -- 172.21.15.114:0/67582990 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f83f40eff80 con 0x7f8408102360 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.342+0000 7f8407e00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f83ec0a32c0 0x7f83ec0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.342+0000 7f8407e00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f83ec0a32c0 0x7f83ec0a5780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f83f800bf50 tx=0x7f83f800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:28.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.345+0000 7f8404c00640 1 -- 172.21.15.114:0/67582990 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f83f40bc3b0 con 0x7f8408102360 2024-09-19T06:49:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.524+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84081a72b0 con 0x7f8408102360 2024-09-19T06:49:28.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.526+0000 7f8404c00640 1 -- 172.21.15.114:0/67582990 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24271 (secure 0 0 0) 0x7f83f40c06b0 con 0x7f8408102360 2024-09-19T06:49:28.528 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:28.528 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":91,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:49:23.496526+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f83ec0a32c0 msgr2=0x7f83ec0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f83ec0a32c0 0x7f83ec0a5780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f83f800bf50 tx=0x7f83f800b390 comp rx=0 tx=0).stop 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8408102360 msgr2=0x7f840819aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8408102360 0x7f840819aa40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f83f4009230 tx=0x7f83f4004270 comp rx=0 tx=0).stop 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 shutdown_connections 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f83ec0a32c0 0x7f83ec0a5780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84081079c0 0x7f840819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8408102360 0x7f840819aa40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 --2- 172.21.15.114:0/67582990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f84081019c0 0x7f840819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:28.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.528+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 >> 172.21.15.114:0/67582990 conn(0x7f84080fd6e0 msgr2=0x7f840810d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:28.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.529+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 shutdown_connections 2024-09-19T06:49:28.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:28.529+0000 7f840dc00640 1 -- 172.21.15.114:0/67582990 wait complete. 2024-09-19T06:49:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:28 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/67582990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:28 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/67582990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:29.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:28 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/67582990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:29.404 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:49:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:29 smithi114 ceph-mon[30820]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:29.770 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:29 smithi116 ceph-mon[35411]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:30.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:29 smithi118 ceph-mon[35527]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:30.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.325+0000 7fe321400640 1 -- 172.21.15.114:0/2726056671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074eb0 msgr2=0x7fe31c112650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:30.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.325+0000 7fe321400640 1 --2- 172.21.15.114:0/2726056671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074eb0 0x7fe31c112650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe308009900 tx=0x7fe30802a480 comp rx=0 tx=0).stop 2024-09-19T06:49:30.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.326+0000 7fe321400640 1 -- 172.21.15.114:0/2726056671 shutdown_connections 2024-09-19T06:49:30.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.326+0000 7fe321400640 1 --2- 172.21.15.114:0/2726056671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c112b90 0x7fe31c114f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.326+0000 7fe321400640 1 --2- 172.21.15.114:0/2726056671 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074eb0 0x7fe31c112650 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.326+0000 7fe321400640 1 --2- 172.21.15.114:0/2726056671 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe31c074590 0x7fe31c074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.326+0000 7fe321400640 1 -- 172.21.15.114:0/2726056671 >> 172.21.15.114:0/2726056671 conn(0x7fe31c0fd8c0 msgr2=0x7fe31c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:30.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.326+0000 7fe321400640 1 -- 172.21.15.114:0/2726056671 shutdown_connections 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.326+0000 7fe321400640 1 -- 172.21.15.114:0/2726056671 wait complete. 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.327+0000 7fe321400640 1 Processor -- start 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.327+0000 7fe321400640 1 -- start start 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe321400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074590 0x7fe31c1a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe321400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c074eb0 0x7fe31c1a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe321400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe31c112b90 0x7fe31c19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe321400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c118590 con 0x7fe31c074590 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe321400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c118410 con 0x7fe31c074eb0 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe321400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c118710 con 0x7fe31c112b90 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074590 0x7fe31c1a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe31c112b90 0x7fe31c19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe31c112b90 0x7fe31c19e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:45210/0 (socket says 172.21.15.114:45210) 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31b200640 1 -- 172.21.15.114:0/4179129300 learned_addr learned my addr 172.21.15.114:0/4179129300 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:30.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe319e00640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c074eb0 0x7fe31c1a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31b200640 1 -- 172.21.15.114:0/4179129300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c074eb0 msgr2=0x7fe31c1a4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31b200640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c074eb0 0x7fe31c1a4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31b200640 1 -- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074590 msgr2=0x7fe31c1a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31b200640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074590 0x7fe31c1a4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31b200640 1 -- 172.21.15.114:0/4179129300 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c19f000 con 0x7fe31c112b90 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.328+0000 7fe31a800640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074590 0x7fe31c1a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.329+0000 7fe31b200640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe31c112b90 0x7fe31c19e8d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe31400e970 tx=0x7fe31400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.329+0000 7fe313400640 1 -- 172.21.15.114:0/4179129300 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31400ccc0 con 0x7fe31c112b90 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.329+0000 7fe313400640 1 -- 172.21.15.114:0/4179129300 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe314005500 con 0x7fe31c112b90 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.329+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31c19f2f0 con 0x7fe31c112b90 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.329+0000 7fe313400640 1 -- 172.21.15.114:0/4179129300 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314010690 con 0x7fe31c112b90 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.329+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31c1ab5e0 con 0x7fe31c112b90 2024-09-19T06:49:30.371 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.330+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe31c074970 con 0x7fe31c112b90 2024-09-19T06:49:30.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.332+0000 7fe313400640 1 -- 172.21.15.114:0/4179129300 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fe3140046c0 con 0x7fe31c112b90 2024-09-19T06:49:30.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.333+0000 7fe313400640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe3000a3240 0x7fe3000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:30.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.333+0000 7fe313400640 1 -- 172.21.15.114:0/4179129300 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe3140d01c0 con 0x7fe31c112b90 2024-09-19T06:49:30.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.334+0000 7fe31a800640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe3000a3240 0x7fe3000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:30.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.334+0000 7fe31a800640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe3000a3240 0x7fe3000a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe31c19fa10 tx=0x7fe30c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:30.372 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.335+0000 7fe313400640 1 -- 172.21.15.114:0/4179129300 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe31409cc40 con 0x7fe31c112b90 2024-09-19T06:49:30.512 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.511+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe31c19fe00 con 0x7fe31c112b90 2024-09-19T06:49:30.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.513+0000 7fe313400640 1 -- 172.21.15.114:0/4179129300 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24271 (secure 0 0 0) 0x7fe31c19fe00 con 0x7fe31c112b90 2024-09-19T06:49:30.515 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:30.515 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":91,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:49:23.496526+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:49:30.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe3000a3240 msgr2=0x7fe3000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:30.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe3000a3240 0x7fe3000a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe31c19fa10 tx=0x7fe30c009500 comp rx=0 tx=0).stop 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe31c112b90 msgr2=0x7fe31c19e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe31c112b90 0x7fe31c19e8d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe31400e970 tx=0x7fe31400ee30 comp rx=0 tx=0).stop 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 shutdown_connections 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe3000a3240 0x7fe3000a5700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe31c112b90 0x7fe31c19e8d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe31c074eb0 0x7fe31c1a4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 --2- 172.21.15.114:0/4179129300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe31c074590 0x7fe31c1a4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 >> 172.21.15.114:0/4179129300 conn(0x7fe31c0fd8c0 msgr2=0x7fe31c0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:30.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 shutdown_connections 2024-09-19T06:49:30.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:30.516+0000 7fe321400640 1 -- 172.21.15.114:0/4179129300 wait complete. 2024-09-19T06:49:30.878 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:30 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/4179129300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:30 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/4179129300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:31.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:30 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/4179129300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:31.296 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:49:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:31 smithi114 ceph-mon[30820]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:31 smithi116 ceph-mon[35411]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:32.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:31 smithi118 ceph-mon[35527]: pgmap v311: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:32.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 -- 172.21.15.114:0/873467665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f981047e0 msgr2=0x7f5f98108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:32.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 --2- 172.21.15.114:0/873467665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f981047e0 0x7f5f98108830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009900 tx=0x7f5f8402a400 comp rx=0 tx=0).stop 2024-09-19T06:49:32.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 -- 172.21.15.114:0/873467665 shutdown_connections 2024-09-19T06:49:32.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 --2- 172.21.15.114:0/873467665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f98108f60 0x7f5f98110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 --2- 172.21.15.114:0/873467665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f981047e0 0x7f5f98108830 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 --2- 172.21.15.114:0/873467665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f98103e30 0x7f5f98104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 -- 172.21.15.114:0/873467665 >> 172.21.15.114:0/873467665 conn(0x7f5f980ff8d0 msgr2=0x7f5f98101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:32.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 -- 172.21.15.114:0/873467665 shutdown_connections 2024-09-19T06:49:32.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.203+0000 7f5f9da00640 1 -- 172.21.15.114:0/873467665 wait complete. 2024-09-19T06:49:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.204+0000 7f5f9da00640 1 Processor -- start 2024-09-19T06:49:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.204+0000 7f5f9da00640 1 -- start start 2024-09-19T06:49:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.204+0000 7f5f9da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f98103e30 0x7f5f9819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.204+0000 7f5f9da00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f981047e0 0x7f5f9819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.204+0000 7f5f9da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98108f60 0x7f5f9819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.204+0000 7f5f9da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5f98114010 con 0x7f5f98103e30 2024-09-19T06:49:32.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.204+0000 7f5f9da00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f98113e90 con 0x7f5f98108f60 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.204+0000 7f5f9da00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5f98114190 con 0x7f5f981047e0 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f98103e30 0x7f5f9819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f98103e30 0x7f5f9819a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60028/0 (socket says 172.21.15.114:60028) 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 -- 172.21.15.114:0/1848540694 learned_addr learned my addr 172.21.15.114:0/1848540694 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97400640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f981047e0 0x7f5f9819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f9d000640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98108f60 0x7f5f9819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 -- 172.21.15.114:0/1848540694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f981047e0 msgr2=0x7f5f9819a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f981047e0 0x7f5f9819a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 -- 172.21.15.114:0/1848540694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98108f60 msgr2=0x7f5f9819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98108f60 0x7f5f9819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 -- 172.21.15.114:0/1848540694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9819f440 con 0x7f5f98103e30 2024-09-19T06:49:32.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.205+0000 7f5f97e00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f98103e30 0x7f5f9819a490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5f8800ea60 tx=0x7f5f8800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:32.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.206+0000 7f5f94c00640 1 -- 172.21.15.114:0/1848540694 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8800cde0 con 0x7f5f98103e30 2024-09-19T06:49:32.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.206+0000 7f5f94c00640 1 -- 172.21.15.114:0/1848540694 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f880035f0 con 0x7f5f98103e30 2024-09-19T06:49:32.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.206+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f9819f730 con 0x7f5f98103e30 2024-09-19T06:49:32.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.206+0000 7f5f94c00640 1 -- 172.21.15.114:0/1848540694 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88010670 con 0x7f5f98103e30 2024-09-19T06:49:32.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.206+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f981a7010 con 0x7f5f98103e30 2024-09-19T06:49:32.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.207+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f98104210 con 0x7f5f98103e30 2024-09-19T06:49:32.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.209+0000 7f5f94c00640 1 -- 172.21.15.114:0/1848540694 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f5f880037a0 con 0x7f5f98103e30 2024-09-19T06:49:32.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.210+0000 7f5f94c00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5f7c0a3240 0x7f5f7c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:32.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.210+0000 7f5f94c00640 1 -- 172.21.15.114:0/1848540694 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f5f880cfe60 con 0x7f5f98103e30 2024-09-19T06:49:32.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.210+0000 7f5f97400640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5f7c0a3240 0x7f5f7c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:32.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.211+0000 7f5f97400640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5f7c0a3240 0x7f5f7c0a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f84000c00 tx=0x7f5f84005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:32.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.214+0000 7f5f94c00640 1 -- 172.21.15.114:0/1848540694 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5f8809c8e0 con 0x7f5f98103e30 2024-09-19T06:49:32.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.426+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f9819b7f0 con 0x7f5f98103e30 2024-09-19T06:49:32.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.427+0000 7f5f94c00640 1 -- 172.21.15.114:0/1848540694 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11863 (secure 0 0 0) 0x7f5f880bae20 con 0x7f5f98103e30 2024-09-19T06:49:32.429 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:32.429 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"btime":"2024-09-19T06:49:24:427295+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14520,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24482,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-19T06:49:11.450651+0000","modified":"2024-09-19T06:49:23.496663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24538},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24538,"qdb_cluster":[24538]},"id":1}]} 2024-09-19T06:49:32.429 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 17 2024-09-19T06:49:32.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5f7c0a3240 msgr2=0x7f5f7c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5f7c0a3240 0x7f5f7c0a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f84000c00 tx=0x7f5f84005b50 comp rx=0 tx=0).stop 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f98103e30 msgr2=0x7f5f9819a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f98103e30 0x7f5f9819a490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5f8800ea60 tx=0x7f5f8800ef20 comp rx=0 tx=0).stop 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 shutdown_connections 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5f7c0a3240 0x7f5f7c0a5700 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98108f60 0x7f5f9819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f981047e0 0x7f5f9819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 --2- 172.21.15.114:0/1848540694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f98103e30 0x7f5f9819a490 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 >> 172.21.15.114:0/1848540694 conn(0x7f5f980ff8d0 msgr2=0x7f5f98100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.429+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 shutdown_connections 2024-09-19T06:49:32.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:32.430+0000 7f5f9da00640 1 -- 172.21.15.114:0/1848540694 wait complete. 2024-09-19T06:49:32.848 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:32 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1848540694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:32 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1848540694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:33.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:32 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1848540694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:49:33.339 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:49:33.707 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:33.875 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:33 smithi114 ceph-mon[30820]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:33 smithi116 ceph-mon[35411]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:34.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:33 smithi118 ceph-mon[35527]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:34.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.268+0000 7f0b6d600640 1 -- 172.21.15.114:0/3046764783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68102790 msgr2=0x7f0b681067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:34.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.268+0000 7f0b6d600640 1 --2- 172.21.15.114:0/3046764783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68102790 0x7f0b681067e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009900 tx=0x7f0b5002a3f0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.269+0000 7f0b6d600640 1 -- 172.21.15.114:0/3046764783 shutdown_connections 2024-09-19T06:49:34.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.269+0000 7f0b6d600640 1 --2- 172.21.15.114:0/3046764783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b68106f10 0x7f0b6810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.269+0000 7f0b6d600640 1 --2- 172.21.15.114:0/3046764783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68102790 0x7f0b681067e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.269+0000 7f0b6d600640 1 --2- 172.21.15.114:0/3046764783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68101de0 0x7f0b681021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.269+0000 7f0b6d600640 1 -- 172.21.15.114:0/3046764783 >> 172.21.15.114:0/3046764783 conn(0x7f0b680fd8c0 msgr2=0x7f0b680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:34.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.269+0000 7f0b6d600640 1 -- 172.21.15.114:0/3046764783 shutdown_connections 2024-09-19T06:49:34.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.269+0000 7f0b6d600640 1 -- 172.21.15.114:0/3046764783 wait complete. 2024-09-19T06:49:34.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b6d600640 1 Processor -- start 2024-09-19T06:49:34.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b6d600640 1 -- start start 2024-09-19T06:49:34.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b6d600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68101de0 0x7f0b6819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:34.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b6d600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b68102790 0x7f0b6819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:34.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b6d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b6d600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0b68111fc0 con 0x7f0b68106f10 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b6d600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0b68111e40 con 0x7f0b68101de0 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b6d600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0b68112140 con 0x7f0b68102790 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b67400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b67400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60048/0 (socket says 172.21.15.114:60048) 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b67400640 1 -- 172.21.15.114:0/2200861317 learned_addr learned my addr 172.21.15.114:0/2200861317 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.270+0000 7f0b66a00640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68101de0 0x7f0b6819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.271+0000 7f0b5e000640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b68102790 0x7f0b6819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.271+0000 7f0b66a00640 1 -- 172.21.15.114:0/2200861317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b68102790 msgr2=0x7f0b6819abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.271+0000 7f0b66a00640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b68102790 0x7f0b6819abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.271+0000 7f0b66a00640 1 -- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68106f10 msgr2=0x7f0b6819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.271+0000 7f0b66a00640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.271+0000 7f0b66a00640 1 -- 172.21.15.114:0/2200861317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b6819f630 con 0x7f0b68101de0 2024-09-19T06:49:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.271+0000 7f0b67400640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.271+0000 7f0b66a00640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68101de0 0x7f0b6819a680 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800e9f0 tx=0x7f0b5800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.272+0000 7f0b5fe00640 1 -- 172.21.15.114:0/2200861317 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5800ccc0 con 0x7f0b68101de0 2024-09-19T06:49:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.272+0000 7f0b5fe00640 1 -- 172.21.15.114:0/2200861317 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b58010910 con 0x7f0b68101de0 2024-09-19T06:49:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.272+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b6819f920 con 0x7f0b68101de0 2024-09-19T06:49:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.272+0000 7f0b5fe00640 1 -- 172.21.15.114:0/2200861317 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58018960 con 0x7f0b68101de0 2024-09-19T06:49:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.272+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b681a7200 con 0x7f0b68101de0 2024-09-19T06:49:34.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.273+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b681021c0 con 0x7f0b68101de0 2024-09-19T06:49:34.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.275+0000 7f0b5fe00640 1 -- 172.21.15.114:0/2200861317 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f0b58010ac0 con 0x7f0b68101de0 2024-09-19T06:49:34.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.276+0000 7f0b5fe00640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0b4c0a3240 0x7f0b4c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:34.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.276+0000 7f0b5fe00640 1 -- 172.21.15.114:0/2200861317 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f0b580d00b0 con 0x7f0b68101de0 2024-09-19T06:49:34.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.276+0000 7f0b5e000640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0b4c0a3240 0x7f0b4c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:34.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.277+0000 7f0b5e000640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0b4c0a3240 0x7f0b4c0a5700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b50000c00 tx=0x7f0b50005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:34.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.278+0000 7f0b5fe00640 1 -- 172.21.15.114:0/2200861317 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f0b5809c4e0 con 0x7f0b68101de0 2024-09-19T06:49:34.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.461+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0b681a74a0 con 0x7f0b68101de0 2024-09-19T06:49:34.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.463+0000 7f0b5fe00640 1 -- 172.21.15.114:0/2200861317 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24271 (secure 0 0 0) 0x7f0b580a07e0 con 0x7f0b68101de0 2024-09-19T06:49:34.465 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:34.465 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":91,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:49:23.496526+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:49:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.465+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0b4c0a3240 msgr2=0x7f0b4c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.465+0000 7f0b6d600640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0b4c0a3240 0x7f0b4c0a5700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0b50000c00 tx=0x7f0b50005b50 comp rx=0 tx=0).stop 2024-09-19T06:49:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.465+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68101de0 msgr2=0x7f0b6819a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.465+0000 7f0b6d600640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68101de0 0x7f0b6819a680 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800e9f0 tx=0x7f0b5800eeb0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.465+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 shutdown_connections 2024-09-19T06:49:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.465+0000 7f0b6d600640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0b4c0a3240 0x7f0b4c0a5700 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.465+0000 7f0b6d600640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0b68106f10 0x7f0b6819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.465+0000 7f0b6d600640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b68102790 0x7f0b6819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.466+0000 7f0b6d600640 1 --2- 172.21.15.114:0/2200861317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68101de0 0x7f0b6819a680 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:34.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.466+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 >> 172.21.15.114:0/2200861317 conn(0x7f0b680fd8c0 msgr2=0x7f0b680feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:34.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.466+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 shutdown_connections 2024-09-19T06:49:34.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:34.466+0000 7f0b6d600640 1 -- 172.21.15.114:0/2200861317 wait complete. 2024-09-19T06:49:34.820 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:34 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2200861317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:34 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2200861317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:35.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:34 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2200861317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:35.349 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-19T06:49:11.450651+0000', 'modified': '2024-09-19T06:49:23.496663+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24538}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.118:6839/345383252', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6838', 'nonce': 345383252}, {'type': 'v1', 'addr': '172.21.15.118:6839', 'nonce': 345383252}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24538, 'qdb_cluster': [24538]} 2024-09-19T06:49:35.349 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-19T06:49:35.350 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-19T06:49:35.719 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:35.857 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:35 smithi114 ceph-mon[30820]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:35 smithi116 ceph-mon[35411]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:36.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:35 smithi118 ceph-mon[35527]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 -- 172.21.15.114:0/2716937884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8104800 msgr2=0x7f6fe8108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 --2- 172.21.15.114:0/2716937884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8104800 0x7f6fe8108960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009900 tx=0x7f6fd002a410 comp rx=0 tx=0).stop 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 -- 172.21.15.114:0/2716937884 shutdown_connections 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 --2- 172.21.15.114:0/2716937884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fe8109090 0x7f6fe8110c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 --2- 172.21.15.114:0/2716937884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8104800 0x7f6fe8108960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 --2- 172.21.15.114:0/2716937884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe8103e50 0x7f6fe8104230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 -- 172.21.15.114:0/2716937884 >> 172.21.15.114:0/2716937884 conn(0x7f6fe80ff8b0 msgr2=0x7f6fe8101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 -- 172.21.15.114:0/2716937884 shutdown_connections 2024-09-19T06:49:36.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.321+0000 7f6fece00640 1 -- 172.21.15.114:0/2716937884 wait complete. 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.322+0000 7f6fece00640 1 Processor -- start 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.322+0000 7f6fece00640 1 -- start start 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fece00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fe8103e50 0x7f6fe819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fece00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe8104800 0x7f6fe819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fece00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8109090 0x7f6fe819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fece00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8114090 con 0x7f6fe8104800 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fece00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8113f10 con 0x7f6fe8109090 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fece00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8114210 con 0x7f6fe8103e50 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fe8103e50 0x7f6fe819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fdfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe8104800 0x7f6fe819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fe8103e50 0x7f6fe819a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:45274/0 (socket says 172.21.15.114:45274) 2024-09-19T06:49:36.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7400640 1 -- 172.21.15.114:0/3581269561 learned_addr learned my addr 172.21.15.114:0/3581269561 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:36.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8109090 0x7f6fe819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7400640 1 -- 172.21.15.114:0/3581269561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8109090 msgr2=0x7f6fe819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7400640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8109090 0x7f6fe819ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7400640 1 -- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe8104800 msgr2=0x7f6fe819ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7400640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe8104800 0x7f6fe819ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.323+0000 7f6fe7400640 1 -- 172.21.15.114:0/3581269561 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe819f5e0 con 0x7f6fe8103e50 2024-09-19T06:49:36.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.324+0000 7f6fe7400640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fe8103e50 0x7f6fe819a630 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800e9f0 tx=0x7f6fd800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:36.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.324+0000 7f6fe4c00640 1 -- 172.21.15.114:0/3581269561 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd800cd80 con 0x7f6fe8103e50 2024-09-19T06:49:36.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.324+0000 7f6fe4c00640 1 -- 172.21.15.114:0/3581269561 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fd8010910 con 0x7f6fe8103e50 2024-09-19T06:49:36.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.324+0000 7f6fece00640 1 -- 172.21.15.114:0/3581269561 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe819f8d0 con 0x7f6fe8103e50 2024-09-19T06:49:36.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.324+0000 7f6fe4c00640 1 -- 172.21.15.114:0/3581269561 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8018990 con 0x7f6fe8103e50 2024-09-19T06:49:36.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.324+0000 7f6fece00640 1 -- 172.21.15.114:0/3581269561 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81a71b0 con 0x7f6fe8103e50 2024-09-19T06:49:36.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.327+0000 7f6ff0e90740 1 Processor -- start 2024-09-19T06:49:36.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.327+0000 7f6fe4c00640 1 -- 172.21.15.114:0/3581269561 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f6fd8010ac0 con 0x7f6fe8103e50 2024-09-19T06:49:36.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.327+0000 7f6ff0e90740 1 -- start start 2024-09-19T06:49:36.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f6c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f6ce00 0x556ce1f77180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f776c0 0x556ce1f79ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ce1ed20b0 con 0x556ce1f6c4e0 2024-09-19T06:49:36.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ce1e79ab0 con 0x556ce1f6ce00 2024-09-19T06:49:36.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ce1ece070 con 0x556ce1f776c0 2024-09-19T06:49:36.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f6c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f6c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60056/0 (socket says 172.21.15.114:60056) 2024-09-19T06:49:36.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7e00640 1 -- 172.21.15.114:0/4063970598 learned_addr learned my addr 172.21.15.114:0/4063970598 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:36.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7400640 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f6ce00 0x556ce1f77180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fdfe00640 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f776c0 0x556ce1f79ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7e00640 1 -- 172.21.15.114:0/4063970598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f776c0 msgr2=0x556ce1f79ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f776c0 0x556ce1f79ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7e00640 1 -- 172.21.15.114:0/4063970598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f6ce00 msgr2=0x556ce1f77180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f6ce00 0x556ce1f77180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.328+0000 7f6fe7e00640 1 -- 172.21.15.114:0/4063970598 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ce1e864a0 con 0x556ce1f6c4e0 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.329+0000 7f6fe4c00640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6fcc0a32c0 0x7f6fcc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.329+0000 7f6fe4c00640 1 -- 172.21.15.114:0/3581269561 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6fd80cfe20 con 0x7f6fe8103e50 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.329+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f6c8c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009f50 tx=0x7f6fe00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51de867609229aaf server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.329+0000 7f6fe7400640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6fcc0a32c0 0x7f6fcc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.329+0000 7f6fdcc00640 1 -- 172.21.15.114:0/4063970598 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0004d20 con 0x556ce1f6c4e0 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.329+0000 7f6fdcc00640 1 -- 172.21.15.114:0/4063970598 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe0004050 con 0x556ce1f6c4e0 2024-09-19T06:49:36.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.329+0000 7f6fdcc00640 1 -- 172.21.15.114:0/4063970598 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0005220 con 0x556ce1f6c4e0 2024-09-19T06:49:36.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 -- 172.21.15.114:0/4063970598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 msgr2=0x556ce1f6c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f6c8c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009f50 tx=0x7f6fe00118d0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6fe7400640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6fcc0a32c0 0x7f6fcc0a5780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8098c20 tx=0x7f6fd80d3860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:36.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 -- 172.21.15.114:0/4063970598 shutdown_connections 2024-09-19T06:49:36.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f776c0 0x556ce1f79ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f6ce00 0x556ce1f77180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/4063970598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f6c8c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 -- 172.21.15.114:0/4063970598 >> 172.21.15.114:0/4063970598 conn(0x556ce1f68a70 msgr2=0x556ce1f68e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 -- 172.21.15.114:0/4063970598 shutdown_connections 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.330+0000 7f6ff0e90740 1 -- 172.21.15.114:0/4063970598 wait complete. 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 1 -- start start 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 1 Processor -- start 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f745d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f74b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f75050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ce1f7a330 con 0x556ce1f6ce00 2024-09-19T06:49:36.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ce1e5a920 con 0x556ce1f776c0 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ce1f7a4b0 con 0x556ce1f6c4e0 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.331+0000 7f6ff0e90740 10 client.0 _ll_register_callbacks cb 0x556ce1e7e8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fdfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f74b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:45290/0 (socket says 172.21.15.114:45290) 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fe7e00640 1 -- 172.21.15.114:0/1925145050 learned_addr learned my addr 172.21.15.114:0/1925145050 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f75050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fdfe00640 1 -- 172.21.15.114:0/1925145050 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 msgr2=0x556ce1f745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fdfe00640 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fdfe00640 1 -- 172.21.15.114:0/1925145050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 msgr2=0x556ce1f75050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fdfe00640 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f75050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fdfe00640 1 -- 172.21.15.114:0/1925145050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ce1c432b0 con 0x556ce1f6ce00 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fdfe00640 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f74b10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0005940 tx=0x7f6fd00317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc01dd73ed16a21 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.332+0000 7f6fd5600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.333+0000 7f6fd5600640 1 -- 172.21.15.114:0/1925145050 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0035040 con 0x556ce1f6ce00 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.333+0000 7f6fd5600640 1 -- 172.21.15.114:0/1925145050 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6fd0004240 con 0x556ce1f6ce00 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.333+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1925145050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fe0004050 con 0x556ce1f6ce00 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.333+0000 7f6fd5600640 1 -- 172.21.15.114:0/1925145050 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f6fd0031a10 con 0x556ce1f6ce00 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.333+0000 7f6ff0e90740 10 client.14622 fetch_fsmap learned FSMap version 17 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.333+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1925145050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x556ce1f86060 con 0x556ce1f6ce00 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.333+0000 7f6fd5600640 1 -- 172.21.15.114:0/1925145050 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe0004050 con 0x556ce1f6ce00 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.333+0000 7f6fd5600640 1 -- 172.21.15.114:0/1925145050 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f6fd004d4f0 con 0x556ce1f6ce00 2024-09-19T06:49:36.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.334+0000 7f6ff0e90740 10 client.14622 fetch_fsmap finished waiting for FSMap version 17 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.334+0000 7f6ff0e90740 10 client.14622 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.334+0000 7f6ff0e90740 20 client.14622 populate_metadata read hostname 'smithi114' 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.334+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x556ce1f88630 0x556ce1f89aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.334+0000 7f6ff0e90740 4 client.14622 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.334+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1925145050 --> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ce1edf620 con 0x556ce1f88630 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.334+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x556ce1f88630 0x556ce1f89aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.335+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x556ce1f88630 0x556ce1f89aa0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b372466959131c41 server_cookie=a23495054b775604 in_seq=0 out_seq=0 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.335+0000 7f6fd5600640 10 client.14622 ms_handle_connect on v2:172.21.15.118:6838/345383252 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.335+0000 7f6fd5600640 1 -- 172.21.15.114:0/1925145050 <== mds.0 v2:172.21.15.118:6838/345383252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556ce1edf620 con 0x556ce1f88630 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.335+0000 7f6fd5600640 10 client.14622 handle_command_reply: tid=0 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 client.14622 shutdown 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1925145050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 msgr2=0x556ce1f74b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f74b10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0005940 tx=0x7f6fd00317e0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1925145050 shutdown_connections 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x556ce1f88630 0x556ce1f89aa0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f75050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f74b10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1925145050 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6fd5600640 0 client.14622 ms_handle_reset on v2:172.21.15.118:6838/345383252 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1925145050 >> 172.21.15.114:0/1925145050 conn(0x556ce1f68a70 msgr2=0x556ce1f67bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1925145050 shutdown_connections 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1925145050 wait complete. 2024-09-19T06:49:36.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.336+0000 7f6ff0e90740 20 client.14622 trim_cache size 0 max 16384 2024-09-19T06:49:36.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6ff0e90740 1 Processor -- start 2024-09-19T06:49:36.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6ff0e90740 1 -- start start 2024-09-19T06:49:36.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f74ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f76540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f91d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ce1f87de0 con 0x556ce1f6ce00 2024-09-19T06:49:36.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ce1f884b0 con 0x556ce1f776c0 2024-09-19T06:49:36.357 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ce1f88b80 con 0x556ce1f6c4e0 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.355+0000 7f6fdfe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f76540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f74ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f74ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:45294/0 (socket says 172.21.15.114:45294) 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 -- 172.21.15.114:0/1096521012 learned_addr learned my addr 172.21.15.114:0/1096521012 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/1096521012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f91d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 -- 172.21.15.114:0/1096521012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 msgr2=0x556ce1f91d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/1096521012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f91d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 -- 172.21.15.114:0/1096521012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 msgr2=0x556ce1f76540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:49:36.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/1096521012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f76540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 -- 172.21.15.114:0/1096521012 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ce1f72b20 con 0x556ce1f6c4e0 2024-09-19T06:49:36.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/1096521012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f74ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00042c0 tx=0x7f6fe0005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91be005e5b21ebb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:36.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.356+0000 7f6fd7e00640 1 -- 172.21.15.114:0/1096521012 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0034020 con 0x556ce1f6c4e0 2024-09-19T06:49:36.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6fd7e00640 1 -- 172.21.15.114:0/1096521012 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe0004970 con 0x556ce1f6c4e0 2024-09-19T06:49:36.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6fd7e00640 1 -- 172.21.15.114:0/1096521012 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0027aa0 con 0x556ce1f6c4e0 2024-09-19T06:49:36.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1096521012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 msgr2=0x556ce1f74ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1096521012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f74ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00042c0 tx=0x7f6fe0005fd0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1096521012 shutdown_connections 2024-09-19T06:49:36.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1096521012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f91d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1096521012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6ce00 0x556ce1f76540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/1096521012 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6c4e0 0x556ce1f74ea0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1096521012 >> 172.21.15.114:0/1096521012 conn(0x556ce1f68a70 msgr2=0x556ce1f75e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1096521012 shutdown_connections 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.357+0000 7f6ff0e90740 1 -- 172.21.15.114:0/1096521012 wait complete. 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 1 -- start start 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 1 Processor -- start 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f8bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6ce00 0x556ce1f8c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f8c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ce1f89920 con 0x556ce1f6c4e0 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ce1f89250 con 0x556ce1f776c0 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556ce1f89ff0 con 0x556ce1f6ce00 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.358+0000 7f6ff0e90740 10 client.0 _ll_register_callbacks cb 0x7f6fd0005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:49:36.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f8bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fdfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6ce00 0x556ce1f8c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f8bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60094/0 (socket says 172.21.15.114:60094) 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fe7e00640 1 -- 172.21.15.114:0/562858870 learned_addr learned my addr 172.21.15.114:0/562858870 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f8c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fdfe00640 1 -- 172.21.15.114:0/562858870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 msgr2=0x556ce1f8c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fdfe00640 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f8c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fdfe00640 1 -- 172.21.15.114:0/562858870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 msgr2=0x556ce1f8bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fdfe00640 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f8bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fdfe00640 1 -- 172.21.15.114:0/562858870 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ce1f90750 con 0x556ce1f6ce00 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f8bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fdfe00640 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6ce00 0x556ce1f8c220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6fd002a930 tx=0x7f6fd004dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e3ac67c90a1cc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fd5600640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-09-19T06:49:36.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6fd5600640 1 -- 172.21.15.114:0/562858870 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0035070 con 0x556ce1f6ce00 2024-09-19T06:49:36.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.359+0000 7f6ff0e90740 1 -- 172.21.15.114:0/562858870 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fe0004970 con 0x556ce1f6ce00 2024-09-19T06:49:36.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.360+0000 7f6fd5600640 1 -- 172.21.15.114:0/562858870 <== mon.1 v2:172.21.15.118:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6fd0004240 con 0x556ce1f6ce00 2024-09-19T06:49:36.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.360+0000 7f6fd5600640 1 -- 172.21.15.114:0/562858870 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f6fd0009520 con 0x556ce1f6ce00 2024-09-19T06:49:36.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.360+0000 7f6ff0e90740 10 client.24670 fetch_fsmap learned FSMap version 17 2024-09-19T06:49:36.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.360+0000 7f6ff0e90740 1 -- 172.21.15.114:0/562858870 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x556ce1ff32b0 con 0x556ce1f6ce00 2024-09-19T06:49:36.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.360+0000 7f6fd5600640 1 -- 172.21.15.114:0/562858870 <== mon.1 v2:172.21.15.118:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe0004970 con 0x556ce1f6ce00 2024-09-19T06:49:36.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.360+0000 7f6fd5600640 1 -- 172.21.15.114:0/562858870 <== mon.1 v2:172.21.15.118:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f6fd00046c0 con 0x556ce1f6ce00 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.360+0000 7f6ff0e90740 10 client.24670 fetch_fsmap finished waiting for FSMap version 17 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.361+0000 7f6ff0e90740 10 client.24670 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.361+0000 7f6ff0e90740 20 client.24670 populate_metadata read hostname 'smithi114' 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.361+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x556ce1ff7e90 0x556ce1ff6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.361+0000 7f6ff0e90740 4 client.24670 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.361+0000 7f6ff0e90740 1 -- 172.21.15.114:0/562858870 --> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] -- command(tid 0: {"prefix": "status"}) -- 0x556ce1f69620 con 0x556ce1ff7e90 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.361+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x556ce1ff7e90 0x556ce1ff6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.361+0000 7f6fe7e00640 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x556ce1ff7e90 0x556ce1ff6ba0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2b223ef0fe518b2 server_cookie=76b3929bb3eff719 in_seq=0 out_seq=0 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.361+0000 7f6fd5600640 10 client.24670 ms_handle_connect on v2:172.21.15.118:6838/345383252 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6fd5600640 1 -- 172.21.15.114:0/562858870 <== mds.0 v2:172.21.15.118:6838/345383252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556ce1f69620 con 0x556ce1ff7e90 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6fd5600640 10 client.24670 handle_command_reply: tid=0 2024-09-19T06:49:36.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 client.24670 shutdown 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 -- 172.21.15.114:0/562858870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6ce00 msgr2=0x556ce1f8c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6ce00 0x556ce1f8c220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6fd002a930 tx=0x7f6fd004dcc0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 -- 172.21.15.114:0/562858870 shutdown_connections 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x556ce1ff7e90 0x556ce1ff6ba0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ce1f776c0 0x556ce1f8c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ce1f6ce00 0x556ce1f8c220 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 --2- 172.21.15.114:0/562858870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ce1f6c4e0 0x556ce1f8bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 -- 172.21.15.114:0/562858870 >> 172.21.15.114:0/562858870 conn(0x556ce1f68a70 msgr2=0x556ce1f67bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6fd5600640 0 client.24670 ms_handle_reset on v2:172.21.15.118:6838/345383252 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 -- 172.21.15.114:0/562858870 shutdown_connections 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 1 -- 172.21.15.114:0/562858870 wait complete. 2024-09-19T06:49:36.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.362+0000 7f6ff0e90740 20 client.24670 trim_cache size 0 max 16384 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24538, 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 23.997211907000001, 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 17, 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 91, 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 91, 2024-09-19T06:49:36.368 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 48.452955707999998 2024-09-19T06:49:36.369 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:49:36.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 -- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6fcc0a32c0 msgr2=0x7f6fcc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6fcc0a32c0 0x7f6fcc0a5780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8098c20 tx=0x7f6fd80d3860 comp rx=0 tx=0).stop 2024-09-19T06:49:36.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 -- 172.21.15.114:0/3581269561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fe8103e50 msgr2=0x7f6fe819a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:36.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fe8103e50 0x7f6fe819a630 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fd800e9f0 tx=0x7f6fd800eeb0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 -- 172.21.15.114:0/3581269561 shutdown_connections 2024-09-19T06:49:36.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6fcc0a32c0 0x7f6fcc0a5780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8109090 0x7f6fe819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fe8104800 0x7f6fe819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 --2- 172.21.15.114:0/3581269561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6fe8103e50 0x7f6fe819a630 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:36.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 -- 172.21.15.114:0/3581269561 >> 172.21.15.114:0/3581269561 conn(0x7f6fe80ff8b0 msgr2=0x7f6fe8100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:36.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 -- 172.21.15.114:0/3581269561 shutdown_connections 2024-09-19T06:49:36.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:36.363+0000 7f6fece00640 1 -- 172.21.15.114:0/3581269561 wait complete. 2024-09-19T06:49:37.138 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-19T06:49:37.139 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:37.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-19T06:49:37.503 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:38.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 -- 172.21.15.114:0/754047738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60101de0 msgr2=0x7fec601021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:38.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 --2- 172.21.15.114:0/754047738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60101de0 0x7fec601021c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fec50009900 tx=0x7fec5002a3f0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 -- 172.21.15.114:0/754047738 shutdown_connections 2024-09-19T06:49:38.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 --2- 172.21.15.114:0/754047738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec60106f10 0x7fec6010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 --2- 172.21.15.114:0/754047738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec60102790 0x7fec601067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 --2- 172.21.15.114:0/754047738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60101de0 0x7fec601021c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 -- 172.21.15.114:0/754047738 >> 172.21.15.114:0/754047738 conn(0x7fec600fd8c0 msgr2=0x7fec600ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:38.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 -- 172.21.15.114:0/754047738 shutdown_connections 2024-09-19T06:49:38.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.080+0000 7fec64c00640 1 -- 172.21.15.114:0/754047738 wait complete. 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.081+0000 7fec64c00640 1 Processor -- start 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.081+0000 7fec64c00640 1 -- start start 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.081+0000 7fec64c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec60101de0 0x7fec60198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.081+0000 7fec64c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec60102790 0x7fec60198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.081+0000 7fec64c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60106f10 0x7fec6019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.081+0000 7fec64c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fec60111f70 con 0x7fec60106f10 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.081+0000 7fec64c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec60111df0 con 0x7fec60101de0 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.081+0000 7fec64c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fec601120f0 con 0x7fec60102790 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec60102790 0x7fec60198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60106f10 0x7fec6019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec60102790 0x7fec60198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:37896/0 (socket says 172.21.15.114:37896) 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5d400640 1 -- 172.21.15.114:0/2574954904 learned_addr learned my addr 172.21.15.114:0/2574954904 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:38.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5de00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec60101de0 0x7fec60198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5e800640 1 -- 172.21.15.114:0/2574954904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec60102790 msgr2=0x7fec60198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5e800640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec60102790 0x7fec60198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5e800640 1 -- 172.21.15.114:0/2574954904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec60101de0 msgr2=0x7fec60198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5e800640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec60101de0 0x7fec60198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5e800640 1 -- 172.21.15.114:0/2574954904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6019d400 con 0x7fec60106f10 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5d400640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec60102790 0x7fec60198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.082+0000 7fec5e800640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60106f10 0x7fec6019cd20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fec5800ec50 tx=0x7fec5800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.083+0000 7fec56a00640 1 -- 172.21.15.114:0/2574954904 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58003b70 con 0x7fec60106f10 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.083+0000 7fec56a00640 1 -- 172.21.15.114:0/2574954904 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec58003d10 con 0x7fec60106f10 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.083+0000 7fec56a00640 1 -- 172.21.15.114:0/2574954904 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58010890 con 0x7fec60106f10 2024-09-19T06:49:38.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.083+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec6019d6f0 con 0x7fec60106f10 2024-09-19T06:49:38.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.083+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec601a4fd0 con 0x7fec60106f10 2024-09-19T06:49:38.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.084+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec601021c0 con 0x7fec60106f10 2024-09-19T06:49:38.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.086+0000 7fec56a00640 1 -- 172.21.15.114:0/2574954904 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fec58003fc0 con 0x7fec60106f10 2024-09-19T06:49:38.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.087+0000 7fec56a00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec440a32c0 0x7fec440a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:38.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.087+0000 7fec56a00640 1 -- 172.21.15.114:0/2574954904 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fec580d00a0 con 0x7fec60106f10 2024-09-19T06:49:38.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.087+0000 7fec5de00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec440a32c0 0x7fec440a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:38.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.088+0000 7fec5de00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec440a32c0 0x7fec440a5780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fec50009230 tx=0x7fec500058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:38.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.091+0000 7fec56a00640 1 -- 172.21.15.114:0/2574954904 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fec5809cb20 con 0x7fec60106f10 2024-09-19T06:49:38.299 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.299+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fec60109b60 con 0x7fec60106f10 2024-09-19T06:49:38.300 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:38 smithi114 ceph-mon[30820]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:38.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.300+0000 7fec56a00640 1 -- 172.21.15.114:0/2574954904 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fec5809c420 con 0x7fec60106f10 2024-09-19T06:49:38.301 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-19T06:49:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec440a32c0 msgr2=0x7fec440a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec440a32c0 0x7fec440a5780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fec50009230 tx=0x7fec500058d0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60106f10 msgr2=0x7fec6019cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60106f10 0x7fec6019cd20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fec5800ec50 tx=0x7fec5800c6a0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.303 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 shutdown_connections 2024-09-19T06:49:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec440a32c0 0x7fec440a5780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec60106f10 0x7fec6019cd20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec60102790 0x7fec60198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.302+0000 7fec64c00640 1 --2- 172.21.15.114:0/2574954904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec60101de0 0x7fec60198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.303+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 >> 172.21.15.114:0/2574954904 conn(0x7fec600fd8c0 msgr2=0x7fec600fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.303+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 shutdown_connections 2024-09-19T06:49:38.304 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:38.303+0000 7fec64c00640 1 -- 172.21.15.114:0/2574954904 wait complete. 2024-09-19T06:49:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:38 smithi116 ceph-mon[35411]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:38.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:38 smithi118 ceph-mon[35527]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:39.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:39 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2574954904' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-19T06:49:39.219 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-19T06:49:39.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:39.220 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-09-19T06:49:39.256 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-19T06:49:39.256 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-19T06:49:39.256 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-09-19T06:49:39.256 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-19T06:49:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42223sec preferred_lft 42223sec 2024-09-19T06:49:39.258 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link noprefixroute 2024-09-19T06:49:39.258 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-19T06:49:39.258 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-19T06:49:39.258 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:49:39.258 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:49:39.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-09-19T06:49:39.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-09-19T06:49:39.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-09-19T06:49:39.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-19T06:49:39.259 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:49:39.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:49:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:39 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2574954904' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T06:49:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:39 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2574954904' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T06:49:39.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:49:39.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:39.690 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-19T06:49:39.746 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-09-19T06:49:39.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:39.750 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-09-19T06:49:39.800 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-09-19T06:49:39.800 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-19T06:49:39.801 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-19T06:49:39.801 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-19T06:49:39.801 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-19T06:49:39.801 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:49:39.801 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:49:39.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-19T06:49:39.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-19T06:49:39.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-19T06:49:39.802 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:49:39.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:49:40.404 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:40 smithi114 ceph-mon[30820]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:40.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:49:40.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:40.414 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-09-19T06:49:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:40 smithi116 ceph-mon[35411]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:40.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:40.451 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-09-19T06:49:40.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:40 smithi118 ceph-mon[35527]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:40.527 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:49:40.527 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:49:40.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-19T06:49:40.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-19T06:49:40.527 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:49:40.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:49:40.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:49:40.706 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-19T06:49:40.706 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:49:40.706 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:49:40.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-19T06:49:40.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-19T06:49:40.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-19T06:49:40.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-19T06:49:40.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-19T06:49:40.707 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:49:40.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:49:40.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:49:40.948 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:49:40.948 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:49:40.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-09-19T06:49:40.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-09-19T06:49:40.948 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:49:41.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:49:41.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:49:41.114 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-19T06:49:41.114 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T06:49:41.115 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:41.150 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-19T06:49:41.150 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T06:49:41.151 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:41.225 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2024-09-19T06:49:41.322 DEBUG:teuthology.orchestra.run.smithi114:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T06:49:41.361 INFO:teuthology.orchestra.run.smithi114.stdout:/proc 2024-09-19T06:49:41.361 INFO:teuthology.orchestra.run.smithi114.stdout:/sys 2024-09-19T06:49:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:/dev 2024-09-19T06:49:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/security 2024-09-19T06:49:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/shm 2024-09-19T06:49:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/pts 2024-09-19T06:49:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:/run 2024-09-19T06:49:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/cgroup 2024-09-19T06:49:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/pstore 2024-09-19T06:49:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/bpf 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/ 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/selinux 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/hugepages 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/mqueue 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/debug 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/tracing 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/fuse/connections 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/config 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-sysctl.service 2024-09-19T06:49:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/nfs/rpc_pipefs 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/1000 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/ceph 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/560568456d473cc1ff48de7c9dee0358a8e40546564a1b1ebefc5eaf75a8789d/merged 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/c0032a14774f4bfcad4ca3ab479a4d884a066d54bdaee256d777cac2ae2f8928/merged 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/0 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/9959488649cd55f16612342629502bf9984fc818f8cc29df5d2c4153d0d3cb1d/merged 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/9cdae0be81d8d327cf336fc7b04eb5d5b006d07a43e22f5d041836e0b1d14d3f/merged 2024-09-19T06:49:41.364 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/00845e3b716b17f5d86d1ed6438880a955b2a14cd49b70daaa112918edaa6031/merged 2024-09-19T06:49:41.365 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/3d764e6a6f6420b43266db576405fb5c9e14ca66a679ebecc080dd661f98e3ff/merged 2024-09-19T06:49:41.365 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/15b0cb7fbbf5bd6eb1d968f9a0aad06b77f595c1896792774e0417d387924854/merged 2024-09-19T06:49:41.365 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/32aa5c1d5a84b52e4bc181d5a1d54637ae6fc80615d31407f379875e00790321/merged 2024-09-19T06:49:41.365 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/73ac3881e79ff102d037d19015c2813052a66194f0d61ef7e1fcbdd4f471585d/merged 2024-09-19T06:49:41.365 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f74665c1e16eb38bac9ff7bd8891485dc5c67281af4002cc8f40faed7968ce74/merged 2024-09-19T06:49:41.365 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns 2024-09-19T06:49:41.365 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-19T06:49:41.365 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-19T06:49:41.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:41.366 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2024-09-19T06:49:41.442 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-19T06:49:41.442 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-19T06:49:41.487 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2024-09-19T06:49:41.545 DEBUG:teuthology.orchestra.run.smithi114:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T06:49:41.581 INFO:teuthology.orchestra.run.smithi114.stdout:/proc 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/sys 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/dev 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/security 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/shm 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/pts 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/run 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/cgroup 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/pstore 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/bpf 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/ 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/selinux 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/hugepages 2024-09-19T06:49:41.582 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/mqueue 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/debug 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/tracing 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/fuse/connections 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/config 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-sysctl.service 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/nfs/rpc_pipefs 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/1000 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/ceph 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/560568456d473cc1ff48de7c9dee0358a8e40546564a1b1ebefc5eaf75a8789d/merged 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/c0032a14774f4bfcad4ca3ab479a4d884a066d54bdaee256d777cac2ae2f8928/merged 2024-09-19T06:49:41.583 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/0 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/9959488649cd55f16612342629502bf9984fc818f8cc29df5d2c4153d0d3cb1d/merged 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/9cdae0be81d8d327cf336fc7b04eb5d5b006d07a43e22f5d041836e0b1d14d3f/merged 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/00845e3b716b17f5d86d1ed6438880a955b2a14cd49b70daaa112918edaa6031/merged 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/3d764e6a6f6420b43266db576405fb5c9e14ca66a679ebecc080dd661f98e3ff/merged 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/15b0cb7fbbf5bd6eb1d968f9a0aad06b77f595c1896792774e0417d387924854/merged 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/32aa5c1d5a84b52e4bc181d5a1d54637ae6fc80615d31407f379875e00790321/merged 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/73ac3881e79ff102d037d19015c2813052a66194f0d61ef7e1fcbdd4f471585d/merged 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f74665c1e16eb38bac9ff7bd8891485dc5c67281af4002cc8f40faed7968ce74/merged 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-19T06:49:41.584 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-19T06:49:41.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:41.585 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2024-09-19T06:49:41.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi114.stderr:2024-09-19T06:49:41.659+0000 7fc9a0d1e480 -1 init, newargv = 0x55d7c4581400 newargc=15 2024-09-19T06:49:41.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi114.stderr:ceph-fuse[74831]: starting ceph client 2024-09-19T06:49:41.670 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi114.stderr:ceph-fuse[74831]: starting fuse 2024-09-19T06:49:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:42 smithi114 ceph-mon[30820]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:42 smithi116 ceph-mon[35411]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:42 smithi118 ceph-mon[35527]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:42.763 DEBUG:teuthology.orchestra.run.smithi114:> sudo modprobe fuse 2024-09-19T06:49:42.827 DEBUG:teuthology.orchestra.run.smithi114:> cat /proc/self/mounts | awk '{print $2}' 2024-09-19T06:49:42.865 INFO:teuthology.orchestra.run.smithi114.stdout:/proc 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/sys 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/dev 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/security 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/shm 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/pts 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/run 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/cgroup 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/pstore 2024-09-19T06:49:42.866 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/bpf 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/ 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/selinux 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/hugepages 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/dev/mqueue 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/debug 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/tracing 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/fs/fuse/connections 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/sys/kernel/config 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-sysctl.service 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-19T06:49:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/nfs/rpc_pipefs 2024-09-19T06:49:42.868 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/1000 2024-09-19T06:49:42.868 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/ceph 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/560568456d473cc1ff48de7c9dee0358a8e40546564a1b1ebefc5eaf75a8789d/merged 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/c0032a14774f4bfcad4ca3ab479a4d884a066d54bdaee256d777cac2ae2f8928/merged 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/run/user/0 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/proc/sys/fs/binfmt_misc 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/9959488649cd55f16612342629502bf9984fc818f8cc29df5d2c4153d0d3cb1d/merged 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/9cdae0be81d8d327cf336fc7b04eb5d5b006d07a43e22f5d041836e0b1d14d3f/merged 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/00845e3b716b17f5d86d1ed6438880a955b2a14cd49b70daaa112918edaa6031/merged 2024-09-19T06:49:42.869 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/3d764e6a6f6420b43266db576405fb5c9e14ca66a679ebecc080dd661f98e3ff/merged 2024-09-19T06:49:42.870 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/15b0cb7fbbf5bd6eb1d968f9a0aad06b77f595c1896792774e0417d387924854/merged 2024-09-19T06:49:42.870 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/32aa5c1d5a84b52e4bc181d5a1d54637ae6fc80615d31407f379875e00790321/merged 2024-09-19T06:49:42.870 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/73ac3881e79ff102d037d19015c2813052a66194f0d61ef7e1fcbdd4f471585d/merged 2024-09-19T06:49:42.870 INFO:teuthology.orchestra.run.smithi114.stdout:/var/lib/containers/storage/overlay/f74665c1e16eb38bac9ff7bd8891485dc5c67281af4002cc8f40faed7968ce74/merged 2024-09-19T06:49:42.870 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns 2024-09-19T06:49:42.870 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-19T06:49:42.870 INFO:teuthology.orchestra.run.smithi114.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-19T06:49:42.870 INFO:teuthology.orchestra.run.smithi114.stdout:/home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:42.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:42.871 DEBUG:teuthology.orchestra.run.smithi114:> ls /sys/fs/fuse/connections 2024-09-19T06:49:42.949 INFO:teuthology.orchestra.run.smithi114.stdout:120 2024-09-19T06:49:42.950 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-19T06:49:42.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo stdin-killer -- python3 -c ' 2024-09-19T06:49:42.950 DEBUG:teuthology.orchestra.run.smithi114:> import glob 2024-09-19T06:49:42.951 DEBUG:teuthology.orchestra.run.smithi114:> import re 2024-09-19T06:49:42.951 DEBUG:teuthology.orchestra.run.smithi114:> import os 2024-09-19T06:49:42.951 DEBUG:teuthology.orchestra.run.smithi114:> import subprocess 2024-09-19T06:49:42.951 DEBUG:teuthology.orchestra.run.smithi114:> 2024-09-19T06:49:42.951 DEBUG:teuthology.orchestra.run.smithi114:> def _find_admin_socket(client_name): 2024-09-19T06:49:42.951 DEBUG:teuthology.orchestra.run.smithi114:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-19T06:49:42.952 DEBUG:teuthology.orchestra.run.smithi114:> files = glob.glob(asok_path) 2024-09-19T06:49:42.952 DEBUG:teuthology.orchestra.run.smithi114:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-19T06:49:42.952 DEBUG:teuthology.orchestra.run.smithi114:> 2024-09-19T06:49:42.952 DEBUG:teuthology.orchestra.run.smithi114:> # Given a non-glob path, it better be there 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> if "*" not in asok_path: 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> assert(len(files) == 1) 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> return files[0] 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> for f in files: 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-19T06:49:42.953 DEBUG:teuthology.orchestra.run.smithi114:> contents = proc_f.read() 2024-09-19T06:49:42.954 DEBUG:teuthology.orchestra.run.smithi114:> if mountpoint in contents: 2024-09-19T06:49:42.954 DEBUG:teuthology.orchestra.run.smithi114:> return f 2024-09-19T06:49:42.954 DEBUG:teuthology.orchestra.run.smithi114:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-19T06:49:42.954 DEBUG:teuthology.orchestra.run.smithi114:> 2024-09-19T06:49:42.954 DEBUG:teuthology.orchestra.run.smithi114:> print(_find_admin_socket("client.admin")) 2024-09-19T06:49:42.954 DEBUG:teuthology.orchestra.run.smithi114:> ' 2024-09-19T06:49:43.112 INFO:teuthology.orchestra.run.smithi114.stdout:/var/run/ceph/ceph-client.admin.74831.asok 2024-09-19T06:49:43.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:49:43.126 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.74831.asok 2024-09-19T06:49:43.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:43.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.74831.asok status 2024-09-19T06:49:43.308 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:49:43.308 INFO:teuthology.orchestra.run.smithi114.stdout: "metadata": { 2024-09-19T06:49:43.308 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_sha1": "2f599a46ad10e37c929e5c43da2f6907ddcd1961", 2024-09-19T06:49:43.308 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.1.1-361-g2f599a46 (2f599a46ad10e37c929e5c43da2f6907ddcd1961) squid (stable)", 2024-09-19T06:49:43.309 INFO:teuthology.orchestra.run.smithi114.stdout: "entity_id": "admin", 2024-09-19T06:49:43.309 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": "smithi114", 2024-09-19T06:49:43.309 INFO:teuthology.orchestra.run.smithi114.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-19T06:49:43.309 INFO:teuthology.orchestra.run.smithi114.stdout: "pid": "74831", 2024-09-19T06:49:43.309 INFO:teuthology.orchestra.run.smithi114.stdout: "root": "/" 2024-09-19T06:49:43.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-19T06:49:43.310 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_count": 0, 2024-09-19T06:49:43.310 INFO:teuthology.orchestra.run.smithi114.stdout: "dentry_pinned_count": 0, 2024-09-19T06:49:43.310 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 14640, 2024-09-19T06:49:43.310 INFO:teuthology.orchestra.run.smithi114.stdout: "inst": { 2024-09-19T06:49:43.310 INFO:teuthology.orchestra.run.smithi114.stdout: "name": { 2024-09-19T06:49:43.310 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2024-09-19T06:49:43.311 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 14640 2024-09-19T06:49:43.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-19T06:49:43.311 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2024-09-19T06:49:43.311 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-19T06:49:43.311 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2024-09-19T06:49:43.312 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2844822525 2024-09-19T06:49:43.312 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-09-19T06:49:43.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-19T06:49:43.312 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": { 2024-09-19T06:49:43.312 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-09-19T06:49:43.312 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "192.168.0.1:0", 2024-09-19T06:49:43.313 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2844822525 2024-09-19T06:49:43.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-09-19T06:49:43.313 INFO:teuthology.orchestra.run.smithi114.stdout: "inst_str": "client.14640 192.168.0.1:0/2844822525", 2024-09-19T06:49:43.313 INFO:teuthology.orchestra.run.smithi114.stdout: "addr_str": "192.168.0.1:0/2844822525", 2024-09-19T06:49:43.313 INFO:teuthology.orchestra.run.smithi114.stdout: "inode_count": 1, 2024-09-19T06:49:43.313 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_epoch": 17, 2024-09-19T06:49:43.314 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch": 91, 2024-09-19T06:49:43.314 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_epoch_barrier": 0, 2024-09-19T06:49:43.314 INFO:teuthology.orchestra.run.smithi114.stdout: "blocklisted": false, 2024-09-19T06:49:43.315 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs" 2024-09-19T06:49:43.315 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:49:43.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:43.320 DEBUG:teuthology.orchestra.run.smithi114:> stat --file-system '--printf=%T 2024-09-19T06:49:43.321 DEBUG:teuthology.orchestra.run.smithi114:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:43.357 INFO:teuthology.orchestra.run.smithi114.stdout:fuseblk 2024-09-19T06:49:43.357 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:43.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:43.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:43.454 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-19T06:49:43.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-19T06:49:43.566 INFO:teuthology.orchestra.run.smithi114.stdout:# file: . 2024-09-19T06:49:43.566 INFO:teuthology.orchestra.run.smithi114.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-19T06:49:43.566 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:43.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:49:43.574 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:49:43.574 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T06:49:43.610 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-19T06:49:44.018 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:44 smithi114 ceph-mon[30820]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:44 smithi116 ceph-mon[35411]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:44.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:44 smithi118 ceph-mon[35527]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:44.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 -- 172.21.15.114:0/4191375807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3264102360 msgr2=0x7f326410e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:44.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 --2- 172.21.15.114:0/4191375807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3264102360 0x7f326410e890 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f325c009cd0 tx=0x7f325c02a8f0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 -- 172.21.15.114:0/4191375807 shutdown_connections 2024-09-19T06:49:44.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 --2- 172.21.15.114:0/4191375807 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3264102360 0x7f326410e890 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 --2- 172.21.15.114:0/4191375807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32641019c0 0x7f3264101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 --2- 172.21.15.114:0/4191375807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32641079c0 0x7f3264107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 -- 172.21.15.114:0/4191375807 >> 172.21.15.114:0/4191375807 conn(0x7f32640fd740 msgr2=0x7f32640ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:44.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 -- 172.21.15.114:0/4191375807 shutdown_connections 2024-09-19T06:49:44.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.595+0000 7f3269c00640 1 -- 172.21.15.114:0/4191375807 wait complete. 2024-09-19T06:49:44.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.596+0000 7f3269c00640 1 Processor -- start 2024-09-19T06:49:44.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.596+0000 7f3269c00640 1 -- start start 2024-09-19T06:49:44.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3269c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32641019c0 0x7f326419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3269c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3264102360 0x7f326419a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3269c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32641079c0 0x7f326419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3269c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3264111e10 con 0x7f3264102360 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3269c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3264111c90 con 0x7f32641019c0 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3269c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f3264111f90 con 0x7f32641079c0 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3263e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32641019c0 0x7f326419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3263e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32641019c0 0x7f326419a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:41536/0 (socket says 172.21.15.114:41536) 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3263e00640 1 -- 172.21.15.114:0/94188262 learned_addr learned my addr 172.21.15.114:0/94188262 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3263400640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3264102360 0x7f326419a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.597+0000 7f3269200640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32641079c0 0x7f326419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3263e00640 1 -- 172.21.15.114:0/94188262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32641079c0 msgr2=0x7f326419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:44.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3263e00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32641079c0 0x7f326419ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3263e00640 1 -- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3264102360 msgr2=0x7f326419a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3263e00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3264102360 0x7f326419a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3263e00640 1 -- 172.21.15.114:0/94188262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326419f3d0 con 0x7f32641019c0 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3263e00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32641019c0 0x7f326419a420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f325400e9f0 tx=0x7f325400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3260c00640 1 -- 172.21.15.114:0/94188262 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325400cd80 con 0x7f32641019c0 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326419f6c0 con 0x7f32641019c0 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3260c00640 1 -- 172.21.15.114:0/94188262 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3254010910 con 0x7f32641019c0 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3260c00640 1 -- 172.21.15.114:0/94188262 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32540188f0 con 0x7f32641019c0 2024-09-19T06:49:44.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.598+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32641a6fa0 con 0x7f32641019c0 2024-09-19T06:49:44.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.599+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3264101e20 con 0x7f32641019c0 2024-09-19T06:49:44.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.601+0000 7f3260c00640 1 -- 172.21.15.114:0/94188262 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f3254010ac0 con 0x7f32641019c0 2024-09-19T06:49:44.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.602+0000 7f3260c00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f32480a3240 0x7f32480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:44.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.603+0000 7f3260c00640 1 -- 172.21.15.114:0/94188262 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f32540cfec0 con 0x7f32641019c0 2024-09-19T06:49:44.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.603+0000 7f3263400640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f32480a3240 0x7f32480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:44.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.603+0000 7f3263400640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f32480a3240 0x7f32480a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f326410ecb0 tx=0x7f3250005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:44.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.607+0000 7f3260c00640 1 -- 172.21.15.114:0/94188262 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f325409c2f0 con 0x7f32641019c0 2024-09-19T06:49:44.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.794+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f32641a7290 con 0x7f32641019c0 2024-09-19T06:49:44.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.795+0000 7f3260c00640 1 -- 172.21.15.114:0/94188262 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+601 (secure 0 0 0) 0x7f32540a05f0 con 0x7f32641019c0 2024-09-19T06:49:44.796 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/3886387391 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:44.796 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3653733983 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1660437080 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2975903254 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3244383942 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/3886387391 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/967314322 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/2384341327 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/2384341327 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1724858398 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:44.797 INFO:teuthology.orchestra.run.smithi114.stderr:listed 10 entries 2024-09-19T06:49:44.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f32480a3240 msgr2=0x7f32480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f32480a3240 0x7f32480a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f326410ecb0 tx=0x7f3250005f90 comp rx=0 tx=0).stop 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32641019c0 msgr2=0x7f326419a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32641019c0 0x7f326419a420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f325400e9f0 tx=0x7f325400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 shutdown_connections 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f32480a3240 0x7f32480a5700 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f32641079c0 0x7f326419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3264102360 0x7f326419a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 --2- 172.21.15.114:0/94188262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32641019c0 0x7f326419a420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.797+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 >> 172.21.15.114:0/94188262 conn(0x7f32640fd740 msgr2=0x7f32640fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.798+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 shutdown_connections 2024-09-19T06:49:44.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:44.798+0000 7f3269c00640 1 -- 172.21.15.114:0/94188262 wait complete. 2024-09-19T06:49:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:45 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/94188262' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T06:49:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:45 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/94188262' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T06:49:45.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:45 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/94188262' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T06:49:45.720 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:49:45.721 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T06:49:45.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-19T06:49:46.162 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:46 smithi114 ceph-mon[30820]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:46 smithi116 ceph-mon[35411]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:46.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:46 smithi118 ceph-mon[35527]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-19T06:49:46.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.722+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1088213571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 msgr2=0x7f9d18108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:46.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.722+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1088213571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 0x7f9d18108830 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009900 tx=0x7f9d0002a410 comp rx=0 tx=0).stop 2024-09-19T06:49:46.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.722+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1088213571 shutdown_connections 2024-09-19T06:49:46.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.722+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1088213571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d18108f60 0x7f9d18110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.722+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1088213571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 0x7f9d18108830 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.722+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1088213571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d18103e30 0x7f9d18104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.722+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1088213571 >> 172.21.15.114:0/1088213571 conn(0x7f9d180ff910 msgr2=0x7f9d18101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.723+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1088213571 shutdown_connections 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.723+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1088213571 wait complete. 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.723+0000 7f9d1dc00640 1 Processor -- start 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d1dc00640 1 -- start start 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d1dc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d18103e30 0x7f9d1819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d1dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 0x7f9d1819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d1dc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d18108f60 0x7f9d1819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d1dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d18114010 con 0x7f9d181047e0 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d1dc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d18113e90 con 0x7f9d18108f60 2024-09-19T06:49:46.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d1dc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9d18114190 con 0x7f9d18103e30 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d0fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 0x7f9d1819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d0fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 0x7f9d1819a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55764/0 (socket says 172.21.15.114:55764) 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d0fe00640 1 -- 172.21.15.114:0/1285017359 learned_addr learned my addr 172.21.15.114:0/1285017359 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d17e00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d18103e30 0x7f9d1819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.724+0000 7f9d1d200640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d18108f60 0x7f9d1819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d0fe00640 1 -- 172.21.15.114:0/1285017359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d18103e30 msgr2=0x7f9d1819a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d0fe00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d18103e30 0x7f9d1819a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d0fe00640 1 -- 172.21.15.114:0/1285017359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d18108f60 msgr2=0x7f9d1819ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d0fe00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d18108f60 0x7f9d1819ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d0fe00640 1 -- 172.21.15.114:0/1285017359 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1819f420 con 0x7f9d181047e0 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d1d200640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d18108f60 0x7f9d1819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d17e00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d18103e30 0x7f9d1819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d0fe00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 0x7f9d1819a9b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d00004770 tx=0x7f9d00003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:46.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d15600640 1 -- 172.21.15.114:0/1285017359 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0003f070 con 0x7f9d181047e0 2024-09-19T06:49:46.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1819f6b0 con 0x7f9d181047e0 2024-09-19T06:49:46.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.725+0000 7f9d15600640 1 -- 172.21.15.114:0/1285017359 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d00031ca0 con 0x7f9d181047e0 2024-09-19T06:49:46.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.726+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d181a6ff0 con 0x7f9d181047e0 2024-09-19T06:49:46.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.726+0000 7f9d15600640 1 -- 172.21.15.114:0/1285017359 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00049060 con 0x7f9d181047e0 2024-09-19T06:49:46.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.727+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d18105310 con 0x7f9d181047e0 2024-09-19T06:49:46.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.729+0000 7f9d15600640 1 -- 172.21.15.114:0/1285017359 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f9d00042070 con 0x7f9d181047e0 2024-09-19T06:49:46.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.731+0000 7f9d15600640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cfc0a3240 0x7f9cfc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:46.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.731+0000 7f9d15600640 1 -- 172.21.15.114:0/1285017359 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f9d000f0810 con 0x7f9d181047e0 2024-09-19T06:49:46.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.731+0000 7f9d17e00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cfc0a3240 0x7f9cfc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:46.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.732+0000 7f9d17e00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cfc0a3240 0x7f9cfc0a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009fc0 tx=0x7f9d08005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:46.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.733+0000 7f9d15600640 1 -- 172.21.15.114:0/1285017359 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9d000bd350 con 0x7f9d181047e0 2024-09-19T06:49:46.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.918+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9d1810c170 con 0x7f9d181047e0 2024-09-19T06:49:46.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.919+0000 7f9d15600640 1 -- 172.21.15.114:0/1285017359 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+601 (secure 0 0 0) 0x7f9d000bcc50 con 0x7f9d181047e0 2024-09-19T06:49:46.920 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/3886387391 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:46.920 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3653733983 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1660437080 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2975903254 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/3244383942 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/3886387391 2024-09-20T06:41:50.994561+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/967314322 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/2384341327 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/2384341327 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1724858398 2024-09-20T06:41:23.468444+0000 2024-09-19T06:49:46.921 INFO:teuthology.orchestra.run.smithi114.stderr:listed 10 entries 2024-09-19T06:49:46.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.921+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cfc0a3240 msgr2=0x7f9cfc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:46.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.921+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cfc0a3240 0x7f9cfc0a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9d08009fc0 tx=0x7f9d08005f90 comp rx=0 tx=0).stop 2024-09-19T06:49:46.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.921+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 msgr2=0x7f9d1819a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:46.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.921+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 0x7f9d1819a9b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d00004770 tx=0x7f9d00003c10 comp rx=0 tx=0).stop 2024-09-19T06:49:46.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.922+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 shutdown_connections 2024-09-19T06:49:46.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.922+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cfc0a3240 0x7f9cfc0a5700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.922+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d18108f60 0x7f9d1819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.922+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d181047e0 0x7f9d1819a9b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.922+0000 7f9d1dc00640 1 --2- 172.21.15.114:0/1285017359 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d18103e30 0x7f9d1819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:46.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.922+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 >> 172.21.15.114:0/1285017359 conn(0x7f9d180ff910 msgr2=0x7f9d18100c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:46.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.922+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 shutdown_connections 2024-09-19T06:49:46.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:46.922+0000 7f9d1dc00640 1 -- 172.21.15.114:0/1285017359 wait complete. 2024-09-19T06:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:47 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1285017359' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T06:49:47.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:47 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1285017359' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T06:49:47.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:47 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1285017359' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-19T06:49:47.914 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi114.front.sepia.ceph.com... 2024-09-19T06:49:47.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:47.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-19T06:49:47.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi114.stderr:ceph-fuse[74831]: fuse finished with error 0 and tester_r 0 2024-09-19T06:49:47.973 INFO:teuthology.orchestra.run:waiting for 300 2024-09-19T06:49:48.038 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-09-19T06:49:48.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:49:48.038 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-19T06:49:48.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:49:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:48 smithi114 ceph-mon[30820]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-19T06:49:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:48 smithi116 ceph-mon[35411]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-19T06:49:48.454 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:48 smithi118 ceph-mon[35527]: pgmap v319: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 -- 172.21.15.114:0/2826519664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584076720 msgr2=0x7f5584076b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 --2- 172.21.15.114:0/2826519664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584076720 0x7f5584076b00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5578009900 tx=0x7f557802a3f0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 -- 172.21.15.114:0/2826519664 shutdown_connections 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 --2- 172.21.15.114:0/2826519664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584112890 0x7f5584114c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 --2- 172.21.15.114:0/2826519664 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5584077040 0x7f5584112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 --2- 172.21.15.114:0/2826519664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584076720 0x7f5584076b00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 -- 172.21.15.114:0/2826519664 >> 172.21.15.114:0/2826519664 conn(0x7f55840ff8b0 msgr2=0x7f5584101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 -- 172.21.15.114:0/2826519664 shutdown_connections 2024-09-19T06:49:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.021+0000 7f558b200640 1 -- 172.21.15.114:0/2826519664 wait complete. 2024-09-19T06:49:49.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.022+0000 7f558b200640 1 Processor -- start 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.022+0000 7f558b200640 1 -- start start 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f558b200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584076720 0x7f55841a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f558b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584077040 0x7f55841a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f558b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5584112890 0x7f558419e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f558b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5584118590 con 0x7f5584077040 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f558b200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5584118410 con 0x7f5584076720 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f558b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5584118710 con 0x7f5584112890 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584077040 0x7f55841a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584076720 0x7f55841a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:49.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584077040 0x7f55841a4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55026/0 (socket says 172.21.15.114:55026) 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589400640 1 -- 172.21.15.114:0/778408036 learned_addr learned my addr 172.21.15.114:0/778408036 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584076720 0x7f55841a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:36820/0 (socket says 172.21.15.114:36820) 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f558a800640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5584112890 0x7f558419e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589e00640 1 -- 172.21.15.114:0/778408036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5584112890 msgr2=0x7f558419e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589e00640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5584112890 0x7f558419e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589e00640 1 -- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584077040 msgr2=0x7f55841a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589e00640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584077040 0x7f55841a4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.023+0000 7f5589e00640 1 -- 172.21.15.114:0/778408036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558419ef90 con 0x7f5584076720 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.024+0000 7f5589400640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584077040 0x7f55841a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.024+0000 7f5589e00640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584076720 0x7f55841a46a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f557802a910 tx=0x7f5578003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.024+0000 7f557ea00640 1 -- 172.21.15.114:0/778408036 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557803f070 con 0x7f5584076720 2024-09-19T06:49:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.024+0000 7f557ea00640 1 -- 172.21.15.114:0/778408036 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5578031c90 con 0x7f5584076720 2024-09-19T06:49:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.024+0000 7f557ea00640 1 -- 172.21.15.114:0/778408036 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5578035030 con 0x7f5584076720 2024-09-19T06:49:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.024+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558419f220 con 0x7f5584076720 2024-09-19T06:49:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.024+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55841ab570 con 0x7f5584076720 2024-09-19T06:49:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.025+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5584077e60 con 0x7f5584076720 2024-09-19T06:49:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.027+0000 7f557ea00640 1 -- 172.21.15.114:0/778408036 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f5578042070 con 0x7f5584076720 2024-09-19T06:49:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.029+0000 7f557ea00640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f556c0a3240 0x7f556c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.029+0000 7f557ea00640 1 -- 172.21.15.114:0/778408036 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f55780f0a90 con 0x7f5584076720 2024-09-19T06:49:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.029+0000 7f5589400640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f556c0a3240 0x7f556c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:49.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.030+0000 7f5589400640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f556c0a3240 0x7f556c0a5700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5584079330 tx=0x7f5574005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:49.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.031+0000 7f557ea00640 1 -- 172.21.15.114:0/778408036 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f55780bcec0 con 0x7f5584076720 2024-09-19T06:49:49.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.210+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5584076b00 con 0x7f5584076720 2024-09-19T06:49:49.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.212+0000 7f557ea00640 1 -- 172.21.15.114:0/778408036 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24271 (secure 0 0 0) 0x7f55780c11c0 con 0x7f5584076720 2024-09-19T06:49:49.214 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:49.214 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":91,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:49:23.496526+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:49:49.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f556c0a3240 msgr2=0x7f556c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:49.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f556c0a3240 0x7f556c0a5700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5584079330 tx=0x7f5574005f90 comp rx=0 tx=0).stop 2024-09-19T06:49:49.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584076720 msgr2=0x7f55841a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:49.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584076720 0x7f55841a46a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f557802a910 tx=0x7f5578003b20 comp rx=0 tx=0).stop 2024-09-19T06:49:49.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 shutdown_connections 2024-09-19T06:49:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f556c0a3240 0x7f556c0a5700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5584112890 0x7f558419e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5584077040 0x7f55841a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 --2- 172.21.15.114:0/778408036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584076720 0x7f55841a46a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 >> 172.21.15.114:0/778408036 conn(0x7f55840ff8b0 msgr2=0x7f5584101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 shutdown_connections 2024-09-19T06:49:49.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:49.215+0000 7f558b200640 1 -- 172.21.15.114:0/778408036 wait complete. 2024-09-19T06:49:50.019 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:49:50.391 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:50 smithi114 ceph-mon[30820]: pgmap v320: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-19T06:49:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:50 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/778408036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:50 smithi116 ceph-mon[35411]: pgmap v320: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-19T06:49:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:50 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/778408036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:50 smithi118 ceph-mon[35527]: pgmap v320: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-09-19T06:49:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:50 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/778408036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:50.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.951+0000 7f9146a00640 1 -- 172.21.15.114:0/1000916178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 msgr2=0x7f91401021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:50.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.951+0000 7f9146a00640 1 --2- 172.21.15.114:0/1000916178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 0x7f91401021c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9130009860 tx=0x7f913002a340 comp rx=0 tx=0).stop 2024-09-19T06:49:50.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.952+0000 7f9146a00640 1 -- 172.21.15.114:0/1000916178 shutdown_connections 2024-09-19T06:49:50.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.952+0000 7f9146a00640 1 --2- 172.21.15.114:0/1000916178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9140106f10 0x7f914010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:50.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.952+0000 7f9146a00640 1 --2- 172.21.15.114:0/1000916178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9140102790 0x7f91401067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:50.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.952+0000 7f9146a00640 1 --2- 172.21.15.114:0/1000916178 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 0x7f91401021c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:50.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.952+0000 7f9146a00640 1 -- 172.21.15.114:0/1000916178 >> 172.21.15.114:0/1000916178 conn(0x7f91400fd8c0 msgr2=0x7f91400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:50.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.952+0000 7f9146a00640 1 -- 172.21.15.114:0/1000916178 shutdown_connections 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.952+0000 7f9146a00640 1 -- 172.21.15.114:0/1000916178 wait complete. 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.952+0000 7f9146a00640 1 Processor -- start 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9146a00640 1 -- start start 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9146a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 0x7f914019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9146a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9140102790 0x7f914019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9146a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9140106f10 0x7f914019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9146a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9140111f70 con 0x7f9140106f10 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9146a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9140111df0 con 0x7f9140102790 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9146a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f91401120f0 con 0x7f9140101de0 2024-09-19T06:49:50.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9144a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9140106f10 0x7f914019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9137400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9140102790 0x7f914019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f913fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 0x7f914019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9144a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9140106f10 0x7f914019ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55040/0 (socket says 172.21.15.114:55040) 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.953+0000 7f9144a00640 1 -- 172.21.15.114:0/3509588077 learned_addr learned my addr 172.21.15.114:0/3509588077 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9137400640 1 -- 172.21.15.114:0/3509588077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 msgr2=0x7f914019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9137400640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 0x7f914019a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9137400640 1 -- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9140106f10 msgr2=0x7f914019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9137400640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9140106f10 0x7f914019ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9137400640 1 -- 172.21.15.114:0/3509588077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914019f5f0 con 0x7f9140102790 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f913fe00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 0x7f914019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9144a00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9140106f10 0x7f914019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9137400640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9140102790 0x7f914019ab80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9128005fd0 tx=0x7f912800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:50.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f913d600640 1 -- 172.21.15.114:0/3509588077 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912800cd00 con 0x7f9140102790 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914019f8e0 con 0x7f9140102790 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f913d600640 1 -- 172.21.15.114:0/3509588077 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91280035f0 con 0x7f9140102790 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f913d600640 1 -- 172.21.15.114:0/3509588077 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128010670 con 0x7f9140102790 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.954+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91401a71c0 con 0x7f9140102790 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.955+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91401021c0 con 0x7f9140102790 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.957+0000 7f913d600640 1 -- 172.21.15.114:0/3509588077 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f91280037a0 con 0x7f9140102790 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.959+0000 7f913d600640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91240a3240 0x7f91240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.959+0000 7f913d600640 1 -- 172.21.15.114:0/3509588077 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f91280cfe20 con 0x7f9140102790 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.959+0000 7f913fe00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91240a3240 0x7f91240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.959+0000 7f913fe00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91240a3240 0x7f91240a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f913002a930 tx=0x7f91300058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:50.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:50.961+0000 7f913d600640 1 -- 172.21.15.114:0/3509588077 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9128009510 con 0x7f9140102790 2024-09-19T06:49:51.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.145+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f914019b9a0 con 0x7f9140102790 2024-09-19T06:49:51.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.146+0000 7f913d600640 1 -- 172.21.15.114:0/3509588077 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24271 (secure 0 0 0) 0x7f914019b9a0 con 0x7f9140102790 2024-09-19T06:49:51.148 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:49:51.148 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":91,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:49:23.496526+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:49:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91240a3240 msgr2=0x7f91240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91240a3240 0x7f91240a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f913002a930 tx=0x7f91300058d0 comp rx=0 tx=0).stop 2024-09-19T06:49:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9140102790 msgr2=0x7f914019ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:51.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9140102790 0x7f914019ab80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9128005fd0 tx=0x7f912800ee50 comp rx=0 tx=0).stop 2024-09-19T06:49:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 shutdown_connections 2024-09-19T06:49:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91240a3240 0x7f91240a5700 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9140106f10 0x7f914019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9140102790 0x7f914019ab80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 --2- 172.21.15.114:0/3509588077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9140101de0 0x7f914019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 >> 172.21.15.114:0/3509588077 conn(0x7f91400fd8c0 msgr2=0x7f91400feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 shutdown_connections 2024-09-19T06:49:51.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:51.149+0000 7f9146a00640 1 -- 172.21.15.114:0/3509588077 wait complete. 2024-09-19T06:49:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-09-19T06:49:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-09-19T06:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:51 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3509588077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:51.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-09-19T06:49:51.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-09-19T06:49:51.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-09-19T06:49:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-09-19T06:49:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-09-19T06:49:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-09-19T06:49:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-09-19T06:49:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-19T06:49:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-09-19T06:49:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-09-19T06:49:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:51 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3509588077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-09-19T06:49:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:51 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3509588077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:49:51.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-09-19T06:49:52.284 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:52.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-19T06:49:52.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-19T06:49:52.460 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-09-19T06:49:52.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:52 smithi114 ceph-mon[30820]: osdmap e92: 12 total, 12 up, 12 in 2024-09-19T06:49:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-19T06:49:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-19T06:49:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-09-19T06:49:52.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:52 smithi118 ceph-mon[35527]: osdmap e92: 12 total, 12 up, 12 in 2024-09-19T06:49:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: pgmap v321: 129 pgs: 129 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-09-19T06:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-09-19T06:49:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:52 smithi116 ceph-mon[35411]: osdmap e92: 12 total, 12 up, 12 in 2024-09-19T06:49:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 -- 172.21.15.114:0/573757686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0841047e0 msgr2=0x7fa084108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 --2- 172.21.15.114:0/573757686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0841047e0 0x7fa084108830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009900 tx=0x7fa06c02a410 comp rx=0 tx=0).stop 2024-09-19T06:49:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 -- 172.21.15.114:0/573757686 shutdown_connections 2024-09-19T06:49:52.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 --2- 172.21.15.114:0/573757686 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084108f60 0x7fa084110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 --2- 172.21.15.114:0/573757686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0841047e0 0x7fa084108830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 --2- 172.21.15.114:0/573757686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa084103e30 0x7fa084104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 -- 172.21.15.114:0/573757686 >> 172.21.15.114:0/573757686 conn(0x7fa0840ff8d0 msgr2=0x7fa084101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 -- 172.21.15.114:0/573757686 shutdown_connections 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.814+0000 7fa088a00640 1 -- 172.21.15.114:0/573757686 wait complete. 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa088a00640 1 Processor -- start 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa088a00640 1 -- start start 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa088a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa084103e30 0x7fa08419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa088a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841047e0 0x7fa08419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa088a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084108f60 0x7fa08419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa088a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa084113fd0 con 0x7fa084103e30 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa088a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa084113e50 con 0x7fa0841047e0 2024-09-19T06:49:52.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa088a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa084114150 con 0x7fa084108f60 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa082800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa084103e30 0x7fa08419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.815+0000 7fa083200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084108f60 0x7fa08419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa083200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084108f60 0x7fa08419ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:50108/0 (socket says 172.21.15.114:50108) 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa083200640 1 -- 172.21.15.114:0/1413634310 learned_addr learned my addr 172.21.15.114:0/1413634310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa079e00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841047e0 0x7fa08419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa083200640 1 -- 172.21.15.114:0/1413634310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841047e0 msgr2=0x7fa08419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa083200640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841047e0 0x7fa08419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa083200640 1 -- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa084103e30 msgr2=0x7fa08419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa083200640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa084103e30 0x7fa08419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa083200640 1 -- 172.21.15.114:0/1413634310 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08419f440 con 0x7fa084108f60 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa079e00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841047e0 0x7fa08419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa083200640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084108f60 0x7fa08419ed60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00e970 tx=0x7fa07c00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:52.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.816+0000 7fa07be00640 1 -- 172.21.15.114:0/1413634310 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c00ccc0 con 0x7fa084108f60 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.817+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08419f730 con 0x7fa084108f60 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.817+0000 7fa07be00640 1 -- 172.21.15.114:0/1413634310 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa07c00ce60 con 0x7fa084108f60 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.817+0000 7fa07be00640 1 -- 172.21.15.114:0/1413634310 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c010430 con 0x7fa084108f60 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.817+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0841a7010 con 0x7fa084108f60 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.818+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa084104210 con 0x7fa084108f60 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.820+0000 7fa07be00640 1 -- 172.21.15.114:0/1413634310 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fa07c0046c0 con 0x7fa084108f60 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.821+0000 7fa07be00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa0680a3240 0x7fa0680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.821+0000 7fa07be00640 1 -- 172.21.15.114:0/1413634310 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa07c0d04b0 con 0x7fa084108f60 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.821+0000 7fa082800640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa0680a3240 0x7fa0680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.822+0000 7fa082800640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa0680a3240 0x7fa0680a5700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa074005fd0 tx=0x7fa074009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:52.823+0000 7fa07be00640 1 -- 172.21.15.114:0/1413634310 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa07c09ce10 con 0x7fa084108f60 2024-09-19T06:49:53.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.034+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa08410c1a0 con 0x7fa084108f60 2024-09-19T06:49:53.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.198+0000 7fa07be00640 1 -- 172.21.15.114:0/1413634310 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fa07c09c710 con 0x7fa084108f60 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa0680a3240 msgr2=0x7fa0680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa0680a3240 0x7fa0680a5700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa074005fd0 tx=0x7fa074009680 comp rx=0 tx=0).stop 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084108f60 msgr2=0x7fa08419ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084108f60 0x7fa08419ed60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00e970 tx=0x7fa07c00ed70 comp rx=0 tx=0).stop 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 shutdown_connections 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa0680a3240 0x7fa0680a5700 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa084108f60 0x7fa08419ed60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841047e0 0x7fa08419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:53.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 --2- 172.21.15.114:0/1413634310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa084103e30 0x7fa08419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:53.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.200+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 >> 172.21.15.114:0/1413634310 conn(0x7fa0840ff8d0 msgr2=0x7fa084100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:53.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.201+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 shutdown_connections 2024-09-19T06:49:53.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:53.201+0000 7fa088a00640 1 -- 172.21.15.114:0/1413634310 wait complete. 2024-09-19T06:49:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:53 smithi114 ceph-mon[30820]: osdmap e93: 12 total, 12 up, 12 in 2024-09-19T06:49:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:53 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1413634310' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T06:49:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:53 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T06:49:53.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:53 smithi118 ceph-mon[35527]: osdmap e93: 12 total, 12 up, 12 in 2024-09-19T06:49:53.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:53 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1413634310' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T06:49:53.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:53 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T06:49:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:53 smithi116 ceph-mon[35411]: osdmap e93: 12 total, 12 up, 12 in 2024-09-19T06:49:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:53 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1413634310' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T06:49:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:53 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-19T06:49:54.037 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-19T06:49:54.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-09-19T06:49:54.406 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:54 smithi114 ceph-mon[30820]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2024-09-19T06:49:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:54 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-19T06:49:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:54 smithi114 ceph-mon[30820]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:54.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:54 smithi118 ceph-mon[35527]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2024-09-19T06:49:54.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:54 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-19T06:49:54.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:54 smithi118 ceph-mon[35527]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:54 smithi116 ceph-mon[35411]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 0 op/s 2024-09-19T06:49:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:54 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-19T06:49:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:54 smithi116 ceph-mon[35411]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:54.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.956+0000 7f950fa00640 1 -- 172.21.15.114:0/301407142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508106f10 msgr2=0x7f950810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:54.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.956+0000 7f950fa00640 1 --2- 172.21.15.114:0/301407142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508106f10 0x7f950810ea90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9500009ca0 tx=0x7f950002a8b0 comp rx=0 tx=0).stop 2024-09-19T06:49:54.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.956+0000 7f950fa00640 1 -- 172.21.15.114:0/301407142 shutdown_connections 2024-09-19T06:49:54.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.956+0000 7f950fa00640 1 --2- 172.21.15.114:0/301407142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508106f10 0x7f950810ea90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:54.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.956+0000 7f950fa00640 1 --2- 172.21.15.114:0/301407142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9508102790 0x7f95081067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:54.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.957+0000 7f950fa00640 1 --2- 172.21.15.114:0/301407142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9508101de0 0x7f95081021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:54.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.957+0000 7f950fa00640 1 -- 172.21.15.114:0/301407142 >> 172.21.15.114:0/301407142 conn(0x7f95080fd8c0 msgr2=0x7f95080ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:54.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.957+0000 7f950fa00640 1 -- 172.21.15.114:0/301407142 shutdown_connections 2024-09-19T06:49:54.959 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.957+0000 7f950fa00640 1 -- 172.21.15.114:0/301407142 wait complete. 2024-09-19T06:49:54.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.957+0000 7f950fa00640 1 Processor -- start 2024-09-19T06:49:54.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f950fa00640 1 -- start start 2024-09-19T06:49:54.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f950fa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9508101de0 0x7f950819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:54.961 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f950fa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508102790 0x7f950819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:54.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f950fa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9508106f10 0x7f950819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:54.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f950fa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9508111fc0 con 0x7f9508102790 2024-09-19T06:49:54.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f950fa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9508111e40 con 0x7f9508106f10 2024-09-19T06:49:54.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f950fa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9508112140 con 0x7f9508101de0 2024-09-19T06:49:54.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f9507e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508102790 0x7f950819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:54.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f9507e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508102790 0x7f950819ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55080/0 (socket says 172.21.15.114:55080) 2024-09-19T06:49:54.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f9507e00640 1 -- 172.21.15.114:0/2954890611 learned_addr learned my addr 172.21.15.114:0/2954890611 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.958+0000 7f950da00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9508106f10 0x7f950819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f950d000640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9508101de0 0x7f950819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f9507e00640 1 -- 172.21.15.114:0/2954890611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9508101de0 msgr2=0x7f950819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f9507e00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9508101de0 0x7f950819a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f9507e00640 1 -- 172.21.15.114:0/2954890611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9508106f10 msgr2=0x7f950819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f9507e00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9508106f10 0x7f950819ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f9507e00640 1 -- 172.21.15.114:0/2954890611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950819f5e0 con 0x7f9508102790 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f950da00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9508106f10 0x7f950819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:49:54.963 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f9507e00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508102790 0x7f950819ab70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f94f400d950 tx=0x7f94f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:54.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f9505600640 1 -- 172.21.15.114:0/2954890611 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4017070 con 0x7f9508102790 2024-09-19T06:49:54.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950819f8d0 con 0x7f9508102790 2024-09-19T06:49:54.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.959+0000 7f9505600640 1 -- 172.21.15.114:0/2954890611 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f4009c30 con 0x7f9508102790 2024-09-19T06:49:54.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.960+0000 7f9505600640 1 -- 172.21.15.114:0/2954890611 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f400f6b0 con 0x7f9508102790 2024-09-19T06:49:54.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.960+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081a71b0 con 0x7f9508102790 2024-09-19T06:49:54.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.961+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95081021c0 con 0x7f9508102790 2024-09-19T06:49:54.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.963+0000 7f9505600640 1 -- 172.21.15.114:0/2954890611 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f94f40109d0 con 0x7f9508102790 2024-09-19T06:49:54.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.964+0000 7f9505600640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f94f00a3240 0x7f94f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:54.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.965+0000 7f9505600640 1 -- 172.21.15.114:0/2954890611 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f94f40cf7b0 con 0x7f9508102790 2024-09-19T06:49:54.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.965+0000 7f950d000640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f94f00a3240 0x7f94f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:54.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.965+0000 7f950d000640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f94f00a3240 0x7f94f00a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009bd0 tx=0x7f94fc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:54.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:54.969+0000 7f9505600640 1 -- 172.21.15.114:0/2954890611 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f94f4009de0 con 0x7f9508102790 2024-09-19T06:49:55.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.178+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f95081a7410 con 0x7f9508102790 2024-09-19T06:49:55.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.203+0000 7f9505600640 1 -- 172.21.15.114:0/2954890611 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f94f409bac0 con 0x7f9508102790 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.205+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f94f00a3240 msgr2=0x7f94f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.205+0000 7f950fa00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f94f00a3240 0x7f94f00a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009bd0 tx=0x7f94fc005f90 comp rx=0 tx=0).stop 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.205+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508102790 msgr2=0x7f950819ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.205+0000 7f950fa00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508102790 0x7f950819ab70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f94f400d950 tx=0x7f94f400de10 comp rx=0 tx=0).stop 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.206+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 shutdown_connections 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.206+0000 7f950fa00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f94f00a3240 0x7f94f00a5700 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.206+0000 7f950fa00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9508106f10 0x7f950819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.206+0000 7f950fa00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9508102790 0x7f950819ab70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.206+0000 7f950fa00640 1 --2- 172.21.15.114:0/2954890611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9508101de0 0x7f950819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:55.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.206+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 >> 172.21.15.114:0/2954890611 conn(0x7f95080fd8c0 msgr2=0x7f95080ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:55.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.206+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 shutdown_connections 2024-09-19T06:49:55.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:55.206+0000 7f950fa00640 1 -- 172.21.15.114:0/2954890611 wait complete. 2024-09-19T06:49:55.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:55 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2954890611' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-19T06:49:55.601 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:55 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2954890611' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-19T06:49:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:55 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2954890611' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-19T06:49:56.007 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-09-19T06:49:56.382 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:56 smithi114 ceph-mon[30820]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:49:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:56 smithi114 ceph-mon[30820]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-19T06:49:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:56 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2954890611' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-19T06:49:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:56 smithi114 ceph-mon[30820]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-19T06:49:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:56 smithi114 ceph-mon[30820]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:56 smithi114 ceph-mon[30820]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-19T06:49:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:56 smithi114 ceph-mon[30820]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-19T06:49:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:56 smithi118 ceph-mon[35527]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:49:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:56 smithi118 ceph-mon[35527]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-19T06:49:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:56 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2954890611' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-19T06:49:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:56 smithi118 ceph-mon[35527]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-19T06:49:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:56 smithi118 ceph-mon[35527]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:56.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:56 smithi118 ceph-mon[35527]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-19T06:49:56.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:56 smithi118 ceph-mon[35527]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-19T06:49:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:56 smithi116 ceph-mon[35411]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:49:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:56 smithi116 ceph-mon[35411]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-19T06:49:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:56 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2954890611' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-19T06:49:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:56 smithi116 ceph-mon[35411]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-19T06:49:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:56 smithi116 ceph-mon[35411]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-09-19T06:49:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:56 smithi116 ceph-mon[35411]: fsmap cephfs:2 {0=i=up:active,1=b=up:creating} 10 up:standby 2024-09-19T06:49:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:56 smithi116 ceph-mon[35411]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-09-19T06:49:56.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.948+0000 7f409aa00640 1 -- 172.21.15.114:0/3467763275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40941049c0 msgr2=0x7f4094108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.948+0000 7f409aa00640 1 --2- 172.21.15.114:0/3467763275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40941049c0 0x7f4094108a10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4080009900 tx=0x7f408002a410 comp rx=0 tx=0).stop 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.948+0000 7f409aa00640 1 -- 172.21.15.114:0/3467763275 shutdown_connections 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.948+0000 7f409aa00640 1 --2- 172.21.15.114:0/3467763275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4094109140 0x7f4094110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.948+0000 7f409aa00640 1 --2- 172.21.15.114:0/3467763275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40941049c0 0x7f4094108a10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.948+0000 7f409aa00640 1 --2- 172.21.15.114:0/3467763275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4094104010 0x7f40941043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.948+0000 7f409aa00640 1 -- 172.21.15.114:0/3467763275 >> 172.21.15.114:0/3467763275 conn(0x7f40940ffa50 msgr2=0x7f4094101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.949+0000 7f409aa00640 1 -- 172.21.15.114:0/3467763275 shutdown_connections 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.949+0000 7f409aa00640 1 -- 172.21.15.114:0/3467763275 wait complete. 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f409aa00640 1 Processor -- start 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f409aa00640 1 -- start start 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f409aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4094104010 0x7f409419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:56.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f409aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40941049c0 0x7f409419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f409aa00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4094109140 0x7f409419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f409aa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40941141b0 con 0x7f4094104010 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f409aa00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4094114030 con 0x7f4094109140 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f409aa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4094114330 con 0x7f40941049c0 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f4093e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4094104010 0x7f409419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f4093e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4094104010 0x7f409419a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53304/0 (socket says 172.21.15.114:53304) 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.950+0000 7f4093e00640 1 -- 172.21.15.114:0/839279713 learned_addr learned my addr 172.21.15.114:0/839279713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4093400640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40941049c0 0x7f409419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4098a00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4094109140 0x7f409419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4093e00640 1 -- 172.21.15.114:0/839279713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40941049c0 msgr2=0x7f409419abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:56.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4093e00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40941049c0 0x7f409419abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4093e00640 1 -- 172.21.15.114:0/839279713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4094109140 msgr2=0x7f409419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4093e00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4094109140 0x7f409419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4093e00640 1 -- 172.21.15.114:0/839279713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409419f660 con 0x7f4094104010 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4093e00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4094104010 0x7f409419a6b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f408400e9f0 tx=0x7f408400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4090c00640 1 -- 172.21.15.114:0/839279713 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408400ccc0 con 0x7f4094104010 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409419f950 con 0x7f4094104010 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.951+0000 7f4090c00640 1 -- 172.21.15.114:0/839279713 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f408400ce60 con 0x7f4094104010 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.952+0000 7f4090c00640 1 -- 172.21.15.114:0/839279713 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084010690 con 0x7f4094104010 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.952+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40941a7230 con 0x7f4094104010 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.953+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40941043f0 con 0x7f4094104010 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.954+0000 7f4090c00640 1 -- 172.21.15.114:0/839279713 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f40840046c0 con 0x7f4094104010 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.956+0000 7f4090c00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40780a3240 0x7f40780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:56.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.956+0000 7f4090c00640 1 -- 172.21.15.114:0/839279713 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f40840d00f0 con 0x7f4094104010 2024-09-19T06:49:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.956+0000 7f4093400640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40780a3240 0x7f40780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.957+0000 7f4093400640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40780a3240 0x7f40780a5700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4080000c00 tx=0x7f4080005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:56.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:56.960+0000 7f4090c00640 1 -- 172.21.15.114:0/839279713 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f408409ca50 con 0x7f4094104010 2024-09-19T06:49:57.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.177+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f409410c380 con 0x7f4094104010 2024-09-19T06:49:57.220 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.219+0000 7f4090c00640 1 -- 172.21.15.114:0/839279713 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f408409c350 con 0x7f4094104010 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40780a3240 msgr2=0x7f40780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40780a3240 0x7f40780a5700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4080000c00 tx=0x7f4080005b50 comp rx=0 tx=0).stop 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4094104010 msgr2=0x7f409419a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4094104010 0x7f409419a6b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f408400e9f0 tx=0x7f408400edf0 comp rx=0 tx=0).stop 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 shutdown_connections 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f40780a3240 0x7f40780a5700 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4094109140 0x7f409419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f40941049c0 0x7f409419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:57.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 --2- 172.21.15.114:0/839279713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4094104010 0x7f409419a6b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:57.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 >> 172.21.15.114:0/839279713 conn(0x7f40940ffa50 msgr2=0x7f4094100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:57.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.221+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 shutdown_connections 2024-09-19T06:49:57.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:57.222+0000 7f409aa00640 1 -- 172.21.15.114:0/839279713 wait complete. 2024-09-19T06:49:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:57 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] up:active 2024-09-19T06:49:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:57 smithi118 ceph-mon[35527]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-19T06:49:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:57 smithi118 ceph-mon[35527]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-19T06:49:57.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:57 smithi118 ceph-mon[35527]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-19T06:49:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:57 smithi118 ceph-mon[35527]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-19T06:49:57.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:57 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/839279713' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-19T06:49:57.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:57 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] up:active 2024-09-19T06:49:57.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:57 smithi114 ceph-mon[30820]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-19T06:49:57.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:57 smithi114 ceph-mon[30820]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-19T06:49:57.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:57 smithi114 ceph-mon[30820]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-19T06:49:57.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:57 smithi114 ceph-mon[30820]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-19T06:49:57.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:57 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/839279713' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-19T06:49:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:57 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] up:active 2024-09-19T06:49:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:57 smithi116 ceph-mon[35411]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-19T06:49:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:57 smithi116 ceph-mon[35411]: fsmap cephfs:2 {0=i=up:active,1=b=up:active} 10 up:standby 2024-09-19T06:49:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:57 smithi116 ceph-mon[35411]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:creating} 9 up:standby 2024-09-19T06:49:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:57 smithi116 ceph-mon[35411]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-19T06:49:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:57 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/839279713' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-19T06:49:57.990 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-09-19T06:49:58.384 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:49:58.506 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-i[66010]: 2024-09-19T06:49:58.019+0000 7f7e6a4f1640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-19T06:49:58.506 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-i[66010]: 2024-09-19T06:49:58.019+0000 7f7e6a4f1640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-19T06:49:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-mon[35527]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:49:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/839279713' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-19T06:49:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] up:active 2024-09-19T06:49:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-mon[35527]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-19T06:49:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-mon[35527]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-19T06:49:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-mon[35527]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-19T06:49:58.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:58 smithi118 ceph-mon[35527]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-19T06:49:58.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:58 smithi114 ceph-mon[30820]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:49:58.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:58 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/839279713' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-19T06:49:58.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:58 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] up:active 2024-09-19T06:49:58.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:58 smithi114 ceph-mon[30820]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-19T06:49:58.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:58 smithi114 ceph-mon[30820]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-19T06:49:58.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:58 smithi114 ceph-mon[30820]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-19T06:49:58.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:58 smithi114 ceph-mon[30820]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-19T06:49:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:58 smithi116 ceph-mon[35411]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:49:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:58 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/839279713' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-19T06:49:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:58 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] up:active 2024-09-19T06:49:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:58 smithi116 ceph-mon[35411]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-19T06:49:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:58 smithi116 ceph-mon[35411]: fsmap cephfs:3 {0=i=up:active,1=b=up:active,2=c=up:active} 9 up:standby 2024-09-19T06:49:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:58 smithi116 ceph-mon[35411]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:creating} 8 up:standby 2024-09-19T06:49:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:58 smithi116 ceph-mon[35411]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-09-19T06:49:59.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 -- 172.21.15.114:0/2630629503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b741049c0 msgr2=0x7f5b74108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:59.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/2630629503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b741049c0 0x7f5b74108a10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c009900 tx=0x7f5b5c02a410 comp rx=0 tx=0).stop 2024-09-19T06:49:59.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 -- 172.21.15.114:0/2630629503 shutdown_connections 2024-09-19T06:49:59.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/2630629503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b74109140 0x7f5b74110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:59.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/2630629503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b741049c0 0x7f5b74108a10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:59.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/2630629503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74104010 0x7f5b741043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 -- 172.21.15.114:0/2630629503 >> 172.21.15.114:0/2630629503 conn(0x7f5b740ffa50 msgr2=0x7f5b74101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 -- 172.21.15.114:0/2630629503 shutdown_connections 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.018+0000 7f5b7ae00640 1 -- 172.21.15.114:0/2630629503 wait complete. 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.019+0000 7f5b7ae00640 1 Processor -- start 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.019+0000 7f5b7ae00640 1 -- start start 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.019+0000 7f5b7ae00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74104010 0x7f5b7419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.019+0000 7f5b7ae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b741049c0 0x7f5b7419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.019+0000 7f5b7ae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b74109140 0x7f5b7419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b7ae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5b74114160 con 0x7f5b74109140 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b7ae00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5b74113fe0 con 0x7f5b74104010 2024-09-19T06:49:59.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b7ae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f5b741142e0 con 0x7f5b741049c0 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74104010 0x7f5b7419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74104010 0x7f5b7419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:52118/0 (socket says 172.21.15.114:52118) 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 -- 172.21.15.114:0/3006208571 learned_addr learned my addr 172.21.15.114:0/3006208571 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b78e00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b74109140 0x7f5b7419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b6b400640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b741049c0 0x7f5b7419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 -- 172.21.15.114:0/3006208571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b741049c0 msgr2=0x7f5b7419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b741049c0 0x7f5b7419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 -- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b74109140 msgr2=0x7f5b7419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b74109140 0x7f5b7419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:49:59.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 -- 172.21.15.114:0/3006208571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7419f5f0 con 0x7f5b74104010 2024-09-19T06:49:59.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.020+0000 7f5b73e00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74104010 0x7f5b7419a640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9f0 tx=0x7f5b6400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:59.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.021+0000 7f5b71600640 1 -- 172.21.15.114:0/3006208571 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400cd80 con 0x7f5b74104010 2024-09-19T06:49:59.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.021+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7419f8e0 con 0x7f5b74104010 2024-09-19T06:49:59.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.021+0000 7f5b71600640 1 -- 172.21.15.114:0/3006208571 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b64010910 con 0x7f5b74104010 2024-09-19T06:49:59.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.021+0000 7f5b71600640 1 -- 172.21.15.114:0/3006208571 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b640188f0 con 0x7f5b74104010 2024-09-19T06:49:59.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.021+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b741a71c0 con 0x7f5b74104010 2024-09-19T06:49:59.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.022+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b741043f0 con 0x7f5b74104010 2024-09-19T06:49:59.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.024+0000 7f5b71600640 1 -- 172.21.15.114:0/3006208571 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7f5b64010ac0 con 0x7f5b74104010 2024-09-19T06:49:59.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.025+0000 7f5b71600640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5b580a3240 0x7f5b580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:49:59.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.025+0000 7f5b71600640 1 -- 172.21.15.114:0/3006208571 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f5b640d0400 con 0x7f5b74104010 2024-09-19T06:49:59.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.025+0000 7f5b6b400640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5b580a3240 0x7f5b580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:49:59.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.026+0000 7f5b6b400640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5b580a3240 0x7f5b580a5700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c009230 tx=0x7f5b5c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:49:59.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.027+0000 7f5b71600640 1 -- 172.21.15.114:0/3006208571 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5b6409c770 con 0x7f5b74104010 2024-09-19T06:49:59.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:49:59.253+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f5b7410c380 con 0x7f5b74104010 2024-09-19T06:49:59.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:59 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] up:active 2024-09-19T06:49:59.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:59 smithi118 ceph-mon[35527]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-19T06:49:59.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:59 smithi118 ceph-mon[35527]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-19T06:49:59.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:59 smithi118 ceph-mon[35527]: Cluster is now healthy 2024-09-19T06:49:59.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:59 smithi118 ceph-mon[35527]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-19T06:49:59.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:59 smithi118 ceph-mon[35527]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-19T06:49:59.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:49:59 smithi118 ceph-mon[35527]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-19T06:49:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:59 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] up:active 2024-09-19T06:49:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:59 smithi114 ceph-mon[30820]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-19T06:49:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:59 smithi114 ceph-mon[30820]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-19T06:49:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:59 smithi114 ceph-mon[30820]: Cluster is now healthy 2024-09-19T06:49:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:59 smithi114 ceph-mon[30820]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-19T06:49:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:59 smithi114 ceph-mon[30820]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-19T06:49:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:49:59 smithi114 ceph-mon[30820]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-19T06:49:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:59 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] up:active 2024-09-19T06:49:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:59 smithi116 ceph-mon[35411]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-19T06:49:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:59 smithi116 ceph-mon[35411]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-19T06:49:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:59 smithi116 ceph-mon[35411]: Cluster is now healthy 2024-09-19T06:49:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:59 smithi116 ceph-mon[35411]: fsmap cephfs:4 {0=i=up:active,1=b=up:active,2=c=up:active,3=f=up:active} 8 up:standby 2024-09-19T06:49:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:59 smithi116 ceph-mon[35411]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-19T06:49:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:49:59 smithi116 ceph-mon[35411]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-09-19T06:50:00.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.248+0000 7f5b71600640 1 -- 172.21.15.114:0/3006208571 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v29) ==== 115+0+0 (secure 0 0 0) 0x7f5b640a0a70 con 0x7f5b74104010 2024-09-19T06:50:00.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5b580a3240 msgr2=0x7f5b580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:00.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5b580a3240 0x7f5b580a5700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c009230 tx=0x7f5b5c0058d0 comp rx=0 tx=0).stop 2024-09-19T06:50:00.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74104010 msgr2=0x7f5b7419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74104010 0x7f5b7419a640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400e9f0 tx=0x7f5b6400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 shutdown_connections 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f5b580a3240 0x7f5b580a5700 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b74109140 0x7f5b7419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b741049c0 0x7f5b7419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 --2- 172.21.15.114:0/3006208571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b74104010 0x7f5b7419a640 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 >> 172.21.15.114:0/3006208571 conn(0x7f5b740ffa50 msgr2=0x7f5b74100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 shutdown_connections 2024-09-19T06:50:00.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:00.251+0000 7f5b7ae00640 1 -- 172.21.15.114:0/3006208571 wait complete. 2024-09-19T06:50:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:00 smithi118 ceph-mon[35527]: pgmap v327: 129 pgs: 129 active+clean; 583 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:00 smithi118 ceph-mon[35527]: mds.? [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] up:active 2024-09-19T06:50:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:00 smithi118 ceph-mon[35527]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-19T06:50:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:00 smithi118 ceph-mon[35527]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:50:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:00 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3006208571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T06:50:00.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:00 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T06:50:00.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:00 smithi118 ceph-mon[35527]: overall HEALTH_OK 2024-09-19T06:50:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:00 smithi114 ceph-mon[30820]: pgmap v327: 129 pgs: 129 active+clean; 583 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:00 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] up:active 2024-09-19T06:50:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:00 smithi114 ceph-mon[30820]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-19T06:50:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:00 smithi114 ceph-mon[30820]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:50:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:00 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3006208571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T06:50:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:00 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T06:50:00.621 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:00 smithi114 ceph-mon[30820]: overall HEALTH_OK 2024-09-19T06:50:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:00 smithi116 ceph-mon[35411]: pgmap v327: 129 pgs: 129 active+clean; 583 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:00 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] up:active 2024-09-19T06:50:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:00 smithi116 ceph-mon[35411]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-19T06:50:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:00 smithi116 ceph-mon[35411]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:50:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:00 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3006208571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T06:50:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:00 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-19T06:50:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:00 smithi116 ceph-mon[35411]: overall HEALTH_OK 2024-09-19T06:50:01.002 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-09-19T06:50:01.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-09-19T06:50:01.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:01 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-19T06:50:01.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:01 smithi114 ceph-mon[30820]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:50:01.374 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:01.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:01 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-19T06:50:01.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:01 smithi118 ceph-mon[35527]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:50:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:01 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-19T06:50:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:01 smithi116 ceph-mon[35411]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:50:01.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.936+0000 7fb1de600640 1 -- 172.21.15.114:0/2933746670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8106f10 msgr2=0x7fb1d810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:01.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.936+0000 7fb1de600640 1 --2- 172.21.15.114:0/2933746670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8106f10 0x7fb1d810ea90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009ca0 tx=0x7fb1d002a8c0 comp rx=0 tx=0).stop 2024-09-19T06:50:01.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.937+0000 7fb1de600640 1 -- 172.21.15.114:0/2933746670 shutdown_connections 2024-09-19T06:50:01.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.937+0000 7fb1de600640 1 --2- 172.21.15.114:0/2933746670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8106f10 0x7fb1d810ea90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:01.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.937+0000 7fb1de600640 1 --2- 172.21.15.114:0/2933746670 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d8102790 0x7fb1d81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:01.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.937+0000 7fb1de600640 1 --2- 172.21.15.114:0/2933746670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d8101de0 0x7fb1d81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.937+0000 7fb1de600640 1 -- 172.21.15.114:0/2933746670 >> 172.21.15.114:0/2933746670 conn(0x7fb1d80fd8c0 msgr2=0x7fb1d80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.937+0000 7fb1de600640 1 -- 172.21.15.114:0/2933746670 shutdown_connections 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.937+0000 7fb1de600640 1 -- 172.21.15.114:0/2933746670 wait complete. 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1de600640 1 Processor -- start 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1de600640 1 -- start start 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1de600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d8101de0 0x7fb1d81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1de600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8102790 0x7fb1d81988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1de600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d8106f10 0x7fb1d819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1de600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d8111fc0 con 0x7fb1d8102790 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1de600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d8111e40 con 0x7fb1d8106f10 2024-09-19T06:50:01.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1de600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d8112140 con 0x7fb1d8101de0 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1d6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8102790 0x7fb1d81988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1d6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8102790 0x7fb1d81988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53348/0 (socket says 172.21.15.114:53348) 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1d6a00640 1 -- 172.21.15.114:0/1148299388 learned_addr learned my addr 172.21.15.114:0/1148299388 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.938+0000 7fb1d7400640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d8101de0 0x7fb1d81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d7e00640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d8106f10 0x7fb1d819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d7400640 1 -- 172.21.15.114:0/1148299388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d8106f10 msgr2=0x7fb1d819cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d7400640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d8106f10 0x7fb1d819cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d7400640 1 -- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8102790 msgr2=0x7fb1d81988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d7400640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8102790 0x7fb1d81988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d7400640 1 -- 172.21.15.114:0/1148299388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d819d350 con 0x7fb1d8101de0 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d7e00640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d8106f10 0x7fb1d819cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d6a00640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8102790 0x7fb1d81988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:01.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1d7400640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d8101de0 0x7fb1d81983a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800e9f0 tx=0x7fb1c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1cfe00640 1 -- 172.21.15.114:0/1148299388 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c800cd80 con 0x7fb1d8101de0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d819d640 con 0x7fb1d8101de0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1cfe00640 1 -- 172.21.15.114:0/1148299388 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c80035f0 con 0x7fb1d8101de0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.939+0000 7fb1cfe00640 1 -- 172.21.15.114:0/1148299388 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8010670 con 0x7fb1d8101de0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.940+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d81a4f20 con 0x7fb1d8101de0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.940+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1d81021c0 con 0x7fb1d8101de0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.943+0000 7fb1cfe00640 1 -- 172.21.15.114:0/1148299388 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) ==== 138356+0+0 (secure 0 0 0) 0x7fb1c80037a0 con 0x7fb1d8101de0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.944+0000 7fb1cfe00640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb1bc0a3240 0x7fb1bc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.944+0000 7fb1cfe00640 1 -- 172.21.15.114:0/1148299388 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb1c80d00f0 con 0x7fb1d8101de0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.944+0000 7fb1d6a00640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb1bc0a3240 0x7fb1bc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.945+0000 7fb1d6a00640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb1bc0a3240 0x7fb1bc0a5700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80fdce0 tx=0x7fb1c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:01.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:01.946+0000 7fb1cfe00640 1 -- 172.21.15.114:0/1148299388 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb1c809cae0 con 0x7fb1d8101de0 2024-09-19T06:50:02.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.147+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb1d8062630 con 0x7fb1bc0a3240 2024-09-19T06:50:02.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.176+0000 7fb1cfe00640 1 -- 172.21.15.114:0/1148299388 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb1d8062630 con 0x7fb1bc0a3240 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb1bc0a3240 msgr2=0x7fb1bc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb1bc0a3240 0x7fb1bc0a5700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80fdce0 tx=0x7fb1c4005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d8101de0 msgr2=0x7fb1d81983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d8101de0 0x7fb1d81983a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800e9f0 tx=0x7fb1c800eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 shutdown_connections 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb1bc0a3240 0x7fb1bc0a5700 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d8106f10 0x7fb1d819cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1d8102790 0x7fb1d81988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 --2- 172.21.15.114:0/1148299388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb1d8101de0 0x7fb1d81983a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:02.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.178+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 >> 172.21.15.114:0/1148299388 conn(0x7fb1d80fd8c0 msgr2=0x7fb1d80feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:02.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.179+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 shutdown_connections 2024-09-19T06:50:02.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:02.179+0000 7fb1de600640 1 -- 172.21.15.114:0/1148299388 wait complete. 2024-09-19T06:50:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:02 smithi114 ceph-mon[30820]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.7 KiB/s wr, 17 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:50:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:50:02.154+0000 7fe8e4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T06:50:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:50:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:50:02.154+0000 7fe8e4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T06:50:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:50:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:50:02.154+0000 7fe8e4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T06:50:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:50:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:50:02.154+0000 7fe8e4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T06:50:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 06:50:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T06:50:02.154+0000 7fe8e4800640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-19T06:50:02.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:02 smithi118 ceph-mon[35527]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.7 KiB/s wr, 17 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:02 smithi116 ceph-mon[35411]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.7 KiB/s wr, 17 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:02.917 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-19T06:50:03.293 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:03.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:03 smithi118 ceph-mon[35527]: from='client.24736 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:03.506 INFO:journalctl@ceph.mds.i.smithi118.stdout:Sep 19 06:50:03 smithi118 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-i[66010]: 2024-09-19T06:50:03.020+0000 7f7e6a4f1640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-09-19T06:50:03.627 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:03 smithi114 ceph-mon[30820]: from='client.24736 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:03 smithi116 ceph-mon[35411]: from='client.24736 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:03.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 -- 172.21.15.114:0/858862816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01079c0 msgr2=0x7fa5b0107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 --2- 172.21.15.114:0/858862816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01079c0 0x7fa5b0107da0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009930 tx=0x7fa5a002a440 comp rx=0 tx=0).stop 2024-09-19T06:50:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 -- 172.21.15.114:0/858862816 shutdown_connections 2024-09-19T06:50:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 --2- 172.21.15.114:0/858862816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5b0102360 0x7fa5b010e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 --2- 172.21.15.114:0/858862816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b01019c0 0x7fa5b0101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 --2- 172.21.15.114:0/858862816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01079c0 0x7fa5b0107da0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 -- 172.21.15.114:0/858862816 >> 172.21.15.114:0/858862816 conn(0x7fa5b00fd740 msgr2=0x7fa5b00ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 -- 172.21.15.114:0/858862816 shutdown_connections 2024-09-19T06:50:03.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.923+0000 7fa5b5200640 1 -- 172.21.15.114:0/858862816 wait complete. 2024-09-19T06:50:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.924+0000 7fa5b5200640 1 Processor -- start 2024-09-19T06:50:03.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.924+0000 7fa5b5200640 1 -- start start 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5b5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01019c0 0x7fa5b01982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5b5200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5b0102360 0x7fa5b01987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5b5200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b01079c0 0x7fa5b019cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5b5200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0111e10 con 0x7fa5b01019c0 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5b5200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0111c90 con 0x7fa5b01079c0 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5b5200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0111f90 con 0x7fa5b0102360 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5af400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01019c0 0x7fa5b01982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5af400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01019c0 0x7fa5b01982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53382/0 (socket says 172.21.15.114:53382) 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5af400640 1 -- 172.21.15.114:0/1569427735 learned_addr learned my addr 172.21.15.114:0/1569427735 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5afe00640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b01079c0 0x7fa5b019cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5aea00640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5b0102360 0x7fa5b01987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:03.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5afe00640 1 -- 172.21.15.114:0/1569427735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5b0102360 msgr2=0x7fa5b01987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5afe00640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5b0102360 0x7fa5b01987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5afe00640 1 -- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01019c0 msgr2=0x7fa5b01982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5afe00640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01019c0 0x7fa5b01982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5afe00640 1 -- 172.21.15.114:0/1569427735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b019d260 con 0x7fa5b01079c0 2024-09-19T06:50:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5aea00640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5b0102360 0x7fa5b01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:03.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.925+0000 7fa5af400640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01019c0 0x7fa5b01982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.926+0000 7fa5afe00640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b01079c0 0x7fa5b019cb80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009fd0 tx=0x7fa5a800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.926+0000 7fa5a7e00640 1 -- 172.21.15.114:0/1569427735 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a800ce20 con 0x7fa5b01079c0 2024-09-19T06:50:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.926+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b019d550 con 0x7fa5b01079c0 2024-09-19T06:50:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.926+0000 7fa5a7e00640 1 -- 172.21.15.114:0/1569427735 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5a8005020 con 0x7fa5b01079c0 2024-09-19T06:50:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.926+0000 7fa5a7e00640 1 -- 172.21.15.114:0/1569427735 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8010720 con 0x7fa5b01079c0 2024-09-19T06:50:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.926+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b01a4e30 con 0x7fa5b01079c0 2024-09-19T06:50:03.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.927+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b0101e20 con 0x7fa5b01079c0 2024-09-19T06:50:03.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.930+0000 7fa5a7e00640 1 -- 172.21.15.114:0/1569427735 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fa5a8005540 con 0x7fa5b01079c0 2024-09-19T06:50:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.931+0000 7fa5a7e00640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa5940a32d0 0x7fa5940a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.931+0000 7fa5a7e00640 1 -- 172.21.15.114:0/1569427735 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa5a80d0680 con 0x7fa5b01079c0 2024-09-19T06:50:03.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.931+0000 7fa5af400640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa5940a32d0 0x7fa5940a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:03.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.932+0000 7fa5af400640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa5940a32d0 0x7fa5940a5790 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a0031510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:03.934 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:03.933+0000 7fa5a7e00640 1 -- 172.21.15.114:0/1569427735 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fa5a809ce80 con 0x7fa5b01079c0 2024-09-19T06:50:04.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.135+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa5b0062630 con 0x7fa5940a32d0 2024-09-19T06:50:04.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.162+0000 7fa5a7e00640 1 -- 172.21.15.114:0/1569427735 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa5b0062630 con 0x7fa5940a32d0 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa5940a32d0 msgr2=0x7fa5940a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa5940a32d0 0x7fa5940a5790 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0000c00 tx=0x7fa5a0031510 comp rx=0 tx=0).stop 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b01079c0 msgr2=0x7fa5b019cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b01079c0 0x7fa5b019cb80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009fd0 tx=0x7fa5a800ef10 comp rx=0 tx=0).stop 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 shutdown_connections 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fa5940a32d0 0x7fa5940a5790 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5b01079c0 0x7fa5b019cb80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5b0102360 0x7fa5b01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:04.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 --2- 172.21.15.114:0/1569427735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5b01019c0 0x7fa5b01982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:04.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.165+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 >> 172.21.15.114:0/1569427735 conn(0x7fa5b00fd740 msgr2=0x7fa5b00fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:04.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.166+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 shutdown_connections 2024-09-19T06:50:04.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:04.166+0000 7fa5b5200640 1 -- 172.21.15.114:0/1569427735 wait complete. 2024-09-19T06:50:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:04 smithi114 ceph-mon[30820]: pgmap v329: 129 pgs: 129 active+clean; 587 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.4 KiB/s wr, 16 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:04 smithi114 ceph-mon[30820]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-09-19T06:50:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:04 smithi116 ceph-mon[35411]: pgmap v329: 129 pgs: 129 active+clean; 587 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.4 KiB/s wr, 16 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:04 smithi116 ceph-mon[35411]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-09-19T06:50:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:04 smithi118 ceph-mon[35527]: pgmap v329: 129 pgs: 129 active+clean; 587 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.4 KiB/s wr, 16 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:04 smithi118 ceph-mon[35527]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-09-19T06:50:04.927 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-19T06:50:05.295 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:05.595 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:05 smithi114 ceph-mon[30820]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:05.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:05 smithi116 ceph-mon[35411]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:05.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:05 smithi118 ceph-mon[35527]: from='client.24704 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:05.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 -- 172.21.15.114:0/2743169161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2db0102790 msgr2=0x7f2db01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:05.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2743169161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2db0102790 0x7f2db01067e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009900 tx=0x7f2d9802a3f0 comp rx=0 tx=0).stop 2024-09-19T06:50:05.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 -- 172.21.15.114:0/2743169161 shutdown_connections 2024-09-19T06:50:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2743169161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2db0106f10 0x7f2db010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2743169161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2db0102790 0x7f2db01067e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2743169161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2db0101de0 0x7f2db01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 -- 172.21.15.114:0/2743169161 >> 172.21.15.114:0/2743169161 conn(0x7f2db00fd8c0 msgr2=0x7f2db00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 -- 172.21.15.114:0/2743169161 shutdown_connections 2024-09-19T06:50:05.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.874+0000 7f2db6a00640 1 -- 172.21.15.114:0/2743169161 wait complete. 2024-09-19T06:50:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.875+0000 7f2db6a00640 1 Processor -- start 2024-09-19T06:50:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.875+0000 7f2db6a00640 1 -- start start 2024-09-19T06:50:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.875+0000 7f2db6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2db0101de0 0x7f2db019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.875+0000 7f2db6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2db0102790 0x7f2db019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.875+0000 7f2db6a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2db0106f10 0x7f2db019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.875+0000 7f2db6a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0111f80 con 0x7f2db0101de0 2024-09-19T06:50:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.875+0000 7f2db6a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0111e00 con 0x7f2db0106f10 2024-09-19T06:50:05.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.875+0000 7f2db6a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0112100 con 0x7f2db0102790 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2dafe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2db0101de0 0x7f2db019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2db0102790 0x7f2db019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2db0102790 0x7f2db019ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:34646/0 (socket says 172.21.15.114:34646) 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 -- 172.21.15.114:0/2527224803 learned_addr learned my addr 172.21.15.114:0/2527224803 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2db4a00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2db0106f10 0x7f2db019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 -- 172.21.15.114:0/2527224803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2db0106f10 msgr2=0x7f2db019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2db0106f10 0x7f2db019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 -- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2db0101de0 msgr2=0x7f2db019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2db0101de0 0x7f2db019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 -- 172.21.15.114:0/2527224803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db019f6d0 con 0x7f2db0102790 2024-09-19T06:50:05.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.876+0000 7f2da7400640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2db0102790 0x7f2db019ac60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2d98004770 tx=0x7f2d98003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:05.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.877+0000 7f2dad600640 1 -- 172.21.15.114:0/2527224803 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9803f070 con 0x7f2db0102790 2024-09-19T06:50:05.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.877+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db019f960 con 0x7f2db0102790 2024-09-19T06:50:05.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.877+0000 7f2dad600640 1 -- 172.21.15.114:0/2527224803 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d98031ca0 con 0x7f2db0102790 2024-09-19T06:50:05.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.877+0000 7f2dad600640 1 -- 172.21.15.114:0/2527224803 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98035030 con 0x7f2db0102790 2024-09-19T06:50:05.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.877+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db01a72a0 con 0x7f2db0102790 2024-09-19T06:50:05.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.878+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db01021c0 con 0x7f2db0102790 2024-09-19T06:50:05.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.880+0000 7f2dad600640 1 -- 172.21.15.114:0/2527224803 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f2d98042070 con 0x7f2db0102790 2024-09-19T06:50:05.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.881+0000 7f2dad600640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d940a33d0 0x7f2d940a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:05.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.881+0000 7f2dad600640 1 -- 172.21.15.114:0/2527224803 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2d980f0720 con 0x7f2db0102790 2024-09-19T06:50:05.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.881+0000 7f2dafe00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d940a33d0 0x7f2d940a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:05.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.882+0000 7f2dafe00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d940a33d0 0x7f2d940a5890 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2da00095f0 tx=0x7f2da0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:05.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:05.883+0000 7f2dad600640 1 -- 172.21.15.114:0/2527224803 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2d980bc8d0 con 0x7f2db0102790 2024-09-19T06:50:06.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.082+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f2db010de00 con 0x7f2d940a33d0 2024-09-19T06:50:06.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.087+0000 7f2dad600640 1 -- 172.21.15.114:0/2527224803 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2db010de00 con 0x7f2d940a33d0 2024-09-19T06:50:06.089 INFO:teuthology.orchestra.run.smithi114.stdout:/volumes/qa/sv_0/8ed7d85a-7d4b-4c99-84c1-8e6dcbc4d8cb 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d940a33d0 msgr2=0x7f2d940a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d940a33d0 0x7f2d940a5890 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2da00095f0 tx=0x7f2da0005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2db0102790 msgr2=0x7f2db019ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2db0102790 0x7f2db019ac60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2d98004770 tx=0x7f2d98003c10 comp rx=0 tx=0).stop 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 shutdown_connections 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d940a33d0 0x7f2d940a5890 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2db0106f10 0x7f2db019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2db0102790 0x7f2db019ac60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 --2- 172.21.15.114:0/2527224803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2db0101de0 0x7f2db019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:06.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.089+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 >> 172.21.15.114:0/2527224803 conn(0x7f2db00fd8c0 msgr2=0x7f2db00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.090+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 shutdown_connections 2024-09-19T06:50:06.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:06.090+0000 7f2db6a00640 1 -- 172.21.15.114:0/2527224803 wait complete. 2024-09-19T06:50:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:06 smithi114 ceph-mon[30820]: pgmap v330: 129 pgs: 129 active+clean; 587 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 4.2 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:06 smithi116 ceph-mon[35411]: pgmap v330: 129 pgs: 129 active+clean; 587 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 4.2 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:06 smithi118 ceph-mon[35527]: pgmap v330: 129 pgs: 129 active+clean; 587 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 4.2 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-09-19T06:50:06.880 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-19T06:50:07.278 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:07.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:07 smithi118 ceph-mon[35527]: from='client.24754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:07.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:07 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:50:07.599 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:07 smithi114 ceph-mon[30820]: from='client.24754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:07.599 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:50:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:07 smithi116 ceph-mon[35411]: from='client.24754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:50:08.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 -- 172.21.15.114:0/3768288834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76801049a0 msgr2=0x7f76801089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:08.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 --2- 172.21.15.114:0/3768288834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76801049a0 0x7f76801089f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f766c009860 tx=0x7f766c02a340 comp rx=0 tx=0).stop 2024-09-19T06:50:08.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 -- 172.21.15.114:0/3768288834 shutdown_connections 2024-09-19T06:50:08.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 --2- 172.21.15.114:0/3768288834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7680109120 0x7f7680110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 --2- 172.21.15.114:0/3768288834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76801049a0 0x7f76801089f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 --2- 172.21.15.114:0/3768288834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7680103ff0 0x7f76801043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 -- 172.21.15.114:0/3768288834 >> 172.21.15.114:0/3768288834 conn(0x7f76800ffa90 msgr2=0x7f7680101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 -- 172.21.15.114:0/3768288834 shutdown_connections 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.320+0000 7f7687600640 1 -- 172.21.15.114:0/3768288834 wait complete. 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.321+0000 7f7687600640 1 Processor -- start 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.321+0000 7f7687600640 1 -- start start 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.321+0000 7f7687600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7680103ff0 0x7f768019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f7687600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801049a0 0x7f768019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f7687600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7680109120 0x7f768019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f7687600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7680114120 con 0x7f7680103ff0 2024-09-19T06:50:08.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f7687600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7680113fa0 con 0x7f76801049a0 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f7687600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f76801142a0 con 0x7f7680109120 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f7684c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7680103ff0 0x7f768019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f767fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801049a0 0x7f768019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f767fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801049a0 0x7f768019abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:57130/0 (socket says 172.21.15.114:57130) 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f767fe00640 1 -- 172.21.15.114:0/583471565 learned_addr learned my addr 172.21.15.114:0/583471565 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f7685600640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7680109120 0x7f768019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f767fe00640 1 -- 172.21.15.114:0/583471565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7680109120 msgr2=0x7f768019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f767fe00640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7680109120 0x7f768019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f767fe00640 1 -- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7680103ff0 msgr2=0x7f768019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f767fe00640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7680103ff0 0x7f768019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.322+0000 7f767fe00640 1 -- 172.21.15.114:0/583471565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768019f650 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.323+0000 7f767fe00640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801049a0 0x7f768019abe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f766c035040 tx=0x7f766c033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.323+0000 7f767d600640 1 -- 172.21.15.114:0/583471565 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c037040 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.323+0000 7f767d600640 1 -- 172.21.15.114:0/583471565 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f766c040070 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.323+0000 7f767d600640 1 -- 172.21.15.114:0/583471565 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c0039b0 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.323+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f768019f8e0 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.323+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76801a7220 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.325+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76801054d0 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.326+0000 7f767d600640 1 -- 172.21.15.114:0/583471565 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f766c0305e0 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.328+0000 7f767d600640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76680a32d0 0x7f76680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.328+0000 7f767d600640 1 -- 172.21.15.114:0/583471565 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f766c0efa90 con 0x7f76801049a0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.328+0000 7f7684c00640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76680a32d0 0x7f76680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:08.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.329+0000 7f7684c00640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76680a32d0 0x7f76680a5790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7674005fd0 tx=0x7f7674009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:08.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.331+0000 7f767d600640 1 -- 172.21.15.114:0/583471565 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f766c04a020 con 0x7f76801049a0 2024-09-19T06:50:08.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:08 smithi114 ceph-mon[30820]: pgmap v331: 129 pgs: 129 active+clean; 607 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.3 KiB/s wr, 22 op/s 2024-09-19T06:50:08.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:08 smithi116 ceph-mon[35411]: pgmap v331: 129 pgs: 129 active+clean; 607 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.3 KiB/s wr, 22 op/s 2024-09-19T06:50:08.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.610+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7680062630 con 0x7f76680a32d0 2024-09-19T06:50:08.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.634+0000 7f767d600640 1 -- 172.21.15.114:0/583471565 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7680062630 con 0x7f76680a32d0 2024-09-19T06:50:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76680a32d0 msgr2=0x7f76680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76680a32d0 0x7f76680a5790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7674005fd0 tx=0x7f7674009260 comp rx=0 tx=0).stop 2024-09-19T06:50:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801049a0 msgr2=0x7f768019abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801049a0 0x7f768019abe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f766c035040 tx=0x7f766c033b60 comp rx=0 tx=0).stop 2024-09-19T06:50:08.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 shutdown_connections 2024-09-19T06:50:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76680a32d0 0x7f76680a5790 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7680109120 0x7f768019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801049a0 0x7f768019abe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 --2- 172.21.15.114:0/583471565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7680103ff0 0x7f768019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 >> 172.21.15.114:0/583471565 conn(0x7f76800ffa90 msgr2=0x7f7680100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 shutdown_connections 2024-09-19T06:50:08.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:08.638+0000 7f7687600640 1 -- 172.21.15.114:0/583471565 wait complete. 2024-09-19T06:50:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:08 smithi118 ceph-mon[35527]: pgmap v331: 129 pgs: 129 active+clean; 607 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.3 KiB/s wr, 22 op/s 2024-09-19T06:50:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:08 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:09.502 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-19T06:50:09.937 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:10 smithi114 ceph-mon[30820]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:10 smithi114 ceph-mon[30820]: pgmap v332: 129 pgs: 129 active+clean; 607 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-09-19T06:50:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:10.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 -- 172.21.15.114:0/2154424613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c4077040 msgr2=0x7f78c41124f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:10.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 --2- 172.21.15.114:0/2154424613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c4077040 0x7f78c41124f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009900 tx=0x7f78ac02a3f0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 -- 172.21.15.114:0/2154424613 shutdown_connections 2024-09-19T06:50:10.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 --2- 172.21.15.114:0/2154424613 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78c4112a30 0x7f78c4114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 --2- 172.21.15.114:0/2154424613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c4077040 0x7f78c41124f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 --2- 172.21.15.114:0/2154424613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78c4076720 0x7f78c4076b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 -- 172.21.15.114:0/2154424613 >> 172.21.15.114:0/2154424613 conn(0x7f78c40ffa90 msgr2=0x7f78c4101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:10.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 -- 172.21.15.114:0/2154424613 shutdown_connections 2024-09-19T06:50:10.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.562+0000 7f78c8c00640 1 -- 172.21.15.114:0/2154424613 wait complete. 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.563+0000 7f78c8c00640 1 Processor -- start 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.563+0000 7f78c8c00640 1 -- start start 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c8c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78c4076720 0x7f78c41a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c8c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c4077040 0x7f78c41a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c8c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78c4112a30 0x7f78c419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c8c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78c41185f0 con 0x7f78c4112a30 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c8c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78c4118470 con 0x7f78c4077040 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c8c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f78c4118770 con 0x7f78c4076720 2024-09-19T06:50:10.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c1e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78c4076720 0x7f78c41a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c1e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78c4076720 0x7f78c41a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:46136/0 (socket says 172.21.15.114:46136) 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c1e00640 1 -- 172.21.15.114:0/621231022 learned_addr learned my addr 172.21.15.114:0/621231022 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78b9400640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c4077040 0x7f78c41a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c2800640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78c4112a30 0x7f78c419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c1e00640 1 -- 172.21.15.114:0/621231022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c4077040 msgr2=0x7f78c41a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c1e00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c4077040 0x7f78c41a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c1e00640 1 -- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78c4112a30 msgr2=0x7f78c419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c1e00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78c4112a30 0x7f78c419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.564+0000 7f78c1e00640 1 -- 172.21.15.114:0/621231022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c419efb0 con 0x7f78c4076720 2024-09-19T06:50:10.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.565+0000 7f78c1e00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78c4076720 0x7f78c41a46c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78b400e9f0 tx=0x7f78b400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.565+0000 7f78bb400640 1 -- 172.21.15.114:0/621231022 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b400cd80 con 0x7f78c4076720 2024-09-19T06:50:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.565+0000 7f78bb400640 1 -- 172.21.15.114:0/621231022 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b4010910 con 0x7f78c4076720 2024-09-19T06:50:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.565+0000 7f78bb400640 1 -- 172.21.15.114:0/621231022 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4018990 con 0x7f78c4076720 2024-09-19T06:50:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.565+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c419f2a0 con 0x7f78c4076720 2024-09-19T06:50:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.565+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c41ab590 con 0x7f78c4076720 2024-09-19T06:50:10.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.566+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c4076b00 con 0x7f78c4076720 2024-09-19T06:50:10.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.569+0000 7f78bb400640 1 -- 172.21.15.114:0/621231022 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f78b4010ac0 con 0x7f78c4076720 2024-09-19T06:50:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.571+0000 7f78bb400640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78a80a3350 0x7f78a80a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.571+0000 7f78bb400640 1 -- 172.21.15.114:0/621231022 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f78b40d00d0 con 0x7f78c4076720 2024-09-19T06:50:10.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.571+0000 7f78b9400640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78a80a3350 0x7f78a80a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:10.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.572+0000 7f78b9400640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78a80a3350 0x7f78a80a5810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f78ac000c00 tx=0x7f78ac005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:10.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.575+0000 7f78bb400640 1 -- 172.21.15.114:0/621231022 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f78b409c280 con 0x7f78c4076720 2024-09-19T06:50:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:10 smithi116 ceph-mon[35411]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:10 smithi116 ceph-mon[35411]: pgmap v332: 129 pgs: 129 active+clean; 607 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-09-19T06:50:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:10 smithi118 ceph-mon[35527]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:10.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:10 smithi118 ceph-mon[35527]: pgmap v332: 129 pgs: 129 active+clean; 607 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.2 KiB/s wr, 22 op/s 2024-09-19T06:50:10.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:10 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:10.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:10 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:10.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.786+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f78c4114b80 con 0x7f78a80a3350 2024-09-19T06:50:10.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.791+0000 7f78bb400640 1 -- 172.21.15.114:0/621231022 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f78c4114b80 con 0x7f78a80a3350 2024-09-19T06:50:10.793 INFO:teuthology.orchestra.run.smithi114.stdout:/volumes/qa/sv_1/b40f575c-53c4-4511-9cf7-9b6a2aef6040 2024-09-19T06:50:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78a80a3350 msgr2=0x7f78a80a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:10.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78a80a3350 0x7f78a80a5810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f78ac000c00 tx=0x7f78ac005b50 comp rx=0 tx=0).stop 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78c4076720 msgr2=0x7f78c41a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78c4076720 0x7f78c41a46c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78b400e9f0 tx=0x7f78b400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 shutdown_connections 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78a80a3350 0x7f78a80a5810 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78c4112a30 0x7f78c419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78c4077040 0x7f78c41a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 --2- 172.21.15.114:0/621231022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78c4076720 0x7f78c41a46c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 >> 172.21.15.114:0/621231022 conn(0x7f78c40ffa90 msgr2=0x7f78c4101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 shutdown_connections 2024-09-19T06:50:10.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:10.794+0000 7f78c8c00640 1 -- 172.21.15.114:0/621231022 wait complete. 2024-09-19T06:50:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:50:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:50:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:11.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:50:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:50:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:50:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:50:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:50:11.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:50:11.933 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:12.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:12 smithi114 ceph-mon[30820]: from='client.24766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:12.386 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:12 smithi114 ceph-mon[30820]: pgmap v333: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 23 op/s 2024-09-19T06:50:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 -- 172.21.15.114:0/1482153760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 msgr2=0x7f37b4114c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 --2- 172.21.15.114:0/1482153760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 0x7f37b4114c60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009ca0 tx=0x7f37ac02a8c0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 -- 172.21.15.114:0/1482153760 shutdown_connections 2024-09-19T06:50:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 --2- 172.21.15.114:0/1482153760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 0x7f37b4114c60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 --2- 172.21.15.114:0/1482153760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37b4077040 0x7f37b4112330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 --2- 172.21.15.114:0/1482153760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37b4076720 0x7f37b4076b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 -- 172.21.15.114:0/1482153760 >> 172.21.15.114:0/1482153760 conn(0x7f37b40ff8d0 msgr2=0x7f37b4101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:12.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 -- 172.21.15.114:0/1482153760 shutdown_connections 2024-09-19T06:50:12.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.495+0000 7f37ba200640 1 -- 172.21.15.114:0/1482153760 wait complete. 2024-09-19T06:50:12.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.496+0000 7f37ba200640 1 Processor -- start 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.496+0000 7f37ba200640 1 -- start start 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37ba200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37b4076720 0x7f37b41a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37ba200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37b4077040 0x7f37b41a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37ba200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 0x7f37b419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37ba200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37b4118400 con 0x7f37b4112870 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37ba200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37b4118280 con 0x7f37b4077040 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37ba200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f37b4118580 con 0x7f37b4076720 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 0x7f37b419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b8e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37b4076720 0x7f37b41a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37b4077040 0x7f37b41a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:12.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b9800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 0x7f37b419e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48084/0 (socket says 172.21.15.114:48084) 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b9800640 1 -- 172.21.15.114:0/1217477316 learned_addr learned my addr 172.21.15.114:0/1217477316 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b8e00640 1 -- 172.21.15.114:0/1217477316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37b4077040 msgr2=0x7f37b41a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b8e00640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37b4077040 0x7f37b41a4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b8e00640 1 -- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 msgr2=0x7f37b419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b8e00640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 0x7f37b419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37b8e00640 1 -- 172.21.15.114:0/1217477316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b419ef20 con 0x7f37b4076720 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.497+0000 7f37abe00640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37b4077040 0x7f37b41a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.498+0000 7f37b8e00640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37b4076720 0x7f37b41a45e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f37a400e9f0 tx=0x7f37a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.498+0000 7f37b2000640 1 -- 172.21.15.114:0/1217477316 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a400ccc0 con 0x7f37b4076720 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.498+0000 7f37b2000640 1 -- 172.21.15.114:0/1217477316 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a4010910 con 0x7f37b4076720 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.498+0000 7f37b2000640 1 -- 172.21.15.114:0/1217477316 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4018960 con 0x7f37b4076720 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.498+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b419f210 con 0x7f37b4076720 2024-09-19T06:50:12.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.498+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b41ab390 con 0x7f37b4076720 2024-09-19T06:50:12.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.499+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b4077e60 con 0x7f37b4076720 2024-09-19T06:50:12.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.501+0000 7f37b2000640 1 -- 172.21.15.114:0/1217477316 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f37a4010ac0 con 0x7f37b4076720 2024-09-19T06:50:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.502+0000 7f37b2000640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37980a3350 0x7f37980a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.502+0000 7f37b2000640 1 -- 172.21.15.114:0/1217477316 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f37a40d0150 con 0x7f37b4076720 2024-09-19T06:50:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.502+0000 7f37abe00640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37980a3350 0x7f37980a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:12.504 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.503+0000 7f37abe00640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37980a3350 0x7f37980a5810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f37b419ff00 tx=0x7f379c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:12.507 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.506+0000 7f37b2000640 1 -- 172.21.15.114:0/1217477316 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f37a409c300 con 0x7f37b4076720 2024-09-19T06:50:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:12 smithi116 ceph-mon[35411]: from='client.24766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:12 smithi116 ceph-mon[35411]: pgmap v333: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 23 op/s 2024-09-19T06:50:12.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.716+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37b402d7c0 con 0x7f37b4076720 2024-09-19T06:50:12.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.717+0000 7f37b2000640 1 -- 172.21.15.114:0/1217477316 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11999 (secure 0 0 0) 0x7f37a40a0600 con 0x7f37b4076720 2024-09-19T06:50:12.719 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:50:12.719 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"btime":"2024-09-19T06:50:00:244614+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T06:49:11.450651+0000","modified":"2024-09-19T06:49:59.255237+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24538,"mds_1":24524,"mds_2":24520,"mds_3":24508,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14520":{"gid":14520,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":21,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":20,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24538,"qdb_cluster":[24482,24508,24520,24524,24538]},"id":1}]} 2024-09-19T06:50:12.719 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-09-19T06:50:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37980a3350 msgr2=0x7f37980a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37980a3350 0x7f37980a5810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f37b419ff00 tx=0x7f379c005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37b4076720 msgr2=0x7f37b41a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37b4076720 0x7f37b41a45e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f37a400e9f0 tx=0x7f37a400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 shutdown_connections 2024-09-19T06:50:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37980a3350 0x7f37980a5810 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37b4112870 0x7f37b419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37b4077040 0x7f37b41a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 --2- 172.21.15.114:0/1217477316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37b4076720 0x7f37b41a45e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:12.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.719+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 >> 172.21.15.114:0/1217477316 conn(0x7f37b40ff8d0 msgr2=0x7f37b4101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:12.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.720+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 shutdown_connections 2024-09-19T06:50:12.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:12.720+0000 7f37ba200640 1 -- 172.21.15.114:0/1217477316 wait complete. 2024-09-19T06:50:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:12 smithi118 ceph-mon[35527]: from='client.24766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:50:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:12 smithi118 ceph-mon[35527]: pgmap v333: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 23 op/s 2024-09-19T06:50:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:13 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1217477316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:50:13.488 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:50:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:13 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1217477316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:50:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:13 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1217477316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:50:13.854 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 -- 172.21.15.114:0/2405514675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4801019c0 msgr2=0x7fb480101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 --2- 172.21.15.114:0/2405514675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4801019c0 0x7fb480101e20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb468009930 tx=0x7fb46802a420 comp rx=0 tx=0).stop 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 -- 172.21.15.114:0/2405514675 shutdown_connections 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 --2- 172.21.15.114:0/2405514675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb480102360 0x7fb48010e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 --2- 172.21.15.114:0/2405514675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4801019c0 0x7fb480101e20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 --2- 172.21.15.114:0/2405514675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4801079c0 0x7fb480107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 -- 172.21.15.114:0/2405514675 >> 172.21.15.114:0/2405514675 conn(0x7fb4800fd740 msgr2=0x7fb4800ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 -- 172.21.15.114:0/2405514675 shutdown_connections 2024-09-19T06:50:14.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.409+0000 7fb47fe00640 1 -- 172.21.15.114:0/2405514675 wait complete. 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.410+0000 7fb47fe00640 1 Processor -- start 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.410+0000 7fb47fe00640 1 -- start start 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.410+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4801019c0 0x7fb4801982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.410+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb480102360 0x7fb480198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.410+0000 7fb47fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4801079c0 0x7fb48019cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.410+0000 7fb47fe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb480111e10 con 0x7fb480102360 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.410+0000 7fb47fe00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb480111c90 con 0x7fb4801019c0 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.410+0000 7fb47fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb480111f90 con 0x7fb4801079c0 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb477e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb480102360 0x7fb480198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb477e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb480102360 0x7fb480198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48106/0 (socket says 172.21.15.114:48106) 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb477e00640 1 -- 172.21.15.114:0/3825612570 learned_addr learned my addr 172.21.15.114:0/3825612570 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb47ea00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4801019c0 0x7fb4801982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:14.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb47f400640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4801079c0 0x7fb48019cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb477e00640 1 -- 172.21.15.114:0/3825612570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4801079c0 msgr2=0x7fb48019cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb477e00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4801079c0 0x7fb48019cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb477e00640 1 -- 172.21.15.114:0/3825612570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4801019c0 msgr2=0x7fb4801982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb477e00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4801019c0 0x7fb4801982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb477e00640 1 -- 172.21.15.114:0/3825612570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb48019d2a0 con 0x7fb480102360 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.411+0000 7fb47ea00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4801019c0 0x7fb4801982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.412+0000 7fb477e00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb480102360 0x7fb480198830 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb46802a940 tx=0x7fb468003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.412+0000 7fb477400640 1 -- 172.21.15.114:0/3825612570 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46803f070 con 0x7fb480102360 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.412+0000 7fb477400640 1 -- 172.21.15.114:0/3825612570 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb468031c90 con 0x7fb480102360 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.412+0000 7fb477400640 1 -- 172.21.15.114:0/3825612570 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb468035030 con 0x7fb480102360 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.412+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb48019d530 con 0x7fb480102360 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.412+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb48019d9c0 con 0x7fb480102360 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.413+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb480101e20 con 0x7fb480102360 2024-09-19T06:50:14.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.415+0000 7fb477400640 1 -- 172.21.15.114:0/3825612570 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fb468042070 con 0x7fb480102360 2024-09-19T06:50:14.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.416+0000 7fb477400640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb4640a3350 0x7fb4640a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:14.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.416+0000 7fb477400640 1 -- 172.21.15.114:0/3825612570 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb4680f03e0 con 0x7fb480102360 2024-09-19T06:50:14.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.416+0000 7fb47ea00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb4640a3350 0x7fb4640a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:14.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.417+0000 7fb47ea00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb4640a3350 0x7fb4640a5810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb470009f40 tx=0x7fb470005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:14.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.421+0000 7fb477400640 1 -- 172.21.15.114:0/3825612570 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb4680bc590 con 0x7fb480102360 2024-09-19T06:50:14.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.599+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb48002d7c0 con 0x7fb480102360 2024-09-19T06:50:14.601 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:14 smithi114 ceph-mon[30820]: pgmap v334: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.4 KiB/s wr, 12 op/s 2024-09-19T06:50:14.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.601+0000 7fb477400640 1 -- 172.21.15.114:0/3825612570 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24854 (secure 0 0 0) 0x7fb4680c0890 con 0x7fb480102360 2024-09-19T06:50:14.602 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:50:14.603 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":93,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:49:52.180171+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:50:14.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb4640a3350 msgr2=0x7fb4640a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:14.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb4640a3350 0x7fb4640a5810 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb470009f40 tx=0x7fb470005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:14.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb480102360 msgr2=0x7fb480198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:14.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb480102360 0x7fb480198830 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb46802a940 tx=0x7fb468003b20 comp rx=0 tx=0).stop 2024-09-19T06:50:14.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 shutdown_connections 2024-09-19T06:50:14.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb4640a3350 0x7fb4640a5810 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4801079c0 0x7fb48019cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb480102360 0x7fb480198830 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 --2- 172.21.15.114:0/3825612570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb4801019c0 0x7fb4801982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:14.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 >> 172.21.15.114:0/3825612570 conn(0x7fb4800fd740 msgr2=0x7fb4800fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:14.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.603+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 shutdown_connections 2024-09-19T06:50:14.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:14.604+0000 7fb47fe00640 1 -- 172.21.15.114:0/3825612570 wait complete. 2024-09-19T06:50:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:14 smithi116 ceph-mon[35411]: pgmap v334: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.4 KiB/s wr, 12 op/s 2024-09-19T06:50:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:14 smithi118 ceph-mon[35527]: pgmap v334: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.4 KiB/s wr, 12 op/s 2024-09-19T06:50:15.259 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-19T06:50:15.259 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T06:50:15.628 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:15 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3825612570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:15 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3825612570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:15 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3825612570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:16.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 -- 172.21.15.114:0/696450437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1248108f60 msgr2=0x7f1248110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:16.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 --2- 172.21.15.114:0/696450437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1248108f60 0x7f1248110ae0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1240009ca0 tx=0x7f124002a8c0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 -- 172.21.15.114:0/696450437 shutdown_connections 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 --2- 172.21.15.114:0/696450437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1248108f60 0x7f1248110ae0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 --2- 172.21.15.114:0/696450437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12481047e0 0x7f1248108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 --2- 172.21.15.114:0/696450437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1248103e30 0x7f1248104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 -- 172.21.15.114:0/696450437 >> 172.21.15.114:0/696450437 conn(0x7f12480ff8d0 msgr2=0x7f1248101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 -- 172.21.15.114:0/696450437 shutdown_connections 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.245+0000 7f124ca00640 1 -- 172.21.15.114:0/696450437 wait complete. 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.246+0000 7f124ca00640 1 Processor -- start 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.246+0000 7f124ca00640 1 -- start start 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.246+0000 7f124ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1248103e30 0x7f124819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.246+0000 7f124ca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12481047e0 0x7f124819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:16.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.246+0000 7f124ca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1248108f60 0x7f124819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f124ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1248113fc0 con 0x7f1248103e30 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f124ca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1248113e40 con 0x7f12481047e0 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f124ca00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1248114140 con 0x7f1248108f60 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f1247400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1248103e30 0x7f124819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f123ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12481047e0 0x7f124819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f123ea00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12481047e0 0x7f124819a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:57194/0 (socket says 172.21.15.114:57194) 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f123ea00640 1 -- 172.21.15.114:0/2810250274 learned_addr learned my addr 172.21.15.114:0/2810250274 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f1247e00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1248108f60 0x7f124819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f123ea00640 1 -- 172.21.15.114:0/2810250274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1248108f60 msgr2=0x7f124819ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:16.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f123ea00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1248108f60 0x7f124819ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f123ea00640 1 -- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1248103e30 msgr2=0x7f124819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f123ea00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1248103e30 0x7f124819a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.247+0000 7f123ea00640 1 -- 172.21.15.114:0/2810250274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f124819f3b0 con 0x7f12481047e0 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.248+0000 7f123ea00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12481047e0 0x7f124819a940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f123000e9f0 tx=0x7f123000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.248+0000 7f1244c00640 1 -- 172.21.15.114:0/2810250274 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123000cd80 con 0x7f12481047e0 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.248+0000 7f1244c00640 1 -- 172.21.15.114:0/2810250274 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1230010910 con 0x7f12481047e0 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.248+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f124819f6a0 con 0x7f12481047e0 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.248+0000 7f1244c00640 1 -- 172.21.15.114:0/2810250274 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12300188f0 con 0x7f12481047e0 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.248+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12481a6f80 con 0x7f12481047e0 2024-09-19T06:50:16.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.249+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1248104210 con 0x7f12481047e0 2024-09-19T06:50:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.251+0000 7f1244c00640 1 -- 172.21.15.114:0/2810250274 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f1230010ac0 con 0x7f12481047e0 2024-09-19T06:50:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.252+0000 7f1244c00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f122c0a3350 0x7f122c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.253+0000 7f1244c00640 1 -- 172.21.15.114:0/2810250274 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f12300d03e0 con 0x7f12481047e0 2024-09-19T06:50:16.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.253+0000 7f1247400640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f122c0a3350 0x7f122c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:16.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.253+0000 7f1247400640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f122c0a3350 0x7f122c0a5810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1238009bd0 tx=0x7f1238005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:16.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.255+0000 7f1244c00640 1 -- 172.21.15.114:0/2810250274 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f123009c5f0 con 0x7f12481047e0 2024-09-19T06:50:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:16 smithi114 ceph-mon[30820]: pgmap v335: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-19T06:50:16.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.432+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1248062630 con 0x7f122c0a3350 2024-09-19T06:50:16.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.443+0000 7f1244c00640 1 -- 172.21.15.114:0/2810250274 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340667 (secure 0 0 0) 0x7f1248062630 con 0x7f122c0a3350 2024-09-19T06:50:16.457 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:50:16.460 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-19T06:50:16.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f122c0a3350 msgr2=0x7f122c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:16.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f122c0a3350 0x7f122c0a5810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1238009bd0 tx=0x7f1238005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:16.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12481047e0 msgr2=0x7f124819a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:16.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12481047e0 0x7f124819a940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f123000e9f0 tx=0x7f123000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 shutdown_connections 2024-09-19T06:50:16.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f122c0a3350 0x7f122c0a5810 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1248108f60 0x7f124819ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12481047e0 0x7f124819a940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 --2- 172.21.15.114:0/2810250274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1248103e30 0x7f124819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:16.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 >> 172.21.15.114:0/2810250274 conn(0x7f12480ff8d0 msgr2=0x7f1248100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 shutdown_connections 2024-09-19T06:50:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:16.448+0000 7f124ca00640 1 -- 172.21.15.114:0/2810250274 wait complete. 2024-09-19T06:50:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:16 smithi116 ceph-mon[35411]: pgmap v335: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-19T06:50:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:16 smithi118 ceph-mon[35527]: pgmap v335: 129 pgs: 129 active+clean; 627 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-19T06:50:17.206 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-19T06:50:15.098595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":641900,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":296,"num_read_kb":247,"num_write":415,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":341,"ondisk_log_size":341,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":345156,"kb_used_data":12548,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124596668,"statfs":{"total":1151940427776,"available":1151586988032,"internally_reserved":0,"allocated":12849152,"data_stored":8173497,"data_compressed":390732,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19139,"internal_metadata":340374845},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41266,"num_objects":4,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":143,"num_read_kb":110,"num_write":27,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003126"},"pg_stats":[{"pgid":"2.1f","version":"93'13","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:09.266563+0000","last_change":"2024-09-19T06:49:07.909205+0000","last_active":"2024-09-19T06:50:09.266563+0000","last_peered":"2024-09-19T06:50:09.266563+0000","last_clean":"2024-09-19T06:50:09.266563+0000","last_became_active":"2024-09-19T06:49:07.907471+0000","last_became_peered":"2024-09-19T06:49:07.907471+0000","last_unstale":"2024-09-19T06:50:09.266563+0000","last_undegraded":"2024-09-19T06:50:09.266563+0000","last_fullsized":"2024-09-19T06:50:09.266563+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:55:00.655927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.197214+0000","last_change":"2024-09-19T06:49:52.196586+0000","last_active":"2024-09-19T06:49:52.197214+0000","last_peered":"2024-09-19T06:49:52.197214+0000","last_clean":"2024-09-19T06:49:52.197214+0000","last_became_active":"2024-09-19T06:49:52.195680+0000","last_became_peered":"2024-09-19T06:49:52.195680+0000","last_unstale":"2024-09-19T06:49:52.197214+0000","last_undegraded":"2024-09-19T06:49:52.197214+0000","last_fullsized":"2024-09-19T06:49:52.197214+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:46:53.026767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192504+0000","last_change":"2024-09-19T06:49:20.424335+0000","last_active":"2024-09-19T06:49:52.192504+0000","last_peered":"2024-09-19T06:49:52.192504+0000","last_clean":"2024-09-19T06:49:52.192504+0000","last_became_active":"2024-09-19T06:49:20.423890+0000","last_became_peered":"2024-09-19T06:49:20.423890+0000","last_unstale":"2024-09-19T06:49:52.192504+0000","last_undegraded":"2024-09-19T06:49:52.192504+0000","last_fullsized":"2024-09-19T06:49:52.192504+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:30:49.042220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.228465+0000","last_change":"2024-09-19T06:49:08.185286+0000","last_active":"2024-09-19T06:49:52.228465+0000","last_peered":"2024-09-19T06:49:52.228465+0000","last_clean":"2024-09-19T06:49:52.228465+0000","last_became_active":"2024-09-19T06:49:08.184971+0000","last_became_peered":"2024-09-19T06:49:08.184971+0000","last_unstale":"2024-09-19T06:49:52.228465+0000","last_undegraded":"2024-09-19T06:49:52.228465+0000","last_fullsized":"2024-09-19T06:49:52.228465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:26.532594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.188629+0000","last_change":"2024-09-19T06:49:10.385824+0000","last_active":"2024-09-19T06:49:52.188629+0000","last_peered":"2024-09-19T06:49:52.188629+0000","last_clean":"2024-09-19T06:49:52.188629+0000","last_became_active":"2024-09-19T06:49:10.381448+0000","last_became_peered":"2024-09-19T06:49:10.381448+0000","last_unstale":"2024-09-19T06:49:52.188629+0000","last_undegraded":"2024-09-19T06:49:52.188629+0000","last_fullsized":"2024-09-19T06:49:52.188629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:39.192054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184609+0000","last_change":"2024-09-19T06:49:20.429638+0000","last_active":"2024-09-19T06:49:52.184609+0000","last_peered":"2024-09-19T06:49:52.184609+0000","last_clean":"2024-09-19T06:49:52.184609+0000","last_became_active":"2024-09-19T06:49:20.429099+0000","last_became_peered":"2024-09-19T06:49:20.429099+0000","last_unstale":"2024-09-19T06:49:52.184609+0000","last_undegraded":"2024-09-19T06:49:52.184609+0000","last_fullsized":"2024-09-19T06:49:52.184609+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:01:06.221726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'12","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:59.248615+0000","last_change":"2024-09-19T06:49:07.907830+0000","last_active":"2024-09-19T06:49:59.248615+0000","last_peered":"2024-09-19T06:49:59.248615+0000","last_clean":"2024-09-19T06:49:59.248615+0000","last_became_active":"2024-09-19T06:49:07.907510+0000","last_became_peered":"2024-09-19T06:49:07.907510+0000","last_unstale":"2024-09-19T06:49:59.248615+0000","last_undegraded":"2024-09-19T06:49:59.248615+0000","last_fullsized":"2024-09-19T06:49:59.248615+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:52:46.192671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410556+0000","last_change":"2024-09-19T06:49:10.371663+0000","last_active":"2024-09-19T06:49:52.410556+0000","last_peered":"2024-09-19T06:49:52.410556+0000","last_clean":"2024-09-19T06:49:52.410556+0000","last_became_active":"2024-09-19T06:49:10.371325+0000","last_became_peered":"2024-09-19T06:49:10.371325+0000","last_unstale":"2024-09-19T06:49:52.410556+0000","last_undegraded":"2024-09-19T06:49:52.410556+0000","last_fullsized":"2024-09-19T06:49:52.410556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:48:58.406933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185210+0000","last_change":"2024-09-19T06:49:20.429446+0000","last_active":"2024-09-19T06:49:52.185210+0000","last_peered":"2024-09-19T06:49:52.185210+0000","last_clean":"2024-09-19T06:49:52.185210+0000","last_became_active":"2024-09-19T06:49:20.428996+0000","last_became_peered":"2024-09-19T06:49:20.428996+0000","last_unstale":"2024-09-19T06:49:52.185210+0000","last_undegraded":"2024-09-19T06:49:52.185210+0000","last_fullsized":"2024-09-19T06:49:52.185210+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:39.542347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"93'5","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.245239+0000","last_change":"2024-09-19T06:49:07.906940+0000","last_active":"2024-09-19T06:49:58.245239+0000","last_peered":"2024-09-19T06:49:58.245239+0000","last_clean":"2024-09-19T06:49:58.245239+0000","last_became_active":"2024-09-19T06:49:07.906635+0000","last_became_peered":"2024-09-19T06:49:07.906635+0000","last_unstale":"2024-09-19T06:49:58.245239+0000","last_undegraded":"2024-09-19T06:49:58.245239+0000","last_fullsized":"2024-09-19T06:49:58.245239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:00.794049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.411553+0000","last_change":"2024-09-19T06:49:52.411553+0000","last_active":"2024-09-19T06:49:52.411553+0000","last_peered":"2024-09-19T06:49:52.411553+0000","last_clean":"2024-09-19T06:49:52.411553+0000","last_became_active":"2024-09-19T06:49:52.411251+0000","last_became_peered":"2024-09-19T06:49:52.411251+0000","last_unstale":"2024-09-19T06:49:52.411553+0000","last_undegraded":"2024-09-19T06:49:52.411553+0000","last_fullsized":"2024-09-19T06:49:52.411553+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:17.774698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184753+0000","last_change":"2024-09-19T06:49:20.427188+0000","last_active":"2024-09-19T06:49:52.184753+0000","last_peered":"2024-09-19T06:49:52.184753+0000","last_clean":"2024-09-19T06:49:52.184753+0000","last_became_active":"2024-09-19T06:49:20.426716+0000","last_became_peered":"2024-09-19T06:49:20.426716+0000","last_unstale":"2024-09-19T06:49:52.184753+0000","last_undegraded":"2024-09-19T06:49:52.184753+0000","last_fullsized":"2024-09-19T06:49:52.184753+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:25:09.086438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187080+0000","last_change":"2024-09-19T06:49:10.371433+0000","last_active":"2024-09-19T06:49:52.187080+0000","last_peered":"2024-09-19T06:49:52.187080+0000","last_clean":"2024-09-19T06:49:52.187080+0000","last_became_active":"2024-09-19T06:49:10.370947+0000","last_became_peered":"2024-09-19T06:49:10.370947+0000","last_unstale":"2024-09-19T06:49:52.187080+0000","last_undegraded":"2024-09-19T06:49:52.187080+0000","last_fullsized":"2024-09-19T06:49:52.187080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:40.954670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:55.221780+0000","last_change":"2024-09-19T06:49:07.907972+0000","last_active":"2024-09-19T06:49:55.221780+0000","last_peered":"2024-09-19T06:49:55.221780+0000","last_clean":"2024-09-19T06:49:55.221780+0000","last_became_active":"2024-09-19T06:49:07.907631+0000","last_became_peered":"2024-09-19T06:49:07.907631+0000","last_unstale":"2024-09-19T06:49:55.221780+0000","last_undegraded":"2024-09-19T06:49:55.221780+0000","last_fullsized":"2024-09-19T06:49:55.221780+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:12:16.298696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185090+0000","last_change":"2024-09-19T06:49:20.617446+0000","last_active":"2024-09-19T06:49:52.185090+0000","last_peered":"2024-09-19T06:49:52.185090+0000","last_clean":"2024-09-19T06:49:52.185090+0000","last_became_active":"2024-09-19T06:49:20.616950+0000","last_became_peered":"2024-09-19T06:49:20.616950+0000","last_unstale":"2024-09-19T06:49:52.185090+0000","last_undegraded":"2024-09-19T06:49:52.185090+0000","last_fullsized":"2024-09-19T06:49:52.185090+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:37:00.640364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.197003+0000","last_change":"2024-09-19T06:49:52.195937+0000","last_active":"2024-09-19T06:49:52.197003+0000","last_peered":"2024-09-19T06:49:52.197003+0000","last_clean":"2024-09-19T06:49:52.197003+0000","last_became_active":"2024-09-19T06:49:52.195615+0000","last_became_peered":"2024-09-19T06:49:52.195615+0000","last_unstale":"2024-09-19T06:49:52.197003+0000","last_undegraded":"2024-09-19T06:49:52.197003+0000","last_fullsized":"2024-09-19T06:49:52.197003+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:28:33.296849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.a","version":"93'3","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:57.233356+0000","last_change":"2024-09-19T06:49:07.906081+0000","last_active":"2024-09-19T06:49:57.233356+0000","last_peered":"2024-09-19T06:49:57.233356+0000","last_clean":"2024-09-19T06:49:57.233356+0000","last_became_active":"2024-09-19T06:49:07.905559+0000","last_became_peered":"2024-09-19T06:49:07.905559+0000","last_unstale":"2024-09-19T06:49:57.233356+0000","last_undegraded":"2024-09-19T06:49:57.233356+0000","last_fullsized":"2024-09-19T06:49:57.233356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:30:56.922084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184922+0000","last_change":"2024-09-19T06:49:20.431255+0000","last_active":"2024-09-19T06:49:52.184922+0000","last_peered":"2024-09-19T06:49:52.184922+0000","last_clean":"2024-09-19T06:49:52.184922+0000","last_became_active":"2024-09-19T06:49:20.430727+0000","last_became_peered":"2024-09-19T06:49:20.430727+0000","last_unstale":"2024-09-19T06:49:52.184922+0000","last_undegraded":"2024-09-19T06:49:52.184922+0000","last_fullsized":"2024-09-19T06:49:52.184922+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:01.761121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189565+0000","last_change":"2024-09-19T06:49:10.382739+0000","last_active":"2024-09-19T06:49:52.189565+0000","last_peered":"2024-09-19T06:49:52.189565+0000","last_clean":"2024-09-19T06:49:52.189565+0000","last_became_active":"2024-09-19T06:49:10.382216+0000","last_became_peered":"2024-09-19T06:49:10.382216+0000","last_unstale":"2024-09-19T06:49:52.189565+0000","last_undegraded":"2024-09-19T06:49:52.189565+0000","last_fullsized":"2024-09-19T06:49:52.189565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:31:37.908345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"93'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.243678+0000","last_change":"2024-09-19T06:49:07.906876+0000","last_active":"2024-09-19T06:49:58.243678+0000","last_peered":"2024-09-19T06:49:58.243678+0000","last_clean":"2024-09-19T06:49:58.243678+0000","last_became_active":"2024-09-19T06:49:07.906567+0000","last_became_peered":"2024-09-19T06:49:07.906567+0000","last_unstale":"2024-09-19T06:49:58.243678+0000","last_undegraded":"2024-09-19T06:49:58.243678+0000","last_fullsized":"2024-09-19T06:49:58.243678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:21:39.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192165+0000","last_change":"2024-09-19T06:49:20.437455+0000","last_active":"2024-09-19T06:49:52.192165+0000","last_peered":"2024-09-19T06:49:52.192165+0000","last_clean":"2024-09-19T06:49:52.192165+0000","last_became_active":"2024-09-19T06:49:20.437050+0000","last_became_peered":"2024-09-19T06:49:20.437050+0000","last_unstale":"2024-09-19T06:49:52.192165+0000","last_undegraded":"2024-09-19T06:49:52.192165+0000","last_fullsized":"2024-09-19T06:49:52.192165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:46:04.206859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189107+0000","last_change":"2024-09-19T06:49:10.369037+0000","last_active":"2024-09-19T06:49:52.189107+0000","last_peered":"2024-09-19T06:49:52.189107+0000","last_clean":"2024-09-19T06:49:52.189107+0000","last_became_active":"2024-09-19T06:49:10.368653+0000","last_became_peered":"2024-09-19T06:49:10.368653+0000","last_unstale":"2024-09-19T06:49:52.189107+0000","last_undegraded":"2024-09-19T06:49:52.189107+0000","last_fullsized":"2024-09-19T06:49:52.189107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:46:02.268843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:55.222469+0000","last_change":"2024-09-19T06:49:08.185517+0000","last_active":"2024-09-19T06:49:55.222469+0000","last_peered":"2024-09-19T06:49:55.222469+0000","last_clean":"2024-09-19T06:49:55.222469+0000","last_became_active":"2024-09-19T06:49:08.185138+0000","last_became_peered":"2024-09-19T06:49:08.185138+0000","last_unstale":"2024-09-19T06:49:55.222469+0000","last_undegraded":"2024-09-19T06:49:55.222469+0000","last_fullsized":"2024-09-19T06:49:55.222469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:57:59.082821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185580+0000","last_change":"2024-09-19T06:49:20.617423+0000","last_active":"2024-09-19T06:49:52.185580+0000","last_peered":"2024-09-19T06:49:52.185580+0000","last_clean":"2024-09-19T06:49:52.185580+0000","last_became_active":"2024-09-19T06:49:20.616834+0000","last_became_peered":"2024-09-19T06:49:20.616834+0000","last_unstale":"2024-09-19T06:49:52.185580+0000","last_undegraded":"2024-09-19T06:49:52.185580+0000","last_fullsized":"2024-09-19T06:49:52.185580+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:53:47.384169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.188118+0000","last_change":"2024-09-19T06:49:10.388955+0000","last_active":"2024-09-19T06:49:52.188118+0000","last_peered":"2024-09-19T06:49:52.188118+0000","last_clean":"2024-09-19T06:49:52.188118+0000","last_became_active":"2024-09-19T06:49:10.387947+0000","last_became_peered":"2024-09-19T06:49:10.387947+0000","last_unstale":"2024-09-19T06:49:52.188118+0000","last_undegraded":"2024-09-19T06:49:52.188118+0000","last_fullsized":"2024-09-19T06:49:52.188118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:44.556097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"93'7","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.245539+0000","last_change":"2024-09-19T06:49:07.912897+0000","last_active":"2024-09-19T06:49:58.245539+0000","last_peered":"2024-09-19T06:49:58.245539+0000","last_clean":"2024-09-19T06:49:58.245539+0000","last_became_active":"2024-09-19T06:49:07.912520+0000","last_became_peered":"2024-09-19T06:49:07.912520+0000","last_unstale":"2024-09-19T06:49:58.245539+0000","last_undegraded":"2024-09-19T06:49:58.245539+0000","last_fullsized":"2024-09-19T06:49:58.245539+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:23:56.582808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192795+0000","last_change":"2024-09-19T06:49:20.420851+0000","last_active":"2024-09-19T06:49:52.192795+0000","last_peered":"2024-09-19T06:49:52.192795+0000","last_clean":"2024-09-19T06:49:52.192795+0000","last_became_active":"2024-09-19T06:49:20.420307+0000","last_became_peered":"2024-09-19T06:49:20.420307+0000","last_unstale":"2024-09-19T06:49:52.192795+0000","last_undegraded":"2024-09-19T06:49:52.192795+0000","last_fullsized":"2024-09-19T06:49:52.192795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:20.633070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.228471+0000","last_change":"2024-09-19T06:49:10.372935+0000","last_active":"2024-09-19T06:49:52.228471+0000","last_peered":"2024-09-19T06:49:52.228471+0000","last_clean":"2024-09-19T06:49:52.228471+0000","last_became_active":"2024-09-19T06:49:10.372641+0000","last_became_peered":"2024-09-19T06:49:10.372641+0000","last_unstale":"2024-09-19T06:49:52.228471+0000","last_undegraded":"2024-09-19T06:49:52.228471+0000","last_fullsized":"2024-09-19T06:49:52.228471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:13:07.917094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'11","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:08.379089+0000","last_change":"2024-09-19T06:49:07.908184+0000","last_active":"2024-09-19T06:50:08.379089+0000","last_peered":"2024-09-19T06:50:08.379089+0000","last_clean":"2024-09-19T06:50:08.379089+0000","last_became_active":"2024-09-19T06:49:07.907722+0000","last_became_peered":"2024-09-19T06:49:07.907722+0000","last_unstale":"2024-09-19T06:50:08.379089+0000","last_undegraded":"2024-09-19T06:50:08.379089+0000","last_fullsized":"2024-09-19T06:50:08.379089+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:24:25.957501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227438+0000","last_change":"2024-09-19T06:49:20.616966+0000","last_active":"2024-09-19T06:49:52.227438+0000","last_peered":"2024-09-19T06:49:52.227438+0000","last_clean":"2024-09-19T06:49:52.227438+0000","last_became_active":"2024-09-19T06:49:20.616402+0000","last_became_peered":"2024-09-19T06:49:20.616402+0000","last_unstale":"2024-09-19T06:49:52.227438+0000","last_undegraded":"2024-09-19T06:49:52.227438+0000","last_fullsized":"2024-09-19T06:49:52.227438+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:15:12.572214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189770+0000","last_change":"2024-09-19T06:49:10.373048+0000","last_active":"2024-09-19T06:49:52.189770+0000","last_peered":"2024-09-19T06:49:52.189770+0000","last_clean":"2024-09-19T06:49:52.189770+0000","last_became_active":"2024-09-19T06:49:10.372257+0000","last_became_peered":"2024-09-19T06:49:10.372257+0000","last_unstale":"2024-09-19T06:49:52.189770+0000","last_undegraded":"2024-09-19T06:49:52.189770+0000","last_fullsized":"2024-09-19T06:49:52.189770+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:30.707948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.243735+0000","last_change":"2024-09-19T06:49:52.195314+0000","last_active":"2024-09-19T06:49:58.243735+0000","last_peered":"2024-09-19T06:49:58.243735+0000","last_clean":"2024-09-19T06:49:58.243735+0000","last_became_active":"2024-09-19T06:49:52.194849+0000","last_became_peered":"2024-09-19T06:49:52.194849+0000","last_unstale":"2024-09-19T06:49:58.243735+0000","last_undegraded":"2024-09-19T06:49:58.243735+0000","last_fullsized":"2024-09-19T06:49:58.243735+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:01:20.902382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189734+0000","last_change":"2024-09-19T06:49:20.422658+0000","last_active":"2024-09-19T06:49:52.189734+0000","last_peered":"2024-09-19T06:49:52.189734+0000","last_clean":"2024-09-19T06:49:52.189734+0000","last_became_active":"2024-09-19T06:49:20.422190+0000","last_became_peered":"2024-09-19T06:49:20.422190+0000","last_unstale":"2024-09-19T06:49:52.189734+0000","last_undegraded":"2024-09-19T06:49:52.189734+0000","last_fullsized":"2024-09-19T06:49:52.189734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:13:14.338388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189710+0000","last_change":"2024-09-19T06:49:10.376579+0000","last_active":"2024-09-19T06:49:52.189710+0000","last_peered":"2024-09-19T06:49:52.189710+0000","last_clean":"2024-09-19T06:49:52.189710+0000","last_became_active":"2024-09-19T06:49:10.375920+0000","last_became_peered":"2024-09-19T06:49:10.375920+0000","last_unstale":"2024-09-19T06:49:52.189710+0000","last_undegraded":"2024-09-19T06:49:52.189710+0000","last_fullsized":"2024-09-19T06:49:52.189710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:47:35.976117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"93'1","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190394+0000","last_change":"2024-09-19T06:49:08.187353+0000","last_active":"2024-09-19T06:49:52.190394+0000","last_peered":"2024-09-19T06:49:52.190394+0000","last_clean":"2024-09-19T06:49:52.190394+0000","last_became_active":"2024-09-19T06:49:08.186860+0000","last_became_peered":"2024-09-19T06:49:08.186860+0000","last_unstale":"2024-09-19T06:49:52.190394+0000","last_undegraded":"2024-09-19T06:49:52.190394+0000","last_fullsized":"2024-09-19T06:49:52.190394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:45:06.236143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189149+0000","last_change":"2024-09-19T06:49:20.617028+0000","last_active":"2024-09-19T06:49:52.189149+0000","last_peered":"2024-09-19T06:49:52.189149+0000","last_clean":"2024-09-19T06:49:52.189149+0000","last_became_active":"2024-09-19T06:49:20.616443+0000","last_became_peered":"2024-09-19T06:49:20.616443+0000","last_unstale":"2024-09-19T06:49:52.189149+0000","last_undegraded":"2024-09-19T06:49:52.189149+0000","last_fullsized":"2024-09-19T06:49:52.189149+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:46:30.394051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189702+0000","last_change":"2024-09-19T06:49:10.386030+0000","last_active":"2024-09-19T06:49:52.189702+0000","last_peered":"2024-09-19T06:49:52.189702+0000","last_clean":"2024-09-19T06:49:52.189702+0000","last_became_active":"2024-09-19T06:49:10.385698+0000","last_became_peered":"2024-09-19T06:49:10.385698+0000","last_unstale":"2024-09-19T06:49:52.189702+0000","last_undegraded":"2024-09-19T06:49:52.189702+0000","last_fullsized":"2024-09-19T06:49:52.189702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:56:12.688507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'5","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:12.271306+0000","last_change":"2024-09-19T06:49:07.910172+0000","last_active":"2024-09-19T06:50:12.271306+0000","last_peered":"2024-09-19T06:50:12.271306+0000","last_clean":"2024-09-19T06:50:12.271306+0000","last_became_active":"2024-09-19T06:49:07.909798+0000","last_became_peered":"2024-09-19T06:49:07.909798+0000","last_unstale":"2024-09-19T06:50:12.271306+0000","last_undegraded":"2024-09-19T06:50:12.271306+0000","last_fullsized":"2024-09-19T06:50:12.271306+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.592988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189691+0000","last_change":"2024-09-19T06:49:20.424233+0000","last_active":"2024-09-19T06:49:52.189691+0000","last_peered":"2024-09-19T06:49:52.189691+0000","last_clean":"2024-09-19T06:49:52.189691+0000","last_became_active":"2024-09-19T06:49:20.423797+0000","last_became_peered":"2024-09-19T06:49:20.423797+0000","last_unstale":"2024-09-19T06:49:52.189691+0000","last_undegraded":"2024-09-19T06:49:52.189691+0000","last_fullsized":"2024-09-19T06:49:52.189691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:58:54.983250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184233+0000","last_change":"2024-09-19T06:49:10.376408+0000","last_active":"2024-09-19T06:49:52.184233+0000","last_peered":"2024-09-19T06:49:52.184233+0000","last_clean":"2024-09-19T06:49:52.184233+0000","last_became_active":"2024-09-19T06:49:10.376065+0000","last_became_peered":"2024-09-19T06:49:10.376065+0000","last_unstale":"2024-09-19T06:49:52.184233+0000","last_undegraded":"2024-09-19T06:49:52.184233+0000","last_fullsized":"2024-09-19T06:49:52.184233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:28:06.175670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:13.272071+0000","last_change":"2024-09-19T06:49:08.185183+0000","last_active":"2024-09-19T06:50:13.272071+0000","last_peered":"2024-09-19T06:50:13.272071+0000","last_clean":"2024-09-19T06:50:13.272071+0000","last_became_active":"2024-09-19T06:49:08.184738+0000","last_became_peered":"2024-09-19T06:49:08.184738+0000","last_unstale":"2024-09-19T06:50:13.272071+0000","last_undegraded":"2024-09-19T06:50:13.272071+0000","last_fullsized":"2024-09-19T06:50:13.272071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:19:54.966966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191461+0000","last_change":"2024-09-19T06:49:20.617986+0000","last_active":"2024-09-19T06:49:52.191461+0000","last_peered":"2024-09-19T06:49:52.191461+0000","last_clean":"2024-09-19T06:49:52.191461+0000","last_became_active":"2024-09-19T06:49:20.617325+0000","last_became_peered":"2024-09-19T06:49:20.617325+0000","last_unstale":"2024-09-19T06:49:52.191461+0000","last_undegraded":"2024-09-19T06:49:52.191461+0000","last_fullsized":"2024-09-19T06:49:52.191461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:03.825005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189212+0000","last_change":"2024-09-19T06:49:10.385230+0000","last_active":"2024-09-19T06:49:52.189212+0000","last_peered":"2024-09-19T06:49:52.189212+0000","last_clean":"2024-09-19T06:49:52.189212+0000","last_became_active":"2024-09-19T06:49:10.384633+0000","last_became_peered":"2024-09-19T06:49:10.384633+0000","last_unstale":"2024-09-19T06:49:52.189212+0000","last_undegraded":"2024-09-19T06:49:52.189212+0000","last_fullsized":"2024-09-19T06:49:52.189212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:16:27.751338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:56.224385+0000","last_change":"2024-09-19T06:49:07.908823+0000","last_active":"2024-09-19T06:49:56.224385+0000","last_peered":"2024-09-19T06:49:56.224385+0000","last_clean":"2024-09-19T06:49:56.224385+0000","last_became_active":"2024-09-19T06:49:07.908340+0000","last_became_peered":"2024-09-19T06:49:07.908340+0000","last_unstale":"2024-09-19T06:49:56.224385+0000","last_undegraded":"2024-09-19T06:49:56.224385+0000","last_fullsized":"2024-09-19T06:49:56.224385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:25.198073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184542+0000","last_change":"2024-09-19T06:49:20.617412+0000","last_active":"2024-09-19T06:49:52.184542+0000","last_peered":"2024-09-19T06:49:52.184542+0000","last_clean":"2024-09-19T06:49:52.184542+0000","last_became_active":"2024-09-19T06:49:20.616908+0000","last_became_peered":"2024-09-19T06:49:20.616908+0000","last_unstale":"2024-09-19T06:49:52.184542+0000","last_undegraded":"2024-09-19T06:49:52.184542+0000","last_fullsized":"2024-09-19T06:49:52.184542+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:44:12.121754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227902+0000","last_change":"2024-09-19T06:49:10.374761+0000","last_active":"2024-09-19T06:49:52.227902+0000","last_peered":"2024-09-19T06:49:52.227902+0000","last_clean":"2024-09-19T06:49:52.227902+0000","last_became_active":"2024-09-19T06:49:10.374258+0000","last_became_peered":"2024-09-19T06:49:10.374258+0000","last_unstale":"2024-09-19T06:49:52.227902+0000","last_undegraded":"2024-09-19T06:49:52.227902+0000","last_fullsized":"2024-09-19T06:49:52.227902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:23:23.850658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'6","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:59.248949+0000","last_change":"2024-09-19T06:49:07.912648+0000","last_active":"2024-09-19T06:49:59.248949+0000","last_peered":"2024-09-19T06:49:59.248949+0000","last_clean":"2024-09-19T06:49:59.248949+0000","last_became_active":"2024-09-19T06:49:07.912289+0000","last_became_peered":"2024-09-19T06:49:07.912289+0000","last_unstale":"2024-09-19T06:49:59.248949+0000","last_undegraded":"2024-09-19T06:49:59.248949+0000","last_fullsized":"2024-09-19T06:49:59.248949+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:36.211030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187276+0000","last_change":"2024-09-19T06:47:53.158465+0000","last_active":"2024-09-19T06:49:52.187276+0000","last_peered":"2024-09-19T06:49:52.187276+0000","last_clean":"2024-09-19T06:49:52.187276+0000","last_became_active":"2024-09-19T06:47:53.158059+0000","last_became_peered":"2024-09-19T06:47:53.158059+0000","last_unstale":"2024-09-19T06:49:52.187276+0000","last_undegraded":"2024-09-19T06:49:52.187276+0000","last_fullsized":"2024-09-19T06:49:52.187276+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:44:13.613197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:44:13.613197+0000","last_clean_scrub_stamp":"2024-09-19T06:44:13.613197+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:16:41.310229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187421+0000","last_change":"2024-09-19T06:49:20.422112+0000","last_active":"2024-09-19T06:49:52.187421+0000","last_peered":"2024-09-19T06:49:52.187421+0000","last_clean":"2024-09-19T06:49:52.187421+0000","last_became_active":"2024-09-19T06:49:20.421720+0000","last_became_peered":"2024-09-19T06:49:20.421720+0000","last_unstale":"2024-09-19T06:49:52.187421+0000","last_undegraded":"2024-09-19T06:49:52.187421+0000","last_fullsized":"2024-09-19T06:49:52.187421+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:35:25.923946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187492+0000","last_change":"2024-09-19T06:49:10.379414+0000","last_active":"2024-09-19T06:49:52.187492+0000","last_peered":"2024-09-19T06:49:52.187492+0000","last_clean":"2024-09-19T06:49:52.187492+0000","last_became_active":"2024-09-19T06:49:10.379075+0000","last_became_peered":"2024-09-19T06:49:10.379075+0000","last_unstale":"2024-09-19T06:49:52.187492+0000","last_undegraded":"2024-09-19T06:49:52.187492+0000","last_fullsized":"2024-09-19T06:49:52.187492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:32:57.976929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'5","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:12.270106+0000","last_change":"2024-09-19T06:49:08.186883+0000","last_active":"2024-09-19T06:50:12.270106+0000","last_peered":"2024-09-19T06:50:12.270106+0000","last_clean":"2024-09-19T06:50:12.270106+0000","last_became_active":"2024-09-19T06:49:08.186472+0000","last_became_peered":"2024-09-19T06:49:08.186472+0000","last_unstale":"2024-09-19T06:50:12.270106+0000","last_undegraded":"2024-09-19T06:50:12.270106+0000","last_fullsized":"2024-09-19T06:50:12.270106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:58:12.146032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410556+0000","last_change":"2024-09-19T06:49:20.418542+0000","last_active":"2024-09-19T06:49:52.410556+0000","last_peered":"2024-09-19T06:49:52.410556+0000","last_clean":"2024-09-19T06:49:52.410556+0000","last_became_active":"2024-09-19T06:49:20.418044+0000","last_became_peered":"2024-09-19T06:49:20.418044+0000","last_unstale":"2024-09-19T06:49:52.410556+0000","last_undegraded":"2024-09-19T06:49:52.410556+0000","last_fullsized":"2024-09-19T06:49:52.410556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:44:02.695919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187189+0000","last_change":"2024-09-19T06:49:10.373862+0000","last_active":"2024-09-19T06:49:52.187189+0000","last_peered":"2024-09-19T06:49:52.187189+0000","last_clean":"2024-09-19T06:49:52.187189+0000","last_became_active":"2024-09-19T06:49:10.373345+0000","last_became_peered":"2024-09-19T06:49:10.373345+0000","last_unstale":"2024-09-19T06:49:52.187189+0000","last_undegraded":"2024-09-19T06:49:52.187189+0000","last_fullsized":"2024-09-19T06:49:52.187189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:33.562984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'3","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.243953+0000","last_change":"2024-09-19T06:49:07.909407+0000","last_active":"2024-09-19T06:49:58.243953+0000","last_peered":"2024-09-19T06:49:58.243953+0000","last_clean":"2024-09-19T06:49:58.243953+0000","last_became_active":"2024-09-19T06:49:07.909101+0000","last_became_peered":"2024-09-19T06:49:07.909101+0000","last_unstale":"2024-09-19T06:49:58.243953+0000","last_undegraded":"2024-09-19T06:49:58.243953+0000","last_fullsized":"2024-09-19T06:49:58.243953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:08:47.317018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187005+0000","last_change":"2024-09-19T06:49:20.425109+0000","last_active":"2024-09-19T06:49:52.187005+0000","last_peered":"2024-09-19T06:49:52.187005+0000","last_clean":"2024-09-19T06:49:52.187005+0000","last_became_active":"2024-09-19T06:49:20.424701+0000","last_became_peered":"2024-09-19T06:49:20.424701+0000","last_unstale":"2024-09-19T06:49:52.187005+0000","last_undegraded":"2024-09-19T06:49:52.187005+0000","last_fullsized":"2024-09-19T06:49:52.187005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:13:10.878691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187467+0000","last_change":"2024-09-19T06:49:10.374089+0000","last_active":"2024-09-19T06:49:52.187467+0000","last_peered":"2024-09-19T06:49:52.187467+0000","last_clean":"2024-09-19T06:49:52.187467+0000","last_became_active":"2024-09-19T06:49:10.373729+0000","last_became_peered":"2024-09-19T06:49:10.373729+0000","last_unstale":"2024-09-19T06:49:52.187467+0000","last_undegraded":"2024-09-19T06:49:52.187467+0000","last_fullsized":"2024-09-19T06:49:52.187467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:52.876874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"93'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.244235+0000","last_change":"2024-09-19T06:49:07.909616+0000","last_active":"2024-09-19T06:49:58.244235+0000","last_peered":"2024-09-19T06:49:58.244235+0000","last_clean":"2024-09-19T06:49:58.244235+0000","last_became_active":"2024-09-19T06:49:07.909172+0000","last_became_peered":"2024-09-19T06:49:07.909172+0000","last_unstale":"2024-09-19T06:49:58.244235+0000","last_undegraded":"2024-09-19T06:49:58.244235+0000","last_fullsized":"2024-09-19T06:49:58.244235+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:57:27.685639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.183999+0000","last_change":"2024-09-19T06:49:20.419188+0000","last_active":"2024-09-19T06:49:52.183999+0000","last_peered":"2024-09-19T06:49:52.183999+0000","last_clean":"2024-09-19T06:49:52.183999+0000","last_became_active":"2024-09-19T06:49:20.418641+0000","last_became_peered":"2024-09-19T06:49:20.418641+0000","last_unstale":"2024-09-19T06:49:52.183999+0000","last_undegraded":"2024-09-19T06:49:52.183999+0000","last_fullsized":"2024-09-19T06:49:52.183999+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:06:05.278027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189695+0000","last_change":"2024-09-19T06:49:10.378491+0000","last_active":"2024-09-19T06:49:52.189695+0000","last_peered":"2024-09-19T06:49:52.189695+0000","last_clean":"2024-09-19T06:49:52.189695+0000","last_became_active":"2024-09-19T06:49:10.378141+0000","last_became_peered":"2024-09-19T06:49:10.378141+0000","last_unstale":"2024-09-19T06:49:52.189695+0000","last_undegraded":"2024-09-19T06:49:52.189695+0000","last_fullsized":"2024-09-19T06:49:52.189695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:57:39.195604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.221046+0000","last_change":"2024-09-19T06:49:07.909836+0000","last_active":"2024-09-19T06:49:58.221046+0000","last_peered":"2024-09-19T06:49:58.221046+0000","last_clean":"2024-09-19T06:49:58.221046+0000","last_became_active":"2024-09-19T06:49:07.909527+0000","last_became_peered":"2024-09-19T06:49:07.909527+0000","last_unstale":"2024-09-19T06:49:58.221046+0000","last_undegraded":"2024-09-19T06:49:58.221046+0000","last_fullsized":"2024-09-19T06:49:58.221046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:51.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184155+0000","last_change":"2024-09-19T06:49:20.426184+0000","last_active":"2024-09-19T06:49:52.184155+0000","last_peered":"2024-09-19T06:49:52.184155+0000","last_clean":"2024-09-19T06:49:52.184155+0000","last_became_active":"2024-09-19T06:49:20.425844+0000","last_became_peered":"2024-09-19T06:49:20.425844+0000","last_unstale":"2024-09-19T06:49:52.184155+0000","last_undegraded":"2024-09-19T06:49:52.184155+0000","last_fullsized":"2024-09-19T06:49:52.184155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.120462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184354+0000","last_change":"2024-09-19T06:49:10.380589+0000","last_active":"2024-09-19T06:49:52.184354+0000","last_peered":"2024-09-19T06:49:52.184354+0000","last_clean":"2024-09-19T06:49:52.184354+0000","last_became_active":"2024-09-19T06:49:10.380263+0000","last_became_peered":"2024-09-19T06:49:10.380263+0000","last_unstale":"2024-09-19T06:49:52.184354+0000","last_undegraded":"2024-09-19T06:49:52.184354+0000","last_fullsized":"2024-09-19T06:49:52.184354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:30:48.426240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"93'17","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.245456+0000","last_change":"2024-09-19T06:49:52.199971+0000","last_active":"2024-09-19T06:49:58.245456+0000","last_peered":"2024-09-19T06:49:58.245456+0000","last_clean":"2024-09-19T06:49:58.245456+0000","last_became_active":"2024-09-19T06:49:52.191887+0000","last_became_peered":"2024-09-19T06:49:52.191887+0000","last_unstale":"2024-09-19T06:49:58.245456+0000","last_undegraded":"2024-09-19T06:49:58.245456+0000","last_fullsized":"2024-09-19T06:49:58.245456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:22:40.775893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191728+0000","last_change":"2024-09-19T06:49:20.435213+0000","last_active":"2024-09-19T06:49:52.191728+0000","last_peered":"2024-09-19T06:49:52.191728+0000","last_clean":"2024-09-19T06:49:52.191728+0000","last_became_active":"2024-09-19T06:49:20.434776+0000","last_became_peered":"2024-09-19T06:49:20.434776+0000","last_unstale":"2024-09-19T06:49:52.191728+0000","last_undegraded":"2024-09-19T06:49:52.191728+0000","last_fullsized":"2024-09-19T06:49:52.191728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:37.067947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.186939+0000","last_change":"2024-09-19T06:49:10.375394+0000","last_active":"2024-09-19T06:49:52.186939+0000","last_peered":"2024-09-19T06:49:52.186939+0000","last_clean":"2024-09-19T06:49:52.186939+0000","last_became_active":"2024-09-19T06:49:10.375056+0000","last_became_peered":"2024-09-19T06:49:10.375056+0000","last_unstale":"2024-09-19T06:49:52.186939+0000","last_undegraded":"2024-09-19T06:49:52.186939+0000","last_fullsized":"2024-09-19T06:49:52.186939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:18:40.158481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'6","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:12.271341+0000","last_change":"2024-09-19T06:49:07.910215+0000","last_active":"2024-09-19T06:50:12.271341+0000","last_peered":"2024-09-19T06:50:12.271341+0000","last_clean":"2024-09-19T06:50:12.271341+0000","last_became_active":"2024-09-19T06:49:07.909908+0000","last_became_peered":"2024-09-19T06:49:07.909908+0000","last_unstale":"2024-09-19T06:50:12.271341+0000","last_undegraded":"2024-09-19T06:50:12.271341+0000","last_fullsized":"2024-09-19T06:50:12.271341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:45:05.432960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192261+0000","last_change":"2024-09-19T06:49:20.434400+0000","last_active":"2024-09-19T06:49:52.192261+0000","last_peered":"2024-09-19T06:49:52.192261+0000","last_clean":"2024-09-19T06:49:52.192261+0000","last_became_active":"2024-09-19T06:49:20.433877+0000","last_became_peered":"2024-09-19T06:49:20.433877+0000","last_unstale":"2024-09-19T06:49:52.192261+0000","last_undegraded":"2024-09-19T06:49:52.192261+0000","last_fullsized":"2024-09-19T06:49:52.192261+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:24:16.378509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187261+0000","last_change":"2024-09-19T06:49:10.378048+0000","last_active":"2024-09-19T06:49:52.187261+0000","last_peered":"2024-09-19T06:49:52.187261+0000","last_clean":"2024-09-19T06:49:52.187261+0000","last_became_active":"2024-09-19T06:49:10.377710+0000","last_became_peered":"2024-09-19T06:49:10.377710+0000","last_unstale":"2024-09-19T06:49:52.187261+0000","last_undegraded":"2024-09-19T06:49:52.187261+0000","last_fullsized":"2024-09-19T06:49:52.187261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:41:54.620218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"93'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.244459+0000","last_change":"2024-09-19T06:49:07.907169+0000","last_active":"2024-09-19T06:49:58.244459+0000","last_peered":"2024-09-19T06:49:58.244459+0000","last_clean":"2024-09-19T06:49:58.244459+0000","last_became_active":"2024-09-19T06:49:07.906811+0000","last_became_peered":"2024-09-19T06:49:07.906811+0000","last_unstale":"2024-09-19T06:49:58.244459+0000","last_undegraded":"2024-09-19T06:49:58.244459+0000","last_fullsized":"2024-09-19T06:49:58.244459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:58:19.184424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189874+0000","last_change":"2024-09-19T06:49:20.429645+0000","last_active":"2024-09-19T06:49:52.189874+0000","last_peered":"2024-09-19T06:49:52.189874+0000","last_clean":"2024-09-19T06:49:52.189874+0000","last_became_active":"2024-09-19T06:49:20.429198+0000","last_became_peered":"2024-09-19T06:49:20.429198+0000","last_unstale":"2024-09-19T06:49:52.189874+0000","last_undegraded":"2024-09-19T06:49:52.189874+0000","last_fullsized":"2024-09-19T06:49:52.189874+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:36:42.145928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190178+0000","last_change":"2024-09-19T06:49:10.382223+0000","last_active":"2024-09-19T06:49:52.190178+0000","last_peered":"2024-09-19T06:49:52.190178+0000","last_clean":"2024-09-19T06:49:52.190178+0000","last_became_active":"2024-09-19T06:49:10.381869+0000","last_became_peered":"2024-09-19T06:49:10.381869+0000","last_unstale":"2024-09-19T06:49:52.190178+0000","last_undegraded":"2024-09-19T06:49:52.190178+0000","last_fullsized":"2024-09-19T06:49:52.190178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:23:34.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"93'9","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:59.248344+0000","last_change":"2024-09-19T06:49:07.907812+0000","last_active":"2024-09-19T06:49:59.248344+0000","last_peered":"2024-09-19T06:49:59.248344+0000","last_clean":"2024-09-19T06:49:59.248344+0000","last_became_active":"2024-09-19T06:49:07.907524+0000","last_became_peered":"2024-09-19T06:49:07.907524+0000","last_unstale":"2024-09-19T06:49:59.248344+0000","last_undegraded":"2024-09-19T06:49:59.248344+0000","last_fullsized":"2024-09-19T06:49:59.248344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:26:43.780682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184608+0000","last_change":"2024-09-19T06:49:20.436311+0000","last_active":"2024-09-19T06:49:52.184608+0000","last_peered":"2024-09-19T06:49:52.184608+0000","last_clean":"2024-09-19T06:49:52.184608+0000","last_became_active":"2024-09-19T06:49:20.435936+0000","last_became_peered":"2024-09-19T06:49:20.435936+0000","last_unstale":"2024-09-19T06:49:52.184608+0000","last_undegraded":"2024-09-19T06:49:52.184608+0000","last_fullsized":"2024-09-19T06:49:52.184608+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:54:04.688922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191610+0000","last_change":"2024-09-19T06:49:10.373764+0000","last_active":"2024-09-19T06:49:52.191610+0000","last_peered":"2024-09-19T06:49:52.191610+0000","last_clean":"2024-09-19T06:49:52.191610+0000","last_became_active":"2024-09-19T06:49:10.373324+0000","last_became_peered":"2024-09-19T06:49:10.373324+0000","last_unstale":"2024-09-19T06:49:52.191610+0000","last_undegraded":"2024-09-19T06:49:52.191610+0000","last_fullsized":"2024-09-19T06:49:52.191610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:25:03.898831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'28","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:09.267017+0000","last_change":"2024-09-19T06:49:07.907319+0000","last_active":"2024-09-19T06:50:09.267017+0000","last_peered":"2024-09-19T06:50:09.267017+0000","last_clean":"2024-09-19T06:50:09.267017+0000","last_became_active":"2024-09-19T06:49:07.906888+0000","last_became_peered":"2024-09-19T06:49:07.906888+0000","last_unstale":"2024-09-19T06:50:09.267017+0000","last_undegraded":"2024-09-19T06:50:09.267017+0000","last_fullsized":"2024-09-19T06:50:09.267017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:49.051957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44483,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":57,"num_write":30,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191821+0000","last_change":"2024-09-19T06:49:20.428184+0000","last_active":"2024-09-19T06:49:52.191821+0000","last_peered":"2024-09-19T06:49:52.191821+0000","last_clean":"2024-09-19T06:49:52.191821+0000","last_became_active":"2024-09-19T06:49:20.427631+0000","last_became_peered":"2024-09-19T06:49:20.427631+0000","last_unstale":"2024-09-19T06:49:52.191821+0000","last_undegraded":"2024-09-19T06:49:52.191821+0000","last_fullsized":"2024-09-19T06:49:52.191821+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:14:51.063413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410607+0000","last_change":"2024-09-19T06:49:10.373073+0000","last_active":"2024-09-19T06:49:52.410607+0000","last_peered":"2024-09-19T06:49:52.410607+0000","last_clean":"2024-09-19T06:49:52.410607+0000","last_became_active":"2024-09-19T06:49:10.371611+0000","last_became_peered":"2024-09-19T06:49:10.371611+0000","last_unstale":"2024-09-19T06:49:52.410607+0000","last_undegraded":"2024-09-19T06:49:52.410607+0000","last_fullsized":"2024-09-19T06:49:52.410607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:58.228946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"93'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:57.236475+0000","last_change":"2024-09-19T06:49:08.186128+0000","last_active":"2024-09-19T06:49:57.236475+0000","last_peered":"2024-09-19T06:49:57.236475+0000","last_clean":"2024-09-19T06:49:57.236475+0000","last_became_active":"2024-09-19T06:49:08.185817+0000","last_became_peered":"2024-09-19T06:49:08.185817+0000","last_unstale":"2024-09-19T06:49:57.236475+0000","last_undegraded":"2024-09-19T06:49:57.236475+0000","last_fullsized":"2024-09-19T06:49:57.236475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:59:53.066405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185090+0000","last_change":"2024-09-19T06:49:20.431251+0000","last_active":"2024-09-19T06:49:52.185090+0000","last_peered":"2024-09-19T06:49:52.185090+0000","last_clean":"2024-09-19T06:49:52.185090+0000","last_became_active":"2024-09-19T06:49:20.430709+0000","last_became_peered":"2024-09-19T06:49:20.430709+0000","last_unstale":"2024-09-19T06:49:52.185090+0000","last_undegraded":"2024-09-19T06:49:52.185090+0000","last_fullsized":"2024-09-19T06:49:52.185090+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:55:06.742567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.188717+0000","last_change":"2024-09-19T06:49:10.379045+0000","last_active":"2024-09-19T06:49:52.188717+0000","last_peered":"2024-09-19T06:49:52.188717+0000","last_clean":"2024-09-19T06:49:52.188717+0000","last_became_active":"2024-09-19T06:49:10.378481+0000","last_became_peered":"2024-09-19T06:49:10.378481+0000","last_unstale":"2024-09-19T06:49:52.188717+0000","last_undegraded":"2024-09-19T06:49:52.188717+0000","last_fullsized":"2024-09-19T06:49:52.188717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:29:55.770670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"93'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.244718+0000","last_change":"2024-09-19T06:49:07.910246+0000","last_active":"2024-09-19T06:49:58.244718+0000","last_peered":"2024-09-19T06:49:58.244718+0000","last_clean":"2024-09-19T06:49:58.244718+0000","last_became_active":"2024-09-19T06:49:07.909855+0000","last_became_peered":"2024-09-19T06:49:07.909855+0000","last_unstale":"2024-09-19T06:49:58.244718+0000","last_undegraded":"2024-09-19T06:49:58.244718+0000","last_fullsized":"2024-09-19T06:49:58.244718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:33:24.011418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189372+0000","last_change":"2024-09-19T06:49:20.421847+0000","last_active":"2024-09-19T06:49:52.189372+0000","last_peered":"2024-09-19T06:49:52.189372+0000","last_clean":"2024-09-19T06:49:52.189372+0000","last_became_active":"2024-09-19T06:49:20.421391+0000","last_became_peered":"2024-09-19T06:49:20.421391+0000","last_unstale":"2024-09-19T06:49:52.189372+0000","last_undegraded":"2024-09-19T06:49:52.189372+0000","last_fullsized":"2024-09-19T06:49:52.189372+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:54:24.207872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410917+0000","last_change":"2024-09-19T06:49:10.376029+0000","last_active":"2024-09-19T06:49:52.410917+0000","last_peered":"2024-09-19T06:49:52.410917+0000","last_clean":"2024-09-19T06:49:52.410917+0000","last_became_active":"2024-09-19T06:49:10.374242+0000","last_became_peered":"2024-09-19T06:49:10.374242+0000","last_unstale":"2024-09-19T06:49:52.410917+0000","last_undegraded":"2024-09-19T06:49:52.410917+0000","last_fullsized":"2024-09-19T06:49:52.410917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:59:43.722173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:09.264418+0000","last_change":"2024-09-19T06:49:07.908090+0000","last_active":"2024-09-19T06:50:09.264418+0000","last_peered":"2024-09-19T06:50:09.264418+0000","last_clean":"2024-09-19T06:50:09.264418+0000","last_became_active":"2024-09-19T06:49:07.906933+0000","last_became_peered":"2024-09-19T06:49:07.906933+0000","last_unstale":"2024-09-19T06:50:09.264418+0000","last_undegraded":"2024-09-19T06:50:09.264418+0000","last_fullsized":"2024-09-19T06:50:09.264418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:21.711371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227514+0000","last_change":"2024-09-19T06:49:20.433797+0000","last_active":"2024-09-19T06:49:52.227514+0000","last_peered":"2024-09-19T06:49:52.227514+0000","last_clean":"2024-09-19T06:49:52.227514+0000","last_became_active":"2024-09-19T06:49:20.433419+0000","last_became_peered":"2024-09-19T06:49:20.433419+0000","last_unstale":"2024-09-19T06:49:52.227514+0000","last_undegraded":"2024-09-19T06:49:52.227514+0000","last_fullsized":"2024-09-19T06:49:52.227514+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:18:39.691678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187738+0000","last_change":"2024-09-19T06:49:20.424995+0000","last_active":"2024-09-19T06:49:52.187738+0000","last_peered":"2024-09-19T06:49:52.187738+0000","last_clean":"2024-09-19T06:49:52.187738+0000","last_became_active":"2024-09-19T06:49:20.424589+0000","last_became_peered":"2024-09-19T06:49:20.424589+0000","last_unstale":"2024-09-19T06:49:52.187738+0000","last_undegraded":"2024-09-19T06:49:52.187738+0000","last_fullsized":"2024-09-19T06:49:52.187738+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:17.968571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191637+0000","last_change":"2024-09-19T06:49:10.382157+0000","last_active":"2024-09-19T06:49:52.191637+0000","last_peered":"2024-09-19T06:49:52.191637+0000","last_clean":"2024-09-19T06:49:52.191637+0000","last_became_active":"2024-09-19T06:49:10.381848+0000","last_became_peered":"2024-09-19T06:49:10.381848+0000","last_unstale":"2024-09-19T06:49:52.191637+0000","last_undegraded":"2024-09-19T06:49:52.191637+0000","last_fullsized":"2024-09-19T06:49:52.191637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:10.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"93'3","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.244718+0000","last_change":"2024-09-19T06:49:07.906088+0000","last_active":"2024-09-19T06:49:58.244718+0000","last_peered":"2024-09-19T06:49:58.244718+0000","last_clean":"2024-09-19T06:49:58.244718+0000","last_became_active":"2024-09-19T06:49:07.905798+0000","last_became_peered":"2024-09-19T06:49:07.905798+0000","last_unstale":"2024-09-19T06:49:58.244718+0000","last_undegraded":"2024-09-19T06:49:58.244718+0000","last_fullsized":"2024-09-19T06:49:58.244718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:02.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189691+0000","last_change":"2024-09-19T06:49:20.432275+0000","last_active":"2024-09-19T06:49:52.189691+0000","last_peered":"2024-09-19T06:49:52.189691+0000","last_clean":"2024-09-19T06:49:52.189691+0000","last_became_active":"2024-09-19T06:49:20.431951+0000","last_became_peered":"2024-09-19T06:49:20.431951+0000","last_unstale":"2024-09-19T06:49:52.189691+0000","last_undegraded":"2024-09-19T06:49:52.189691+0000","last_fullsized":"2024-09-19T06:49:52.189691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:38.649557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227622+0000","last_change":"2024-09-19T06:49:10.368061+0000","last_active":"2024-09-19T06:49:52.227622+0000","last_peered":"2024-09-19T06:49:52.227622+0000","last_clean":"2024-09-19T06:49:52.227622+0000","last_became_active":"2024-09-19T06:49:10.367732+0000","last_became_peered":"2024-09-19T06:49:10.367732+0000","last_unstale":"2024-09-19T06:49:52.227622+0000","last_undegraded":"2024-09-19T06:49:52.227622+0000","last_fullsized":"2024-09-19T06:49:52.227622+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:37.623628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"93'3","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:56.226572+0000","last_change":"2024-09-19T06:49:07.910722+0000","last_active":"2024-09-19T06:49:56.226572+0000","last_peered":"2024-09-19T06:49:56.226572+0000","last_clean":"2024-09-19T06:49:56.226572+0000","last_became_active":"2024-09-19T06:49:07.909960+0000","last_became_peered":"2024-09-19T06:49:07.909960+0000","last_unstale":"2024-09-19T06:49:56.226572+0000","last_undegraded":"2024-09-19T06:49:56.226572+0000","last_fullsized":"2024-09-19T06:49:56.226572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:40:18.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"93'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:57.236663+0000","last_change":"2024-09-19T06:49:52.411869+0000","last_active":"2024-09-19T06:49:57.236663+0000","last_peered":"2024-09-19T06:49:57.236663+0000","last_clean":"2024-09-19T06:49:57.236663+0000","last_became_active":"2024-09-19T06:49:52.411491+0000","last_became_peered":"2024-09-19T06:49:52.411491+0000","last_unstale":"2024-09-19T06:49:57.236663+0000","last_undegraded":"2024-09-19T06:49:57.236663+0000","last_fullsized":"2024-09-19T06:49:57.236663+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:05:21.222348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184771+0000","last_change":"2024-09-19T06:49:10.388831+0000","last_active":"2024-09-19T06:49:52.184771+0000","last_peered":"2024-09-19T06:49:52.184771+0000","last_clean":"2024-09-19T06:49:52.184771+0000","last_became_active":"2024-09-19T06:49:10.388556+0000","last_became_peered":"2024-09-19T06:49:10.388556+0000","last_unstale":"2024-09-19T06:49:52.184771+0000","last_undegraded":"2024-09-19T06:49:52.184771+0000","last_fullsized":"2024-09-19T06:49:52.184771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:28:20.319429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190980+0000","last_change":"2024-09-19T06:49:20.424827+0000","last_active":"2024-09-19T06:49:52.190980+0000","last_peered":"2024-09-19T06:49:52.190980+0000","last_clean":"2024-09-19T06:49:52.190980+0000","last_became_active":"2024-09-19T06:49:20.424318+0000","last_became_peered":"2024-09-19T06:49:20.424318+0000","last_unstale":"2024-09-19T06:49:52.190980+0000","last_undegraded":"2024-09-19T06:49:52.190980+0000","last_fullsized":"2024-09-19T06:49:52.190980+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:37.814108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:13.271686+0000","last_change":"2024-09-19T06:49:52.188404+0000","last_active":"2024-09-19T06:50:13.271686+0000","last_peered":"2024-09-19T06:50:13.271686+0000","last_clean":"2024-09-19T06:50:13.271686+0000","last_became_active":"2024-09-19T06:49:52.187918+0000","last_became_peered":"2024-09-19T06:49:52.187918+0000","last_unstale":"2024-09-19T06:50:13.271686+0000","last_undegraded":"2024-09-19T06:50:13.271686+0000","last_fullsized":"2024-09-19T06:50:13.271686+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:53:38.209357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190447+0000","last_change":"2024-09-19T06:49:10.374333+0000","last_active":"2024-09-19T06:49:52.190447+0000","last_peered":"2024-09-19T06:49:52.190447+0000","last_clean":"2024-09-19T06:49:52.190447+0000","last_became_active":"2024-09-19T06:49:10.373944+0000","last_became_peered":"2024-09-19T06:49:10.373944+0000","last_unstale":"2024-09-19T06:49:52.190447+0000","last_undegraded":"2024-09-19T06:49:52.190447+0000","last_fullsized":"2024-09-19T06:49:52.190447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:01:43.609310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190950+0000","last_change":"2024-09-19T06:49:20.417030+0000","last_active":"2024-09-19T06:49:52.190950+0000","last_peered":"2024-09-19T06:49:52.190950+0000","last_clean":"2024-09-19T06:49:52.190950+0000","last_became_active":"2024-09-19T06:49:20.416620+0000","last_became_peered":"2024-09-19T06:49:20.416620+0000","last_unstale":"2024-09-19T06:49:52.190950+0000","last_undegraded":"2024-09-19T06:49:52.190950+0000","last_fullsized":"2024-09-19T06:49:52.190950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:47:18.177613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189469+0000","last_change":"2024-09-19T06:49:10.372503+0000","last_active":"2024-09-19T06:49:52.189469+0000","last_peered":"2024-09-19T06:49:52.189469+0000","last_clean":"2024-09-19T06:49:52.189469+0000","last_became_active":"2024-09-19T06:49:10.372201+0000","last_became_peered":"2024-09-19T06:49:10.372201+0000","last_unstale":"2024-09-19T06:49:52.189469+0000","last_undegraded":"2024-09-19T06:49:52.189469+0000","last_fullsized":"2024-09-19T06:49:52.189469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:31:44.270457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192797+0000","last_change":"2024-09-19T06:49:10.374149+0000","last_active":"2024-09-19T06:49:52.192797+0000","last_peered":"2024-09-19T06:49:52.192797+0000","last_clean":"2024-09-19T06:49:52.192797+0000","last_became_active":"2024-09-19T06:49:10.372570+0000","last_became_peered":"2024-09-19T06:49:10.372570+0000","last_unstale":"2024-09-19T06:49:52.192797+0000","last_undegraded":"2024-09-19T06:49:52.192797+0000","last_fullsized":"2024-09-19T06:49:52.192797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:55:56.086106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190791+0000","last_change":"2024-09-19T06:49:10.373383+0000","last_active":"2024-09-19T06:49:52.190791+0000","last_peered":"2024-09-19T06:49:52.190791+0000","last_clean":"2024-09-19T06:49:52.190791+0000","last_became_active":"2024-09-19T06:49:10.372144+0000","last_became_peered":"2024-09-19T06:49:10.372144+0000","last_unstale":"2024-09-19T06:49:52.190791+0000","last_undegraded":"2024-09-19T06:49:52.190791+0000","last_fullsized":"2024-09-19T06:49:52.190791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:08:34.000815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187118+0000","last_change":"2024-09-19T06:49:10.379720+0000","last_active":"2024-09-19T06:49:52.187118+0000","last_peered":"2024-09-19T06:49:52.187118+0000","last_clean":"2024-09-19T06:49:52.187118+0000","last_became_active":"2024-09-19T06:49:10.379325+0000","last_became_peered":"2024-09-19T06:49:10.379325+0000","last_unstale":"2024-09-19T06:49:52.187118+0000","last_undegraded":"2024-09-19T06:49:52.187118+0000","last_fullsized":"2024-09-19T06:49:52.187118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:17:58.471617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.194997+0000","last_change":"2024-09-19T06:49:52.194997+0000","last_active":"2024-09-19T06:49:52.194997+0000","last_peered":"2024-09-19T06:49:52.194997+0000","last_clean":"2024-09-19T06:49:52.194997+0000","last_became_active":"2024-09-19T06:49:52.194662+0000","last_became_peered":"2024-09-19T06:49:52.194662+0000","last_unstale":"2024-09-19T06:49:52.194997+0000","last_undegraded":"2024-09-19T06:49:52.194997+0000","last_fullsized":"2024-09-19T06:49:52.194997+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:18.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190025+0000","last_change":"2024-09-19T06:49:10.375868+0000","last_active":"2024-09-19T06:49:52.190025+0000","last_peered":"2024-09-19T06:49:52.190025+0000","last_clean":"2024-09-19T06:49:52.190025+0000","last_became_active":"2024-09-19T06:49:10.375552+0000","last_became_peered":"2024-09-19T06:49:10.375552+0000","last_unstale":"2024-09-19T06:49:52.190025+0000","last_undegraded":"2024-09-19T06:49:52.190025+0000","last_fullsized":"2024-09-19T06:49:52.190025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:55:22.292330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191333+0000","last_change":"2024-09-19T06:49:10.365959+0000","last_active":"2024-09-19T06:49:52.191333+0000","last_peered":"2024-09-19T06:49:52.191333+0000","last_clean":"2024-09-19T06:49:52.191333+0000","last_became_active":"2024-09-19T06:49:10.365018+0000","last_became_peered":"2024-09-19T06:49:10.365018+0000","last_unstale":"2024-09-19T06:49:52.191333+0000","last_undegraded":"2024-09-19T06:49:52.191333+0000","last_fullsized":"2024-09-19T06:49:52.191333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:45:50.417270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.228424+0000","last_change":"2024-09-19T06:49:10.372988+0000","last_active":"2024-09-19T06:49:52.228424+0000","last_peered":"2024-09-19T06:49:52.228424+0000","last_clean":"2024-09-19T06:49:52.228424+0000","last_became_active":"2024-09-19T06:49:10.368995+0000","last_became_peered":"2024-09-19T06:49:10.368995+0000","last_unstale":"2024-09-19T06:49:52.228424+0000","last_undegraded":"2024-09-19T06:49:52.228424+0000","last_fullsized":"2024-09-19T06:49:52.228424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:29:04.939254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.229096+0000","last_change":"2024-09-19T06:49:52.228705+0000","last_active":"2024-09-19T06:49:52.229096+0000","last_peered":"2024-09-19T06:49:52.229096+0000","last_clean":"2024-09-19T06:49:52.229096+0000","last_became_active":"2024-09-19T06:49:52.228371+0000","last_became_peered":"2024-09-19T06:49:52.228371+0000","last_unstale":"2024-09-19T06:49:52.229096+0000","last_undegraded":"2024-09-19T06:49:52.229096+0000","last_fullsized":"2024-09-19T06:49:52.229096+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:24:58.959892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227671+0000","last_change":"2024-09-19T06:49:10.372657+0000","last_active":"2024-09-19T06:49:52.227671+0000","last_peered":"2024-09-19T06:49:52.227671+0000","last_clean":"2024-09-19T06:49:52.227671+0000","last_became_active":"2024-09-19T06:49:10.371641+0000","last_became_peered":"2024-09-19T06:49:10.371641+0000","last_unstale":"2024-09-19T06:49:52.227671+0000","last_undegraded":"2024-09-19T06:49:52.227671+0000","last_fullsized":"2024-09-19T06:49:52.227671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:54:09.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410805+0000","last_change":"2024-09-19T06:49:10.370765+0000","last_active":"2024-09-19T06:49:52.410805+0000","last_peered":"2024-09-19T06:49:52.410805+0000","last_clean":"2024-09-19T06:49:52.410805+0000","last_became_active":"2024-09-19T06:49:10.369741+0000","last_became_peered":"2024-09-19T06:49:10.369741+0000","last_unstale":"2024-09-19T06:49:52.410805+0000","last_undegraded":"2024-09-19T06:49:52.410805+0000","last_fullsized":"2024-09-19T06:49:52.410805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:35:31.377034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189842+0000","last_change":"2024-09-19T06:49:10.386328+0000","last_active":"2024-09-19T06:49:52.189842+0000","last_peered":"2024-09-19T06:49:52.189842+0000","last_clean":"2024-09-19T06:49:52.189842+0000","last_became_active":"2024-09-19T06:49:10.385970+0000","last_became_peered":"2024-09-19T06:49:10.385970+0000","last_unstale":"2024-09-19T06:49:52.189842+0000","last_undegraded":"2024-09-19T06:49:52.189842+0000","last_fullsized":"2024-09-19T06:49:52.189842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:52:27.404825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190032+0000","last_change":"2024-09-19T06:49:10.372047+0000","last_active":"2024-09-19T06:49:52.190032+0000","last_peered":"2024-09-19T06:49:52.190032+0000","last_clean":"2024-09-19T06:49:52.190032+0000","last_became_active":"2024-09-19T06:49:10.371743+0000","last_became_peered":"2024-09-19T06:49:10.371743+0000","last_unstale":"2024-09-19T06:49:52.190032+0000","last_undegraded":"2024-09-19T06:49:52.190032+0000","last_fullsized":"2024-09-19T06:49:52.190032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:06.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191151+0000","last_change":"2024-09-19T06:49:10.373221+0000","last_active":"2024-09-19T06:49:52.191151+0000","last_peered":"2024-09-19T06:49:52.191151+0000","last_clean":"2024-09-19T06:49:52.191151+0000","last_became_active":"2024-09-19T06:49:10.372859+0000","last_became_peered":"2024-09-19T06:49:10.372859+0000","last_unstale":"2024-09-19T06:49:52.191151+0000","last_undegraded":"2024-09-19T06:49:52.191151+0000","last_fullsized":"2024-09-19T06:49:52.191151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:53:19.333758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191568+0000","last_change":"2024-09-19T06:49:10.371359+0000","last_active":"2024-09-19T06:49:52.191568+0000","last_peered":"2024-09-19T06:49:52.191568+0000","last_clean":"2024-09-19T06:49:52.191568+0000","last_became_active":"2024-09-19T06:49:10.371057+0000","last_became_peered":"2024-09-19T06:49:10.371057+0000","last_unstale":"2024-09-19T06:49:52.191568+0000","last_undegraded":"2024-09-19T06:49:52.191568+0000","last_fullsized":"2024-09-19T06:49:52.191568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:18:40.387791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190196+0000","last_change":"2024-09-19T06:49:10.381371+0000","last_active":"2024-09-19T06:49:52.190196+0000","last_peered":"2024-09-19T06:49:52.190196+0000","last_clean":"2024-09-19T06:49:52.190196+0000","last_became_active":"2024-09-19T06:49:10.381017+0000","last_became_peered":"2024-09-19T06:49:10.381017+0000","last_unstale":"2024-09-19T06:49:52.190196+0000","last_undegraded":"2024-09-19T06:49:52.190196+0000","last_fullsized":"2024-09-19T06:49:52.190196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:17:54.998597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410189+0000","last_change":"2024-09-19T06:49:10.377596+0000","last_active":"2024-09-19T06:49:52.410189+0000","last_peered":"2024-09-19T06:49:52.410189+0000","last_clean":"2024-09-19T06:49:52.410189+0000","last_became_active":"2024-09-19T06:49:10.377271+0000","last_became_peered":"2024-09-19T06:49:10.377271+0000","last_unstale":"2024-09-19T06:49:52.410189+0000","last_undegraded":"2024-09-19T06:49:52.410189+0000","last_fullsized":"2024-09-19T06:49:52.410189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:32:00.797866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.194675+0000","last_change":"2024-09-19T06:49:52.194675+0000","last_active":"2024-09-19T06:49:52.194675+0000","last_peered":"2024-09-19T06:49:52.194675+0000","last_clean":"2024-09-19T06:49:52.194675+0000","last_became_active":"2024-09-19T06:49:52.194007+0000","last_became_peered":"2024-09-19T06:49:52.194007+0000","last_unstale":"2024-09-19T06:49:52.194675+0000","last_undegraded":"2024-09-19T06:49:52.194675+0000","last_fullsized":"2024-09-19T06:49:52.194675+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:45:33.109454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.186170+0000","last_change":"2024-09-19T06:49:10.387416+0000","last_active":"2024-09-19T06:49:52.186170+0000","last_peered":"2024-09-19T06:49:52.186170+0000","last_clean":"2024-09-19T06:49:52.186170+0000","last_became_active":"2024-09-19T06:49:10.384083+0000","last_became_peered":"2024-09-19T06:49:10.384083+0000","last_unstale":"2024-09-19T06:49:52.186170+0000","last_undegraded":"2024-09-19T06:49:52.186170+0000","last_fullsized":"2024-09-19T06:49:52.186170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:07:05.691663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189758+0000","last_change":"2024-09-19T06:49:10.375636+0000","last_active":"2024-09-19T06:49:52.189758+0000","last_peered":"2024-09-19T06:49:52.189758+0000","last_clean":"2024-09-19T06:49:52.189758+0000","last_became_active":"2024-09-19T06:49:10.375227+0000","last_became_peered":"2024-09-19T06:49:10.375227+0000","last_unstale":"2024-09-19T06:49:52.189758+0000","last_undegraded":"2024-09-19T06:49:52.189758+0000","last_fullsized":"2024-09-19T06:49:52.189758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:41:59.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185687+0000","last_change":"2024-09-19T06:49:10.383453+0000","last_active":"2024-09-19T06:49:52.185687+0000","last_peered":"2024-09-19T06:49:52.185687+0000","last_clean":"2024-09-19T06:49:52.185687+0000","last_became_active":"2024-09-19T06:49:10.383075+0000","last_became_peered":"2024-09-19T06:49:10.383075+0000","last_unstale":"2024-09-19T06:49:52.185687+0000","last_undegraded":"2024-09-19T06:49:52.185687+0000","last_fullsized":"2024-09-19T06:49:52.185687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:18:24.775578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187494+0000","last_change":"2024-09-19T06:49:10.378733+0000","last_active":"2024-09-19T06:49:52.187494+0000","last_peered":"2024-09-19T06:49:52.187494+0000","last_clean":"2024-09-19T06:49:52.187494+0000","last_became_active":"2024-09-19T06:49:10.378231+0000","last_became_peered":"2024-09-19T06:49:10.378231+0000","last_unstale":"2024-09-19T06:49:52.187494+0000","last_undegraded":"2024-09-19T06:49:52.187494+0000","last_fullsized":"2024-09-19T06:49:52.187494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:53:14.959853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190301+0000","last_change":"2024-09-19T06:49:10.379775+0000","last_active":"2024-09-19T06:49:52.190301+0000","last_peered":"2024-09-19T06:49:52.190301+0000","last_clean":"2024-09-19T06:49:52.190301+0000","last_became_active":"2024-09-19T06:49:10.379130+0000","last_became_peered":"2024-09-19T06:49:10.379130+0000","last_unstale":"2024-09-19T06:49:52.190301+0000","last_undegraded":"2024-09-19T06:49:52.190301+0000","last_fullsized":"2024-09-19T06:49:52.190301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:05.914354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.411100+0000","last_change":"2024-09-19T06:49:10.376239+0000","last_active":"2024-09-19T06:49:52.411100+0000","last_peered":"2024-09-19T06:49:52.411100+0000","last_clean":"2024-09-19T06:49:52.411100+0000","last_became_active":"2024-09-19T06:49:10.375913+0000","last_became_peered":"2024-09-19T06:49:10.375913+0000","last_unstale":"2024-09-19T06:49:52.411100+0000","last_undegraded":"2024-09-19T06:49:52.411100+0000","last_fullsized":"2024-09-19T06:49:52.411100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:14:26.373766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227710+0000","last_change":"2024-09-19T06:49:10.375479+0000","last_active":"2024-09-19T06:49:52.227710+0000","last_peered":"2024-09-19T06:49:52.227710+0000","last_clean":"2024-09-19T06:49:52.227710+0000","last_became_active":"2024-09-19T06:49:10.375201+0000","last_became_peered":"2024-09-19T06:49:10.375201+0000","last_unstale":"2024-09-19T06:49:52.227710+0000","last_undegraded":"2024-09-19T06:49:52.227710+0000","last_fullsized":"2024-09-19T06:49:52.227710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:44:56.421410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.186709+0000","last_change":"2024-09-19T06:49:10.374495+0000","last_active":"2024-09-19T06:49:52.186709+0000","last_peered":"2024-09-19T06:49:52.186709+0000","last_clean":"2024-09-19T06:49:52.186709+0000","last_became_active":"2024-09-19T06:49:10.374110+0000","last_became_peered":"2024-09-19T06:49:10.374110+0000","last_unstale":"2024-09-19T06:49:52.186709+0000","last_undegraded":"2024-09-19T06:49:52.186709+0000","last_fullsized":"2024-09-19T06:49:52.186709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:51:27.641785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189468+0000","last_change":"2024-09-19T06:49:10.373548+0000","last_active":"2024-09-19T06:49:52.189468+0000","last_peered":"2024-09-19T06:49:52.189468+0000","last_clean":"2024-09-19T06:49:52.189468+0000","last_became_active":"2024-09-19T06:49:10.373251+0000","last_became_peered":"2024-09-19T06:49:10.373251+0000","last_unstale":"2024-09-19T06:49:52.189468+0000","last_undegraded":"2024-09-19T06:49:52.189468+0000","last_fullsized":"2024-09-19T06:49:52.189468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:59:35.353223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190651+0000","last_change":"2024-09-19T06:49:10.383000+0000","last_active":"2024-09-19T06:49:52.190651+0000","last_peered":"2024-09-19T06:49:52.190651+0000","last_clean":"2024-09-19T06:49:52.190651+0000","last_became_active":"2024-09-19T06:49:10.382653+0000","last_became_peered":"2024-09-19T06:49:10.382653+0000","last_unstale":"2024-09-19T06:49:52.190651+0000","last_undegraded":"2024-09-19T06:49:52.190651+0000","last_fullsized":"2024-09-19T06:49:52.190651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:31.317042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185622+0000","last_change":"2024-09-19T06:49:10.380239+0000","last_active":"2024-09-19T06:49:52.185622+0000","last_peered":"2024-09-19T06:49:52.185622+0000","last_clean":"2024-09-19T06:49:52.185622+0000","last_became_active":"2024-09-19T06:49:10.378491+0000","last_became_peered":"2024-09-19T06:49:10.378491+0000","last_unstale":"2024-09-19T06:49:52.185622+0000","last_undegraded":"2024-09-19T06:49:52.185622+0000","last_fullsized":"2024-09-19T06:49:52.185622+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:56:14.016962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187477+0000","last_change":"2024-09-19T06:49:10.376989+0000","last_active":"2024-09-19T06:49:52.187477+0000","last_peered":"2024-09-19T06:49:52.187477+0000","last_clean":"2024-09-19T06:49:52.187477+0000","last_became_active":"2024-09-19T06:49:10.376701+0000","last_became_peered":"2024-09-19T06:49:10.376701+0000","last_unstale":"2024-09-19T06:49:52.187477+0000","last_undegraded":"2024-09-19T06:49:52.187477+0000","last_fullsized":"2024-09-19T06:49:52.187477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:13:34.153487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187990+0000","last_change":"2024-09-19T06:49:10.382585+0000","last_active":"2024-09-19T06:49:52.187990+0000","last_peered":"2024-09-19T06:49:52.187990+0000","last_clean":"2024-09-19T06:49:52.187990+0000","last_became_active":"2024-09-19T06:49:10.381573+0000","last_became_peered":"2024-09-19T06:49:10.381573+0000","last_unstale":"2024-09-19T06:49:52.187990+0000","last_undegraded":"2024-09-19T06:49:52.187990+0000","last_fullsized":"2024-09-19T06:49:52.187990+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:29:08.324309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189975+0000","last_change":"2024-09-19T06:49:10.381681+0000","last_active":"2024-09-19T06:49:52.189975+0000","last_peered":"2024-09-19T06:49:52.189975+0000","last_clean":"2024-09-19T06:49:52.189975+0000","last_became_active":"2024-09-19T06:49:10.381344+0000","last_became_peered":"2024-09-19T06:49:10.381344+0000","last_unstale":"2024-09-19T06:49:52.189975+0000","last_undegraded":"2024-09-19T06:49:52.189975+0000","last_fullsized":"2024-09-19T06:49:52.189975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:39:01.539679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":51298,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":170,"num_read_kb":138,"num_write":232,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":179444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215885,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449113,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":522313,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":71,"seq":304942678045,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1159168,"data_stored":1153893,"data_compressed":59753,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841572,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":526227,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1597,"internal_metadata":31914435},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103210,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":522397,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1119008,"data_compressed":59753,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495667,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":517635,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757307,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1114197,"data_compressed":59753,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":522836,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":517343,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":562634,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":575923,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":519091,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34605,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":59010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":58496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":46612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T06:50:17.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T06:50:17.582 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.195+0000 7ff122600640 1 -- 172.21.15.114:0/454526366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 msgr2=0x7ff11c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.195+0000 7ff122600640 1 --2- 172.21.15.114:0/454526366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 0x7ff11c1067e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff104009900 tx=0x7ff10402a3f0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.196+0000 7ff122600640 1 -- 172.21.15.114:0/454526366 shutdown_connections 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.196+0000 7ff122600640 1 --2- 172.21.15.114:0/454526366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11c106f10 0x7ff11c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.196+0000 7ff122600640 1 --2- 172.21.15.114:0/454526366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 0x7ff11c1067e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.196+0000 7ff122600640 1 --2- 172.21.15.114:0/454526366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff11c101de0 0x7ff11c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.196+0000 7ff122600640 1 -- 172.21.15.114:0/454526366 >> 172.21.15.114:0/454526366 conn(0x7ff11c0fd8c0 msgr2=0x7ff11c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.196+0000 7ff122600640 1 -- 172.21.15.114:0/454526366 shutdown_connections 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.196+0000 7ff122600640 1 -- 172.21.15.114:0/454526366 wait complete. 2024-09-19T06:50:18.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff122600640 1 Processor -- start 2024-09-19T06:50:18.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff122600640 1 -- start start 2024-09-19T06:50:18.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff122600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11c101de0 0x7ff11c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:18.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff122600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 0x7ff11c19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:18.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff122600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff11c106f10 0x7ff11c19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:18.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff122600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c111f80 con 0x7ff11c102790 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff122600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c111e00 con 0x7ff11c106f10 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff122600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c112100 con 0x7ff11c101de0 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff113e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 0x7ff11c19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff113e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 0x7ff11c19ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36912/0 (socket says 172.21.15.114:36912) 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff113e00640 1 -- 172.21.15.114:0/1538364850 learned_addr learned my addr 172.21.15.114:0/1538364850 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.197+0000 7ff11b400640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11c101de0 0x7ff11c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff11be00640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff11c106f10 0x7ff11c19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff113e00640 1 -- 172.21.15.114:0/1538364850 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11c101de0 msgr2=0x7ff11c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff113e00640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11c101de0 0x7ff11c19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff113e00640 1 -- 172.21.15.114:0/1538364850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff11c106f10 msgr2=0x7ff11c19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff113e00640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff11c106f10 0x7ff11c19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff113e00640 1 -- 172.21.15.114:0/1538364850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c19f5c0 con 0x7ff11c102790 2024-09-19T06:50:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff113e00640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 0x7ff11c19ab50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff104004770 tx=0x7ff104003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff118c00640 1 -- 172.21.15.114:0/1538364850 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10403f070 con 0x7ff11c102790 2024-09-19T06:50:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c19f850 con 0x7ff11c102790 2024-09-19T06:50:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.198+0000 7ff118c00640 1 -- 172.21.15.114:0/1538364850 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff104031ca0 con 0x7ff11c102790 2024-09-19T06:50:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.199+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c1a7190 con 0x7ff11c102790 2024-09-19T06:50:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.199+0000 7ff118c00640 1 -- 172.21.15.114:0/1538364850 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104049060 con 0x7ff11c102790 2024-09-19T06:50:18.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.200+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11c1021c0 con 0x7ff11c102790 2024-09-19T06:50:18.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.202+0000 7ff118c00640 1 -- 172.21.15.114:0/1538364850 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7ff104042070 con 0x7ff11c102790 2024-09-19T06:50:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.203+0000 7ff118c00640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ff1000a32d0 0x7ff1000a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.203+0000 7ff118c00640 1 -- 172.21.15.114:0/1538364850 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff1040f0830 con 0x7ff11c102790 2024-09-19T06:50:18.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.203+0000 7ff11b400640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ff1000a32d0 0x7ff1000a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:18.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.204+0000 7ff11b400640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ff1000a32d0 0x7ff1000a5790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff10c005d00 tx=0x7ff10c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:18.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.207+0000 7ff118c00640 1 -- 172.21.15.114:0/1538364850 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7ff1040bc9e0 con 0x7ff11c102790 2024-09-19T06:50:18.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.378+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff11c10de00 con 0x7ff1000a32d0 2024-09-19T06:50:18.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:18 smithi114 ceph-mon[30820]: from='client.24740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:50:18.380 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:18 smithi114 ceph-mon[30820]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-09-19T06:50:18.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.387+0000 7ff118c00640 1 -- 172.21.15.114:0/1538364850 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340632 (secure 0 0 0) 0x7ff11c10de00 con 0x7ff1000a32d0 2024-09-19T06:50:18.392 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:50:18.393 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-19T06:50:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ff1000a32d0 msgr2=0x7ff1000a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ff1000a32d0 0x7ff1000a5790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff10c005d00 tx=0x7ff10c00b410 comp rx=0 tx=0).stop 2024-09-19T06:50:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 msgr2=0x7ff11c19ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 0x7ff11c19ab50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff104004770 tx=0x7ff104003c10 comp rx=0 tx=0).stop 2024-09-19T06:50:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 shutdown_connections 2024-09-19T06:50:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7ff1000a32d0 0x7ff1000a5790 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff11c106f10 0x7ff11c19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff11c102790 0x7ff11c19ab50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 --2- 172.21.15.114:0/1538364850 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff11c101de0 0x7ff11c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:18.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 >> 172.21.15.114:0/1538364850 conn(0x7ff11c0fd8c0 msgr2=0x7ff11c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:18.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 shutdown_connections 2024-09-19T06:50:18.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:18.391+0000 7ff122600640 1 -- 172.21.15.114:0/1538364850 wait complete. 2024-09-19T06:50:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:18 smithi116 ceph-mon[35411]: from='client.24740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:50:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:18 smithi116 ceph-mon[35411]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-09-19T06:50:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:18 smithi118 ceph-mon[35527]: from='client.24740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:50:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:18 smithi118 ceph-mon[35527]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-09-19T06:50:19.191 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-19T06:50:17.099215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654675,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":322,"num_read_kb":280,"num_write":426,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":345168,"kb_used_data":12560,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124596656,"statfs":{"total":1151940427776,"available":1151586975744,"internally_reserved":0,"allocated":12861440,"data_stored":8186272,"data_compressed":390732,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19139,"internal_metadata":340374845},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":53531,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":130,"num_read_kb":124,"num_write":34,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003293"},"pg_stats":[{"pgid":"2.1f","version":"93'13","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:14.273577+0000","last_change":"2024-09-19T06:49:07.909205+0000","last_active":"2024-09-19T06:50:14.273577+0000","last_peered":"2024-09-19T06:50:14.273577+0000","last_clean":"2024-09-19T06:50:14.273577+0000","last_became_active":"2024-09-19T06:49:07.907471+0000","last_became_peered":"2024-09-19T06:49:07.907471+0000","last_unstale":"2024-09-19T06:50:14.273577+0000","last_undegraded":"2024-09-19T06:50:14.273577+0000","last_fullsized":"2024-09-19T06:50:14.273577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:55:00.655927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.197214+0000","last_change":"2024-09-19T06:49:52.196586+0000","last_active":"2024-09-19T06:49:52.197214+0000","last_peered":"2024-09-19T06:49:52.197214+0000","last_clean":"2024-09-19T06:49:52.197214+0000","last_became_active":"2024-09-19T06:49:52.195680+0000","last_became_peered":"2024-09-19T06:49:52.195680+0000","last_unstale":"2024-09-19T06:49:52.197214+0000","last_undegraded":"2024-09-19T06:49:52.197214+0000","last_fullsized":"2024-09-19T06:49:52.197214+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:46:53.026767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192504+0000","last_change":"2024-09-19T06:49:20.424335+0000","last_active":"2024-09-19T06:49:52.192504+0000","last_peered":"2024-09-19T06:49:52.192504+0000","last_clean":"2024-09-19T06:49:52.192504+0000","last_became_active":"2024-09-19T06:49:20.423890+0000","last_became_peered":"2024-09-19T06:49:20.423890+0000","last_unstale":"2024-09-19T06:49:52.192504+0000","last_undegraded":"2024-09-19T06:49:52.192504+0000","last_fullsized":"2024-09-19T06:49:52.192504+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:30:49.042220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.228465+0000","last_change":"2024-09-19T06:49:08.185286+0000","last_active":"2024-09-19T06:49:52.228465+0000","last_peered":"2024-09-19T06:49:52.228465+0000","last_clean":"2024-09-19T06:49:52.228465+0000","last_became_active":"2024-09-19T06:49:08.184971+0000","last_became_peered":"2024-09-19T06:49:08.184971+0000","last_unstale":"2024-09-19T06:49:52.228465+0000","last_undegraded":"2024-09-19T06:49:52.228465+0000","last_fullsized":"2024-09-19T06:49:52.228465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:26.532594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.188629+0000","last_change":"2024-09-19T06:49:10.385824+0000","last_active":"2024-09-19T06:49:52.188629+0000","last_peered":"2024-09-19T06:49:52.188629+0000","last_clean":"2024-09-19T06:49:52.188629+0000","last_became_active":"2024-09-19T06:49:10.381448+0000","last_became_peered":"2024-09-19T06:49:10.381448+0000","last_unstale":"2024-09-19T06:49:52.188629+0000","last_undegraded":"2024-09-19T06:49:52.188629+0000","last_fullsized":"2024-09-19T06:49:52.188629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:39.192054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184609+0000","last_change":"2024-09-19T06:49:20.429638+0000","last_active":"2024-09-19T06:49:52.184609+0000","last_peered":"2024-09-19T06:49:52.184609+0000","last_clean":"2024-09-19T06:49:52.184609+0000","last_became_active":"2024-09-19T06:49:20.429099+0000","last_became_peered":"2024-09-19T06:49:20.429099+0000","last_unstale":"2024-09-19T06:49:52.184609+0000","last_undegraded":"2024-09-19T06:49:52.184609+0000","last_fullsized":"2024-09-19T06:49:52.184609+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:01:06.221726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'12","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:59.248615+0000","last_change":"2024-09-19T06:49:07.907830+0000","last_active":"2024-09-19T06:49:59.248615+0000","last_peered":"2024-09-19T06:49:59.248615+0000","last_clean":"2024-09-19T06:49:59.248615+0000","last_became_active":"2024-09-19T06:49:07.907510+0000","last_became_peered":"2024-09-19T06:49:07.907510+0000","last_unstale":"2024-09-19T06:49:59.248615+0000","last_undegraded":"2024-09-19T06:49:59.248615+0000","last_fullsized":"2024-09-19T06:49:59.248615+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:52:46.192671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410556+0000","last_change":"2024-09-19T06:49:10.371663+0000","last_active":"2024-09-19T06:49:52.410556+0000","last_peered":"2024-09-19T06:49:52.410556+0000","last_clean":"2024-09-19T06:49:52.410556+0000","last_became_active":"2024-09-19T06:49:10.371325+0000","last_became_peered":"2024-09-19T06:49:10.371325+0000","last_unstale":"2024-09-19T06:49:52.410556+0000","last_undegraded":"2024-09-19T06:49:52.410556+0000","last_fullsized":"2024-09-19T06:49:52.410556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:48:58.406933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185210+0000","last_change":"2024-09-19T06:49:20.429446+0000","last_active":"2024-09-19T06:49:52.185210+0000","last_peered":"2024-09-19T06:49:52.185210+0000","last_clean":"2024-09-19T06:49:52.185210+0000","last_became_active":"2024-09-19T06:49:20.428996+0000","last_became_peered":"2024-09-19T06:49:20.428996+0000","last_unstale":"2024-09-19T06:49:52.185210+0000","last_undegraded":"2024-09-19T06:49:52.185210+0000","last_fullsized":"2024-09-19T06:49:52.185210+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:39.542347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"93'5","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.245239+0000","last_change":"2024-09-19T06:49:07.906940+0000","last_active":"2024-09-19T06:49:58.245239+0000","last_peered":"2024-09-19T06:49:58.245239+0000","last_clean":"2024-09-19T06:49:58.245239+0000","last_became_active":"2024-09-19T06:49:07.906635+0000","last_became_peered":"2024-09-19T06:49:07.906635+0000","last_unstale":"2024-09-19T06:49:58.245239+0000","last_undegraded":"2024-09-19T06:49:58.245239+0000","last_fullsized":"2024-09-19T06:49:58.245239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:00.794049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.411553+0000","last_change":"2024-09-19T06:49:52.411553+0000","last_active":"2024-09-19T06:49:52.411553+0000","last_peered":"2024-09-19T06:49:52.411553+0000","last_clean":"2024-09-19T06:49:52.411553+0000","last_became_active":"2024-09-19T06:49:52.411251+0000","last_became_peered":"2024-09-19T06:49:52.411251+0000","last_unstale":"2024-09-19T06:49:52.411553+0000","last_undegraded":"2024-09-19T06:49:52.411553+0000","last_fullsized":"2024-09-19T06:49:52.411553+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:17.774698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184753+0000","last_change":"2024-09-19T06:49:20.427188+0000","last_active":"2024-09-19T06:49:52.184753+0000","last_peered":"2024-09-19T06:49:52.184753+0000","last_clean":"2024-09-19T06:49:52.184753+0000","last_became_active":"2024-09-19T06:49:20.426716+0000","last_became_peered":"2024-09-19T06:49:20.426716+0000","last_unstale":"2024-09-19T06:49:52.184753+0000","last_undegraded":"2024-09-19T06:49:52.184753+0000","last_fullsized":"2024-09-19T06:49:52.184753+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:25:09.086438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187080+0000","last_change":"2024-09-19T06:49:10.371433+0000","last_active":"2024-09-19T06:49:52.187080+0000","last_peered":"2024-09-19T06:49:52.187080+0000","last_clean":"2024-09-19T06:49:52.187080+0000","last_became_active":"2024-09-19T06:49:10.370947+0000","last_became_peered":"2024-09-19T06:49:10.370947+0000","last_unstale":"2024-09-19T06:49:52.187080+0000","last_undegraded":"2024-09-19T06:49:52.187080+0000","last_fullsized":"2024-09-19T06:49:52.187080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:40.954670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:55.221780+0000","last_change":"2024-09-19T06:49:07.907972+0000","last_active":"2024-09-19T06:49:55.221780+0000","last_peered":"2024-09-19T06:49:55.221780+0000","last_clean":"2024-09-19T06:49:55.221780+0000","last_became_active":"2024-09-19T06:49:07.907631+0000","last_became_peered":"2024-09-19T06:49:07.907631+0000","last_unstale":"2024-09-19T06:49:55.221780+0000","last_undegraded":"2024-09-19T06:49:55.221780+0000","last_fullsized":"2024-09-19T06:49:55.221780+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:12:16.298696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185090+0000","last_change":"2024-09-19T06:49:20.617446+0000","last_active":"2024-09-19T06:49:52.185090+0000","last_peered":"2024-09-19T06:49:52.185090+0000","last_clean":"2024-09-19T06:49:52.185090+0000","last_became_active":"2024-09-19T06:49:20.616950+0000","last_became_peered":"2024-09-19T06:49:20.616950+0000","last_unstale":"2024-09-19T06:49:52.185090+0000","last_undegraded":"2024-09-19T06:49:52.185090+0000","last_fullsized":"2024-09-19T06:49:52.185090+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:37:00.640364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.197003+0000","last_change":"2024-09-19T06:49:52.195937+0000","last_active":"2024-09-19T06:49:52.197003+0000","last_peered":"2024-09-19T06:49:52.197003+0000","last_clean":"2024-09-19T06:49:52.197003+0000","last_became_active":"2024-09-19T06:49:52.195615+0000","last_became_peered":"2024-09-19T06:49:52.195615+0000","last_unstale":"2024-09-19T06:49:52.197003+0000","last_undegraded":"2024-09-19T06:49:52.197003+0000","last_fullsized":"2024-09-19T06:49:52.197003+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:28:33.296849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.a","version":"93'3","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:57.233356+0000","last_change":"2024-09-19T06:49:07.906081+0000","last_active":"2024-09-19T06:49:57.233356+0000","last_peered":"2024-09-19T06:49:57.233356+0000","last_clean":"2024-09-19T06:49:57.233356+0000","last_became_active":"2024-09-19T06:49:07.905559+0000","last_became_peered":"2024-09-19T06:49:07.905559+0000","last_unstale":"2024-09-19T06:49:57.233356+0000","last_undegraded":"2024-09-19T06:49:57.233356+0000","last_fullsized":"2024-09-19T06:49:57.233356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:30:56.922084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184922+0000","last_change":"2024-09-19T06:49:20.431255+0000","last_active":"2024-09-19T06:49:52.184922+0000","last_peered":"2024-09-19T06:49:52.184922+0000","last_clean":"2024-09-19T06:49:52.184922+0000","last_became_active":"2024-09-19T06:49:20.430727+0000","last_became_peered":"2024-09-19T06:49:20.430727+0000","last_unstale":"2024-09-19T06:49:52.184922+0000","last_undegraded":"2024-09-19T06:49:52.184922+0000","last_fullsized":"2024-09-19T06:49:52.184922+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:01.761121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189565+0000","last_change":"2024-09-19T06:49:10.382739+0000","last_active":"2024-09-19T06:49:52.189565+0000","last_peered":"2024-09-19T06:49:52.189565+0000","last_clean":"2024-09-19T06:49:52.189565+0000","last_became_active":"2024-09-19T06:49:10.382216+0000","last_became_peered":"2024-09-19T06:49:10.382216+0000","last_unstale":"2024-09-19T06:49:52.189565+0000","last_undegraded":"2024-09-19T06:49:52.189565+0000","last_fullsized":"2024-09-19T06:49:52.189565+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:31:37.908345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"93'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.243678+0000","last_change":"2024-09-19T06:49:07.906876+0000","last_active":"2024-09-19T06:49:58.243678+0000","last_peered":"2024-09-19T06:49:58.243678+0000","last_clean":"2024-09-19T06:49:58.243678+0000","last_became_active":"2024-09-19T06:49:07.906567+0000","last_became_peered":"2024-09-19T06:49:07.906567+0000","last_unstale":"2024-09-19T06:49:58.243678+0000","last_undegraded":"2024-09-19T06:49:58.243678+0000","last_fullsized":"2024-09-19T06:49:58.243678+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:21:39.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192165+0000","last_change":"2024-09-19T06:49:20.437455+0000","last_active":"2024-09-19T06:49:52.192165+0000","last_peered":"2024-09-19T06:49:52.192165+0000","last_clean":"2024-09-19T06:49:52.192165+0000","last_became_active":"2024-09-19T06:49:20.437050+0000","last_became_peered":"2024-09-19T06:49:20.437050+0000","last_unstale":"2024-09-19T06:49:52.192165+0000","last_undegraded":"2024-09-19T06:49:52.192165+0000","last_fullsized":"2024-09-19T06:49:52.192165+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:46:04.206859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189107+0000","last_change":"2024-09-19T06:49:10.369037+0000","last_active":"2024-09-19T06:49:52.189107+0000","last_peered":"2024-09-19T06:49:52.189107+0000","last_clean":"2024-09-19T06:49:52.189107+0000","last_became_active":"2024-09-19T06:49:10.368653+0000","last_became_peered":"2024-09-19T06:49:10.368653+0000","last_unstale":"2024-09-19T06:49:52.189107+0000","last_undegraded":"2024-09-19T06:49:52.189107+0000","last_fullsized":"2024-09-19T06:49:52.189107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:46:02.268843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:55.222469+0000","last_change":"2024-09-19T06:49:08.185517+0000","last_active":"2024-09-19T06:49:55.222469+0000","last_peered":"2024-09-19T06:49:55.222469+0000","last_clean":"2024-09-19T06:49:55.222469+0000","last_became_active":"2024-09-19T06:49:08.185138+0000","last_became_peered":"2024-09-19T06:49:08.185138+0000","last_unstale":"2024-09-19T06:49:55.222469+0000","last_undegraded":"2024-09-19T06:49:55.222469+0000","last_fullsized":"2024-09-19T06:49:55.222469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:57:59.082821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185580+0000","last_change":"2024-09-19T06:49:20.617423+0000","last_active":"2024-09-19T06:49:52.185580+0000","last_peered":"2024-09-19T06:49:52.185580+0000","last_clean":"2024-09-19T06:49:52.185580+0000","last_became_active":"2024-09-19T06:49:20.616834+0000","last_became_peered":"2024-09-19T06:49:20.616834+0000","last_unstale":"2024-09-19T06:49:52.185580+0000","last_undegraded":"2024-09-19T06:49:52.185580+0000","last_fullsized":"2024-09-19T06:49:52.185580+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:53:47.384169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.188118+0000","last_change":"2024-09-19T06:49:10.388955+0000","last_active":"2024-09-19T06:49:52.188118+0000","last_peered":"2024-09-19T06:49:52.188118+0000","last_clean":"2024-09-19T06:49:52.188118+0000","last_became_active":"2024-09-19T06:49:10.387947+0000","last_became_peered":"2024-09-19T06:49:10.387947+0000","last_unstale":"2024-09-19T06:49:52.188118+0000","last_undegraded":"2024-09-19T06:49:52.188118+0000","last_fullsized":"2024-09-19T06:49:52.188118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:44.556097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"93'7","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.245539+0000","last_change":"2024-09-19T06:49:07.912897+0000","last_active":"2024-09-19T06:49:58.245539+0000","last_peered":"2024-09-19T06:49:58.245539+0000","last_clean":"2024-09-19T06:49:58.245539+0000","last_became_active":"2024-09-19T06:49:07.912520+0000","last_became_peered":"2024-09-19T06:49:07.912520+0000","last_unstale":"2024-09-19T06:49:58.245539+0000","last_undegraded":"2024-09-19T06:49:58.245539+0000","last_fullsized":"2024-09-19T06:49:58.245539+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:23:56.582808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192795+0000","last_change":"2024-09-19T06:49:20.420851+0000","last_active":"2024-09-19T06:49:52.192795+0000","last_peered":"2024-09-19T06:49:52.192795+0000","last_clean":"2024-09-19T06:49:52.192795+0000","last_became_active":"2024-09-19T06:49:20.420307+0000","last_became_peered":"2024-09-19T06:49:20.420307+0000","last_unstale":"2024-09-19T06:49:52.192795+0000","last_undegraded":"2024-09-19T06:49:52.192795+0000","last_fullsized":"2024-09-19T06:49:52.192795+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:20.633070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.228471+0000","last_change":"2024-09-19T06:49:10.372935+0000","last_active":"2024-09-19T06:49:52.228471+0000","last_peered":"2024-09-19T06:49:52.228471+0000","last_clean":"2024-09-19T06:49:52.228471+0000","last_became_active":"2024-09-19T06:49:10.372641+0000","last_became_peered":"2024-09-19T06:49:10.372641+0000","last_unstale":"2024-09-19T06:49:52.228471+0000","last_undegraded":"2024-09-19T06:49:52.228471+0000","last_fullsized":"2024-09-19T06:49:52.228471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:13:07.917094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'15","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:13.380271+0000","last_change":"2024-09-19T06:49:07.908184+0000","last_active":"2024-09-19T06:50:13.380271+0000","last_peered":"2024-09-19T06:50:13.380271+0000","last_clean":"2024-09-19T06:50:13.380271+0000","last_became_active":"2024-09-19T06:49:07.907722+0000","last_became_peered":"2024-09-19T06:49:07.907722+0000","last_unstale":"2024-09-19T06:50:13.380271+0000","last_undegraded":"2024-09-19T06:50:13.380271+0000","last_fullsized":"2024-09-19T06:50:13.380271+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:24:25.957501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227438+0000","last_change":"2024-09-19T06:49:20.616966+0000","last_active":"2024-09-19T06:49:52.227438+0000","last_peered":"2024-09-19T06:49:52.227438+0000","last_clean":"2024-09-19T06:49:52.227438+0000","last_became_active":"2024-09-19T06:49:20.616402+0000","last_became_peered":"2024-09-19T06:49:20.616402+0000","last_unstale":"2024-09-19T06:49:52.227438+0000","last_undegraded":"2024-09-19T06:49:52.227438+0000","last_fullsized":"2024-09-19T06:49:52.227438+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:15:12.572214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189770+0000","last_change":"2024-09-19T06:49:10.373048+0000","last_active":"2024-09-19T06:49:52.189770+0000","last_peered":"2024-09-19T06:49:52.189770+0000","last_clean":"2024-09-19T06:49:52.189770+0000","last_became_active":"2024-09-19T06:49:10.372257+0000","last_became_peered":"2024-09-19T06:49:10.372257+0000","last_unstale":"2024-09-19T06:49:52.189770+0000","last_undegraded":"2024-09-19T06:49:52.189770+0000","last_fullsized":"2024-09-19T06:49:52.189770+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:30.707948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.243735+0000","last_change":"2024-09-19T06:49:52.195314+0000","last_active":"2024-09-19T06:49:58.243735+0000","last_peered":"2024-09-19T06:49:58.243735+0000","last_clean":"2024-09-19T06:49:58.243735+0000","last_became_active":"2024-09-19T06:49:52.194849+0000","last_became_peered":"2024-09-19T06:49:52.194849+0000","last_unstale":"2024-09-19T06:49:58.243735+0000","last_undegraded":"2024-09-19T06:49:58.243735+0000","last_fullsized":"2024-09-19T06:49:58.243735+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:01:20.902382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189734+0000","last_change":"2024-09-19T06:49:20.422658+0000","last_active":"2024-09-19T06:49:52.189734+0000","last_peered":"2024-09-19T06:49:52.189734+0000","last_clean":"2024-09-19T06:49:52.189734+0000","last_became_active":"2024-09-19T06:49:20.422190+0000","last_became_peered":"2024-09-19T06:49:20.422190+0000","last_unstale":"2024-09-19T06:49:52.189734+0000","last_undegraded":"2024-09-19T06:49:52.189734+0000","last_fullsized":"2024-09-19T06:49:52.189734+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:13:14.338388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189710+0000","last_change":"2024-09-19T06:49:10.376579+0000","last_active":"2024-09-19T06:49:52.189710+0000","last_peered":"2024-09-19T06:49:52.189710+0000","last_clean":"2024-09-19T06:49:52.189710+0000","last_became_active":"2024-09-19T06:49:10.375920+0000","last_became_peered":"2024-09-19T06:49:10.375920+0000","last_unstale":"2024-09-19T06:49:52.189710+0000","last_undegraded":"2024-09-19T06:49:52.189710+0000","last_fullsized":"2024-09-19T06:49:52.189710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:47:35.976117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"93'1","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190394+0000","last_change":"2024-09-19T06:49:08.187353+0000","last_active":"2024-09-19T06:49:52.190394+0000","last_peered":"2024-09-19T06:49:52.190394+0000","last_clean":"2024-09-19T06:49:52.190394+0000","last_became_active":"2024-09-19T06:49:08.186860+0000","last_became_peered":"2024-09-19T06:49:08.186860+0000","last_unstale":"2024-09-19T06:49:52.190394+0000","last_undegraded":"2024-09-19T06:49:52.190394+0000","last_fullsized":"2024-09-19T06:49:52.190394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:45:06.236143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189149+0000","last_change":"2024-09-19T06:49:20.617028+0000","last_active":"2024-09-19T06:49:52.189149+0000","last_peered":"2024-09-19T06:49:52.189149+0000","last_clean":"2024-09-19T06:49:52.189149+0000","last_became_active":"2024-09-19T06:49:20.616443+0000","last_became_peered":"2024-09-19T06:49:20.616443+0000","last_unstale":"2024-09-19T06:49:52.189149+0000","last_undegraded":"2024-09-19T06:49:52.189149+0000","last_fullsized":"2024-09-19T06:49:52.189149+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:46:30.394051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189702+0000","last_change":"2024-09-19T06:49:10.386030+0000","last_active":"2024-09-19T06:49:52.189702+0000","last_peered":"2024-09-19T06:49:52.189702+0000","last_clean":"2024-09-19T06:49:52.189702+0000","last_became_active":"2024-09-19T06:49:10.385698+0000","last_became_peered":"2024-09-19T06:49:10.385698+0000","last_unstale":"2024-09-19T06:49:52.189702+0000","last_undegraded":"2024-09-19T06:49:52.189702+0000","last_fullsized":"2024-09-19T06:49:52.189702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:56:12.688507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'5","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:12.271306+0000","last_change":"2024-09-19T06:49:07.910172+0000","last_active":"2024-09-19T06:50:12.271306+0000","last_peered":"2024-09-19T06:50:12.271306+0000","last_clean":"2024-09-19T06:50:12.271306+0000","last_became_active":"2024-09-19T06:49:07.909798+0000","last_became_peered":"2024-09-19T06:49:07.909798+0000","last_unstale":"2024-09-19T06:50:12.271306+0000","last_undegraded":"2024-09-19T06:50:12.271306+0000","last_fullsized":"2024-09-19T06:50:12.271306+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.592988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189691+0000","last_change":"2024-09-19T06:49:20.424233+0000","last_active":"2024-09-19T06:49:52.189691+0000","last_peered":"2024-09-19T06:49:52.189691+0000","last_clean":"2024-09-19T06:49:52.189691+0000","last_became_active":"2024-09-19T06:49:20.423797+0000","last_became_peered":"2024-09-19T06:49:20.423797+0000","last_unstale":"2024-09-19T06:49:52.189691+0000","last_undegraded":"2024-09-19T06:49:52.189691+0000","last_fullsized":"2024-09-19T06:49:52.189691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:58:54.983250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184233+0000","last_change":"2024-09-19T06:49:10.376408+0000","last_active":"2024-09-19T06:49:52.184233+0000","last_peered":"2024-09-19T06:49:52.184233+0000","last_clean":"2024-09-19T06:49:52.184233+0000","last_became_active":"2024-09-19T06:49:10.376065+0000","last_became_peered":"2024-09-19T06:49:10.376065+0000","last_unstale":"2024-09-19T06:49:52.184233+0000","last_undegraded":"2024-09-19T06:49:52.184233+0000","last_fullsized":"2024-09-19T06:49:52.184233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:28:06.175670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:13.272071+0000","last_change":"2024-09-19T06:49:08.185183+0000","last_active":"2024-09-19T06:50:13.272071+0000","last_peered":"2024-09-19T06:50:13.272071+0000","last_clean":"2024-09-19T06:50:13.272071+0000","last_became_active":"2024-09-19T06:49:08.184738+0000","last_became_peered":"2024-09-19T06:49:08.184738+0000","last_unstale":"2024-09-19T06:50:13.272071+0000","last_undegraded":"2024-09-19T06:50:13.272071+0000","last_fullsized":"2024-09-19T06:50:13.272071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:19:54.966966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191461+0000","last_change":"2024-09-19T06:49:20.617986+0000","last_active":"2024-09-19T06:49:52.191461+0000","last_peered":"2024-09-19T06:49:52.191461+0000","last_clean":"2024-09-19T06:49:52.191461+0000","last_became_active":"2024-09-19T06:49:20.617325+0000","last_became_peered":"2024-09-19T06:49:20.617325+0000","last_unstale":"2024-09-19T06:49:52.191461+0000","last_undegraded":"2024-09-19T06:49:52.191461+0000","last_fullsized":"2024-09-19T06:49:52.191461+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:03.825005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189212+0000","last_change":"2024-09-19T06:49:10.385230+0000","last_active":"2024-09-19T06:49:52.189212+0000","last_peered":"2024-09-19T06:49:52.189212+0000","last_clean":"2024-09-19T06:49:52.189212+0000","last_became_active":"2024-09-19T06:49:10.384633+0000","last_became_peered":"2024-09-19T06:49:10.384633+0000","last_unstale":"2024-09-19T06:49:52.189212+0000","last_undegraded":"2024-09-19T06:49:52.189212+0000","last_fullsized":"2024-09-19T06:49:52.189212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:16:27.751338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:56.224385+0000","last_change":"2024-09-19T06:49:07.908823+0000","last_active":"2024-09-19T06:49:56.224385+0000","last_peered":"2024-09-19T06:49:56.224385+0000","last_clean":"2024-09-19T06:49:56.224385+0000","last_became_active":"2024-09-19T06:49:07.908340+0000","last_became_peered":"2024-09-19T06:49:07.908340+0000","last_unstale":"2024-09-19T06:49:56.224385+0000","last_undegraded":"2024-09-19T06:49:56.224385+0000","last_fullsized":"2024-09-19T06:49:56.224385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:25.198073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184542+0000","last_change":"2024-09-19T06:49:20.617412+0000","last_active":"2024-09-19T06:49:52.184542+0000","last_peered":"2024-09-19T06:49:52.184542+0000","last_clean":"2024-09-19T06:49:52.184542+0000","last_became_active":"2024-09-19T06:49:20.616908+0000","last_became_peered":"2024-09-19T06:49:20.616908+0000","last_unstale":"2024-09-19T06:49:52.184542+0000","last_undegraded":"2024-09-19T06:49:52.184542+0000","last_fullsized":"2024-09-19T06:49:52.184542+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:44:12.121754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227902+0000","last_change":"2024-09-19T06:49:10.374761+0000","last_active":"2024-09-19T06:49:52.227902+0000","last_peered":"2024-09-19T06:49:52.227902+0000","last_clean":"2024-09-19T06:49:52.227902+0000","last_became_active":"2024-09-19T06:49:10.374258+0000","last_became_peered":"2024-09-19T06:49:10.374258+0000","last_unstale":"2024-09-19T06:49:52.227902+0000","last_undegraded":"2024-09-19T06:49:52.227902+0000","last_fullsized":"2024-09-19T06:49:52.227902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:23:23.850658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'6","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:59.248949+0000","last_change":"2024-09-19T06:49:07.912648+0000","last_active":"2024-09-19T06:49:59.248949+0000","last_peered":"2024-09-19T06:49:59.248949+0000","last_clean":"2024-09-19T06:49:59.248949+0000","last_became_active":"2024-09-19T06:49:07.912289+0000","last_became_peered":"2024-09-19T06:49:07.912289+0000","last_unstale":"2024-09-19T06:49:59.248949+0000","last_undegraded":"2024-09-19T06:49:59.248949+0000","last_fullsized":"2024-09-19T06:49:59.248949+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:36.211030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187276+0000","last_change":"2024-09-19T06:47:53.158465+0000","last_active":"2024-09-19T06:49:52.187276+0000","last_peered":"2024-09-19T06:49:52.187276+0000","last_clean":"2024-09-19T06:49:52.187276+0000","last_became_active":"2024-09-19T06:47:53.158059+0000","last_became_peered":"2024-09-19T06:47:53.158059+0000","last_unstale":"2024-09-19T06:49:52.187276+0000","last_undegraded":"2024-09-19T06:49:52.187276+0000","last_fullsized":"2024-09-19T06:49:52.187276+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:44:13.613197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:44:13.613197+0000","last_clean_scrub_stamp":"2024-09-19T06:44:13.613197+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:16:41.310229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187421+0000","last_change":"2024-09-19T06:49:20.422112+0000","last_active":"2024-09-19T06:49:52.187421+0000","last_peered":"2024-09-19T06:49:52.187421+0000","last_clean":"2024-09-19T06:49:52.187421+0000","last_became_active":"2024-09-19T06:49:20.421720+0000","last_became_peered":"2024-09-19T06:49:20.421720+0000","last_unstale":"2024-09-19T06:49:52.187421+0000","last_undegraded":"2024-09-19T06:49:52.187421+0000","last_fullsized":"2024-09-19T06:49:52.187421+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:35:25.923946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187492+0000","last_change":"2024-09-19T06:49:10.379414+0000","last_active":"2024-09-19T06:49:52.187492+0000","last_peered":"2024-09-19T06:49:52.187492+0000","last_clean":"2024-09-19T06:49:52.187492+0000","last_became_active":"2024-09-19T06:49:10.379075+0000","last_became_peered":"2024-09-19T06:49:10.379075+0000","last_unstale":"2024-09-19T06:49:52.187492+0000","last_undegraded":"2024-09-19T06:49:52.187492+0000","last_fullsized":"2024-09-19T06:49:52.187492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:32:57.976929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'5","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:12.270106+0000","last_change":"2024-09-19T06:49:08.186883+0000","last_active":"2024-09-19T06:50:12.270106+0000","last_peered":"2024-09-19T06:50:12.270106+0000","last_clean":"2024-09-19T06:50:12.270106+0000","last_became_active":"2024-09-19T06:49:08.186472+0000","last_became_peered":"2024-09-19T06:49:08.186472+0000","last_unstale":"2024-09-19T06:50:12.270106+0000","last_undegraded":"2024-09-19T06:50:12.270106+0000","last_fullsized":"2024-09-19T06:50:12.270106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:58:12.146032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410556+0000","last_change":"2024-09-19T06:49:20.418542+0000","last_active":"2024-09-19T06:49:52.410556+0000","last_peered":"2024-09-19T06:49:52.410556+0000","last_clean":"2024-09-19T06:49:52.410556+0000","last_became_active":"2024-09-19T06:49:20.418044+0000","last_became_peered":"2024-09-19T06:49:20.418044+0000","last_unstale":"2024-09-19T06:49:52.410556+0000","last_undegraded":"2024-09-19T06:49:52.410556+0000","last_fullsized":"2024-09-19T06:49:52.410556+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:44:02.695919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187189+0000","last_change":"2024-09-19T06:49:10.373862+0000","last_active":"2024-09-19T06:49:52.187189+0000","last_peered":"2024-09-19T06:49:52.187189+0000","last_clean":"2024-09-19T06:49:52.187189+0000","last_became_active":"2024-09-19T06:49:10.373345+0000","last_became_peered":"2024-09-19T06:49:10.373345+0000","last_unstale":"2024-09-19T06:49:52.187189+0000","last_undegraded":"2024-09-19T06:49:52.187189+0000","last_fullsized":"2024-09-19T06:49:52.187189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:33.562984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'3","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.243953+0000","last_change":"2024-09-19T06:49:07.909407+0000","last_active":"2024-09-19T06:49:58.243953+0000","last_peered":"2024-09-19T06:49:58.243953+0000","last_clean":"2024-09-19T06:49:58.243953+0000","last_became_active":"2024-09-19T06:49:07.909101+0000","last_became_peered":"2024-09-19T06:49:07.909101+0000","last_unstale":"2024-09-19T06:49:58.243953+0000","last_undegraded":"2024-09-19T06:49:58.243953+0000","last_fullsized":"2024-09-19T06:49:58.243953+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:08:47.317018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187005+0000","last_change":"2024-09-19T06:49:20.425109+0000","last_active":"2024-09-19T06:49:52.187005+0000","last_peered":"2024-09-19T06:49:52.187005+0000","last_clean":"2024-09-19T06:49:52.187005+0000","last_became_active":"2024-09-19T06:49:20.424701+0000","last_became_peered":"2024-09-19T06:49:20.424701+0000","last_unstale":"2024-09-19T06:49:52.187005+0000","last_undegraded":"2024-09-19T06:49:52.187005+0000","last_fullsized":"2024-09-19T06:49:52.187005+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:13:10.878691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187467+0000","last_change":"2024-09-19T06:49:10.374089+0000","last_active":"2024-09-19T06:49:52.187467+0000","last_peered":"2024-09-19T06:49:52.187467+0000","last_clean":"2024-09-19T06:49:52.187467+0000","last_became_active":"2024-09-19T06:49:10.373729+0000","last_became_peered":"2024-09-19T06:49:10.373729+0000","last_unstale":"2024-09-19T06:49:52.187467+0000","last_undegraded":"2024-09-19T06:49:52.187467+0000","last_fullsized":"2024-09-19T06:49:52.187467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:52.876874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"93'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.244235+0000","last_change":"2024-09-19T06:49:07.909616+0000","last_active":"2024-09-19T06:49:58.244235+0000","last_peered":"2024-09-19T06:49:58.244235+0000","last_clean":"2024-09-19T06:49:58.244235+0000","last_became_active":"2024-09-19T06:49:07.909172+0000","last_became_peered":"2024-09-19T06:49:07.909172+0000","last_unstale":"2024-09-19T06:49:58.244235+0000","last_undegraded":"2024-09-19T06:49:58.244235+0000","last_fullsized":"2024-09-19T06:49:58.244235+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:57:27.685639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.183999+0000","last_change":"2024-09-19T06:49:20.419188+0000","last_active":"2024-09-19T06:49:52.183999+0000","last_peered":"2024-09-19T06:49:52.183999+0000","last_clean":"2024-09-19T06:49:52.183999+0000","last_became_active":"2024-09-19T06:49:20.418641+0000","last_became_peered":"2024-09-19T06:49:20.418641+0000","last_unstale":"2024-09-19T06:49:52.183999+0000","last_undegraded":"2024-09-19T06:49:52.183999+0000","last_fullsized":"2024-09-19T06:49:52.183999+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:06:05.278027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189695+0000","last_change":"2024-09-19T06:49:10.378491+0000","last_active":"2024-09-19T06:49:52.189695+0000","last_peered":"2024-09-19T06:49:52.189695+0000","last_clean":"2024-09-19T06:49:52.189695+0000","last_became_active":"2024-09-19T06:49:10.378141+0000","last_became_peered":"2024-09-19T06:49:10.378141+0000","last_unstale":"2024-09-19T06:49:52.189695+0000","last_undegraded":"2024-09-19T06:49:52.189695+0000","last_fullsized":"2024-09-19T06:49:52.189695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:57:39.195604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.221046+0000","last_change":"2024-09-19T06:49:07.909836+0000","last_active":"2024-09-19T06:49:58.221046+0000","last_peered":"2024-09-19T06:49:58.221046+0000","last_clean":"2024-09-19T06:49:58.221046+0000","last_became_active":"2024-09-19T06:49:07.909527+0000","last_became_peered":"2024-09-19T06:49:07.909527+0000","last_unstale":"2024-09-19T06:49:58.221046+0000","last_undegraded":"2024-09-19T06:49:58.221046+0000","last_fullsized":"2024-09-19T06:49:58.221046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:51.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184155+0000","last_change":"2024-09-19T06:49:20.426184+0000","last_active":"2024-09-19T06:49:52.184155+0000","last_peered":"2024-09-19T06:49:52.184155+0000","last_clean":"2024-09-19T06:49:52.184155+0000","last_became_active":"2024-09-19T06:49:20.425844+0000","last_became_peered":"2024-09-19T06:49:20.425844+0000","last_unstale":"2024-09-19T06:49:52.184155+0000","last_undegraded":"2024-09-19T06:49:52.184155+0000","last_fullsized":"2024-09-19T06:49:52.184155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.120462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184354+0000","last_change":"2024-09-19T06:49:10.380589+0000","last_active":"2024-09-19T06:49:52.184354+0000","last_peered":"2024-09-19T06:49:52.184354+0000","last_clean":"2024-09-19T06:49:52.184354+0000","last_became_active":"2024-09-19T06:49:10.380263+0000","last_became_peered":"2024-09-19T06:49:10.380263+0000","last_unstale":"2024-09-19T06:49:52.184354+0000","last_undegraded":"2024-09-19T06:49:52.184354+0000","last_fullsized":"2024-09-19T06:49:52.184354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:30:48.426240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"93'17","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.245456+0000","last_change":"2024-09-19T06:49:52.199971+0000","last_active":"2024-09-19T06:49:58.245456+0000","last_peered":"2024-09-19T06:49:58.245456+0000","last_clean":"2024-09-19T06:49:58.245456+0000","last_became_active":"2024-09-19T06:49:52.191887+0000","last_became_peered":"2024-09-19T06:49:52.191887+0000","last_unstale":"2024-09-19T06:49:58.245456+0000","last_undegraded":"2024-09-19T06:49:58.245456+0000","last_fullsized":"2024-09-19T06:49:58.245456+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:22:40.775893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191728+0000","last_change":"2024-09-19T06:49:20.435213+0000","last_active":"2024-09-19T06:49:52.191728+0000","last_peered":"2024-09-19T06:49:52.191728+0000","last_clean":"2024-09-19T06:49:52.191728+0000","last_became_active":"2024-09-19T06:49:20.434776+0000","last_became_peered":"2024-09-19T06:49:20.434776+0000","last_unstale":"2024-09-19T06:49:52.191728+0000","last_undegraded":"2024-09-19T06:49:52.191728+0000","last_fullsized":"2024-09-19T06:49:52.191728+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:37.067947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.186939+0000","last_change":"2024-09-19T06:49:10.375394+0000","last_active":"2024-09-19T06:49:52.186939+0000","last_peered":"2024-09-19T06:49:52.186939+0000","last_clean":"2024-09-19T06:49:52.186939+0000","last_became_active":"2024-09-19T06:49:10.375056+0000","last_became_peered":"2024-09-19T06:49:10.375056+0000","last_unstale":"2024-09-19T06:49:52.186939+0000","last_undegraded":"2024-09-19T06:49:52.186939+0000","last_fullsized":"2024-09-19T06:49:52.186939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:18:40.158481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'6","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:12.271341+0000","last_change":"2024-09-19T06:49:07.910215+0000","last_active":"2024-09-19T06:50:12.271341+0000","last_peered":"2024-09-19T06:50:12.271341+0000","last_clean":"2024-09-19T06:50:12.271341+0000","last_became_active":"2024-09-19T06:49:07.909908+0000","last_became_peered":"2024-09-19T06:49:07.909908+0000","last_unstale":"2024-09-19T06:50:12.271341+0000","last_undegraded":"2024-09-19T06:50:12.271341+0000","last_fullsized":"2024-09-19T06:50:12.271341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:45:05.432960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192261+0000","last_change":"2024-09-19T06:49:20.434400+0000","last_active":"2024-09-19T06:49:52.192261+0000","last_peered":"2024-09-19T06:49:52.192261+0000","last_clean":"2024-09-19T06:49:52.192261+0000","last_became_active":"2024-09-19T06:49:20.433877+0000","last_became_peered":"2024-09-19T06:49:20.433877+0000","last_unstale":"2024-09-19T06:49:52.192261+0000","last_undegraded":"2024-09-19T06:49:52.192261+0000","last_fullsized":"2024-09-19T06:49:52.192261+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:24:16.378509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187261+0000","last_change":"2024-09-19T06:49:10.378048+0000","last_active":"2024-09-19T06:49:52.187261+0000","last_peered":"2024-09-19T06:49:52.187261+0000","last_clean":"2024-09-19T06:49:52.187261+0000","last_became_active":"2024-09-19T06:49:10.377710+0000","last_became_peered":"2024-09-19T06:49:10.377710+0000","last_unstale":"2024-09-19T06:49:52.187261+0000","last_undegraded":"2024-09-19T06:49:52.187261+0000","last_fullsized":"2024-09-19T06:49:52.187261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:41:54.620218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"93'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.244459+0000","last_change":"2024-09-19T06:49:07.907169+0000","last_active":"2024-09-19T06:49:58.244459+0000","last_peered":"2024-09-19T06:49:58.244459+0000","last_clean":"2024-09-19T06:49:58.244459+0000","last_became_active":"2024-09-19T06:49:07.906811+0000","last_became_peered":"2024-09-19T06:49:07.906811+0000","last_unstale":"2024-09-19T06:49:58.244459+0000","last_undegraded":"2024-09-19T06:49:58.244459+0000","last_fullsized":"2024-09-19T06:49:58.244459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:58:19.184424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189874+0000","last_change":"2024-09-19T06:49:20.429645+0000","last_active":"2024-09-19T06:49:52.189874+0000","last_peered":"2024-09-19T06:49:52.189874+0000","last_clean":"2024-09-19T06:49:52.189874+0000","last_became_active":"2024-09-19T06:49:20.429198+0000","last_became_peered":"2024-09-19T06:49:20.429198+0000","last_unstale":"2024-09-19T06:49:52.189874+0000","last_undegraded":"2024-09-19T06:49:52.189874+0000","last_fullsized":"2024-09-19T06:49:52.189874+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:36:42.145928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190178+0000","last_change":"2024-09-19T06:49:10.382223+0000","last_active":"2024-09-19T06:49:52.190178+0000","last_peered":"2024-09-19T06:49:52.190178+0000","last_clean":"2024-09-19T06:49:52.190178+0000","last_became_active":"2024-09-19T06:49:10.381869+0000","last_became_peered":"2024-09-19T06:49:10.381869+0000","last_unstale":"2024-09-19T06:49:52.190178+0000","last_undegraded":"2024-09-19T06:49:52.190178+0000","last_fullsized":"2024-09-19T06:49:52.190178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:23:34.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"93'9","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:59.248344+0000","last_change":"2024-09-19T06:49:07.907812+0000","last_active":"2024-09-19T06:49:59.248344+0000","last_peered":"2024-09-19T06:49:59.248344+0000","last_clean":"2024-09-19T06:49:59.248344+0000","last_became_active":"2024-09-19T06:49:07.907524+0000","last_became_peered":"2024-09-19T06:49:07.907524+0000","last_unstale":"2024-09-19T06:49:59.248344+0000","last_undegraded":"2024-09-19T06:49:59.248344+0000","last_fullsized":"2024-09-19T06:49:59.248344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:26:43.780682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184608+0000","last_change":"2024-09-19T06:49:20.436311+0000","last_active":"2024-09-19T06:49:52.184608+0000","last_peered":"2024-09-19T06:49:52.184608+0000","last_clean":"2024-09-19T06:49:52.184608+0000","last_became_active":"2024-09-19T06:49:20.435936+0000","last_became_peered":"2024-09-19T06:49:20.435936+0000","last_unstale":"2024-09-19T06:49:52.184608+0000","last_undegraded":"2024-09-19T06:49:52.184608+0000","last_fullsized":"2024-09-19T06:49:52.184608+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:54:04.688922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191610+0000","last_change":"2024-09-19T06:49:10.373764+0000","last_active":"2024-09-19T06:49:52.191610+0000","last_peered":"2024-09-19T06:49:52.191610+0000","last_clean":"2024-09-19T06:49:52.191610+0000","last_became_active":"2024-09-19T06:49:10.373324+0000","last_became_peered":"2024-09-19T06:49:10.373324+0000","last_unstale":"2024-09-19T06:49:52.191610+0000","last_undegraded":"2024-09-19T06:49:52.191610+0000","last_fullsized":"2024-09-19T06:49:52.191610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:25:03.898831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'31","reported_seq":105,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:14.274794+0000","last_change":"2024-09-19T06:49:07.907319+0000","last_active":"2024-09-19T06:50:14.274794+0000","last_peered":"2024-09-19T06:50:14.274794+0000","last_clean":"2024-09-19T06:50:14.274794+0000","last_became_active":"2024-09-19T06:49:07.906888+0000","last_became_peered":"2024-09-19T06:49:07.906888+0000","last_unstale":"2024-09-19T06:50:14.274794+0000","last_undegraded":"2024-09-19T06:50:14.274794+0000","last_fullsized":"2024-09-19T06:50:14.274794+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:49.051957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57258,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":75,"num_write":33,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191821+0000","last_change":"2024-09-19T06:49:20.428184+0000","last_active":"2024-09-19T06:49:52.191821+0000","last_peered":"2024-09-19T06:49:52.191821+0000","last_clean":"2024-09-19T06:49:52.191821+0000","last_became_active":"2024-09-19T06:49:20.427631+0000","last_became_peered":"2024-09-19T06:49:20.427631+0000","last_unstale":"2024-09-19T06:49:52.191821+0000","last_undegraded":"2024-09-19T06:49:52.191821+0000","last_fullsized":"2024-09-19T06:49:52.191821+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:14:51.063413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410607+0000","last_change":"2024-09-19T06:49:10.373073+0000","last_active":"2024-09-19T06:49:52.410607+0000","last_peered":"2024-09-19T06:49:52.410607+0000","last_clean":"2024-09-19T06:49:52.410607+0000","last_became_active":"2024-09-19T06:49:10.371611+0000","last_became_peered":"2024-09-19T06:49:10.371611+0000","last_unstale":"2024-09-19T06:49:52.410607+0000","last_undegraded":"2024-09-19T06:49:52.410607+0000","last_fullsized":"2024-09-19T06:49:52.410607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:58.228946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"93'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:57.236475+0000","last_change":"2024-09-19T06:49:08.186128+0000","last_active":"2024-09-19T06:49:57.236475+0000","last_peered":"2024-09-19T06:49:57.236475+0000","last_clean":"2024-09-19T06:49:57.236475+0000","last_became_active":"2024-09-19T06:49:08.185817+0000","last_became_peered":"2024-09-19T06:49:08.185817+0000","last_unstale":"2024-09-19T06:49:57.236475+0000","last_undegraded":"2024-09-19T06:49:57.236475+0000","last_fullsized":"2024-09-19T06:49:57.236475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:59:53.066405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185090+0000","last_change":"2024-09-19T06:49:20.431251+0000","last_active":"2024-09-19T06:49:52.185090+0000","last_peered":"2024-09-19T06:49:52.185090+0000","last_clean":"2024-09-19T06:49:52.185090+0000","last_became_active":"2024-09-19T06:49:20.430709+0000","last_became_peered":"2024-09-19T06:49:20.430709+0000","last_unstale":"2024-09-19T06:49:52.185090+0000","last_undegraded":"2024-09-19T06:49:52.185090+0000","last_fullsized":"2024-09-19T06:49:52.185090+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:55:06.742567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.188717+0000","last_change":"2024-09-19T06:49:10.379045+0000","last_active":"2024-09-19T06:49:52.188717+0000","last_peered":"2024-09-19T06:49:52.188717+0000","last_clean":"2024-09-19T06:49:52.188717+0000","last_became_active":"2024-09-19T06:49:10.378481+0000","last_became_peered":"2024-09-19T06:49:10.378481+0000","last_unstale":"2024-09-19T06:49:52.188717+0000","last_undegraded":"2024-09-19T06:49:52.188717+0000","last_fullsized":"2024-09-19T06:49:52.188717+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:29:55.770670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"93'4","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.244718+0000","last_change":"2024-09-19T06:49:07.910246+0000","last_active":"2024-09-19T06:49:58.244718+0000","last_peered":"2024-09-19T06:49:58.244718+0000","last_clean":"2024-09-19T06:49:58.244718+0000","last_became_active":"2024-09-19T06:49:07.909855+0000","last_became_peered":"2024-09-19T06:49:07.909855+0000","last_unstale":"2024-09-19T06:49:58.244718+0000","last_undegraded":"2024-09-19T06:49:58.244718+0000","last_fullsized":"2024-09-19T06:49:58.244718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:33:24.011418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189372+0000","last_change":"2024-09-19T06:49:20.421847+0000","last_active":"2024-09-19T06:49:52.189372+0000","last_peered":"2024-09-19T06:49:52.189372+0000","last_clean":"2024-09-19T06:49:52.189372+0000","last_became_active":"2024-09-19T06:49:20.421391+0000","last_became_peered":"2024-09-19T06:49:20.421391+0000","last_unstale":"2024-09-19T06:49:52.189372+0000","last_undegraded":"2024-09-19T06:49:52.189372+0000","last_fullsized":"2024-09-19T06:49:52.189372+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:54:24.207872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410917+0000","last_change":"2024-09-19T06:49:10.376029+0000","last_active":"2024-09-19T06:49:52.410917+0000","last_peered":"2024-09-19T06:49:52.410917+0000","last_clean":"2024-09-19T06:49:52.410917+0000","last_became_active":"2024-09-19T06:49:10.374242+0000","last_became_peered":"2024-09-19T06:49:10.374242+0000","last_unstale":"2024-09-19T06:49:52.410917+0000","last_undegraded":"2024-09-19T06:49:52.410917+0000","last_fullsized":"2024-09-19T06:49:52.410917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:59:43.722173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:14.272406+0000","last_change":"2024-09-19T06:49:07.908090+0000","last_active":"2024-09-19T06:50:14.272406+0000","last_peered":"2024-09-19T06:50:14.272406+0000","last_clean":"2024-09-19T06:50:14.272406+0000","last_became_active":"2024-09-19T06:49:07.906933+0000","last_became_peered":"2024-09-19T06:49:07.906933+0000","last_unstale":"2024-09-19T06:50:14.272406+0000","last_undegraded":"2024-09-19T06:50:14.272406+0000","last_fullsized":"2024-09-19T06:50:14.272406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:21.711371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227514+0000","last_change":"2024-09-19T06:49:20.433797+0000","last_active":"2024-09-19T06:49:52.227514+0000","last_peered":"2024-09-19T06:49:52.227514+0000","last_clean":"2024-09-19T06:49:52.227514+0000","last_became_active":"2024-09-19T06:49:20.433419+0000","last_became_peered":"2024-09-19T06:49:20.433419+0000","last_unstale":"2024-09-19T06:49:52.227514+0000","last_undegraded":"2024-09-19T06:49:52.227514+0000","last_fullsized":"2024-09-19T06:49:52.227514+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:18:39.691678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187738+0000","last_change":"2024-09-19T06:49:20.424995+0000","last_active":"2024-09-19T06:49:52.187738+0000","last_peered":"2024-09-19T06:49:52.187738+0000","last_clean":"2024-09-19T06:49:52.187738+0000","last_became_active":"2024-09-19T06:49:20.424589+0000","last_became_peered":"2024-09-19T06:49:20.424589+0000","last_unstale":"2024-09-19T06:49:52.187738+0000","last_undegraded":"2024-09-19T06:49:52.187738+0000","last_fullsized":"2024-09-19T06:49:52.187738+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:17.968571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191637+0000","last_change":"2024-09-19T06:49:10.382157+0000","last_active":"2024-09-19T06:49:52.191637+0000","last_peered":"2024-09-19T06:49:52.191637+0000","last_clean":"2024-09-19T06:49:52.191637+0000","last_became_active":"2024-09-19T06:49:10.381848+0000","last_became_peered":"2024-09-19T06:49:10.381848+0000","last_unstale":"2024-09-19T06:49:52.191637+0000","last_undegraded":"2024-09-19T06:49:52.191637+0000","last_fullsized":"2024-09-19T06:49:52.191637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:10.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"93'3","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:58.244718+0000","last_change":"2024-09-19T06:49:07.906088+0000","last_active":"2024-09-19T06:49:58.244718+0000","last_peered":"2024-09-19T06:49:58.244718+0000","last_clean":"2024-09-19T06:49:58.244718+0000","last_became_active":"2024-09-19T06:49:07.905798+0000","last_became_peered":"2024-09-19T06:49:07.905798+0000","last_unstale":"2024-09-19T06:49:58.244718+0000","last_undegraded":"2024-09-19T06:49:58.244718+0000","last_fullsized":"2024-09-19T06:49:58.244718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:02.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189691+0000","last_change":"2024-09-19T06:49:20.432275+0000","last_active":"2024-09-19T06:49:52.189691+0000","last_peered":"2024-09-19T06:49:52.189691+0000","last_clean":"2024-09-19T06:49:52.189691+0000","last_became_active":"2024-09-19T06:49:20.431951+0000","last_became_peered":"2024-09-19T06:49:20.431951+0000","last_unstale":"2024-09-19T06:49:52.189691+0000","last_undegraded":"2024-09-19T06:49:52.189691+0000","last_fullsized":"2024-09-19T06:49:52.189691+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:38.649557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227622+0000","last_change":"2024-09-19T06:49:10.368061+0000","last_active":"2024-09-19T06:49:52.227622+0000","last_peered":"2024-09-19T06:49:52.227622+0000","last_clean":"2024-09-19T06:49:52.227622+0000","last_became_active":"2024-09-19T06:49:10.367732+0000","last_became_peered":"2024-09-19T06:49:10.367732+0000","last_unstale":"2024-09-19T06:49:52.227622+0000","last_undegraded":"2024-09-19T06:49:52.227622+0000","last_fullsized":"2024-09-19T06:49:52.227622+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:37.623628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"93'3","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:56.226572+0000","last_change":"2024-09-19T06:49:07.910722+0000","last_active":"2024-09-19T06:49:56.226572+0000","last_peered":"2024-09-19T06:49:56.226572+0000","last_clean":"2024-09-19T06:49:56.226572+0000","last_became_active":"2024-09-19T06:49:07.909960+0000","last_became_peered":"2024-09-19T06:49:07.909960+0000","last_unstale":"2024-09-19T06:49:56.226572+0000","last_undegraded":"2024-09-19T06:49:56.226572+0000","last_fullsized":"2024-09-19T06:49:56.226572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:40:18.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"93'7","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:57.236663+0000","last_change":"2024-09-19T06:49:52.411869+0000","last_active":"2024-09-19T06:49:57.236663+0000","last_peered":"2024-09-19T06:49:57.236663+0000","last_clean":"2024-09-19T06:49:57.236663+0000","last_became_active":"2024-09-19T06:49:52.411491+0000","last_became_peered":"2024-09-19T06:49:52.411491+0000","last_unstale":"2024-09-19T06:49:57.236663+0000","last_undegraded":"2024-09-19T06:49:57.236663+0000","last_fullsized":"2024-09-19T06:49:57.236663+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:05:21.222348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.184771+0000","last_change":"2024-09-19T06:49:10.388831+0000","last_active":"2024-09-19T06:49:52.184771+0000","last_peered":"2024-09-19T06:49:52.184771+0000","last_clean":"2024-09-19T06:49:52.184771+0000","last_became_active":"2024-09-19T06:49:10.388556+0000","last_became_peered":"2024-09-19T06:49:10.388556+0000","last_unstale":"2024-09-19T06:49:52.184771+0000","last_undegraded":"2024-09-19T06:49:52.184771+0000","last_fullsized":"2024-09-19T06:49:52.184771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:28:20.319429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190980+0000","last_change":"2024-09-19T06:49:20.424827+0000","last_active":"2024-09-19T06:49:52.190980+0000","last_peered":"2024-09-19T06:49:52.190980+0000","last_clean":"2024-09-19T06:49:52.190980+0000","last_became_active":"2024-09-19T06:49:20.424318+0000","last_became_peered":"2024-09-19T06:49:20.424318+0000","last_unstale":"2024-09-19T06:49:52.190980+0000","last_undegraded":"2024-09-19T06:49:52.190980+0000","last_fullsized":"2024-09-19T06:49:52.190980+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:37.814108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:50:13.271686+0000","last_change":"2024-09-19T06:49:52.188404+0000","last_active":"2024-09-19T06:50:13.271686+0000","last_peered":"2024-09-19T06:50:13.271686+0000","last_clean":"2024-09-19T06:50:13.271686+0000","last_became_active":"2024-09-19T06:49:52.187918+0000","last_became_peered":"2024-09-19T06:49:52.187918+0000","last_unstale":"2024-09-19T06:50:13.271686+0000","last_undegraded":"2024-09-19T06:50:13.271686+0000","last_fullsized":"2024-09-19T06:50:13.271686+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:53:38.209357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190447+0000","last_change":"2024-09-19T06:49:10.374333+0000","last_active":"2024-09-19T06:49:52.190447+0000","last_peered":"2024-09-19T06:49:52.190447+0000","last_clean":"2024-09-19T06:49:52.190447+0000","last_became_active":"2024-09-19T06:49:10.373944+0000","last_became_peered":"2024-09-19T06:49:10.373944+0000","last_unstale":"2024-09-19T06:49:52.190447+0000","last_undegraded":"2024-09-19T06:49:52.190447+0000","last_fullsized":"2024-09-19T06:49:52.190447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:01:43.609310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190950+0000","last_change":"2024-09-19T06:49:20.417030+0000","last_active":"2024-09-19T06:49:52.190950+0000","last_peered":"2024-09-19T06:49:52.190950+0000","last_clean":"2024-09-19T06:49:52.190950+0000","last_became_active":"2024-09-19T06:49:20.416620+0000","last_became_peered":"2024-09-19T06:49:20.416620+0000","last_unstale":"2024-09-19T06:49:52.190950+0000","last_undegraded":"2024-09-19T06:49:52.190950+0000","last_fullsized":"2024-09-19T06:49:52.190950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:47:18.177613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189469+0000","last_change":"2024-09-19T06:49:10.372503+0000","last_active":"2024-09-19T06:49:52.189469+0000","last_peered":"2024-09-19T06:49:52.189469+0000","last_clean":"2024-09-19T06:49:52.189469+0000","last_became_active":"2024-09-19T06:49:10.372201+0000","last_became_peered":"2024-09-19T06:49:10.372201+0000","last_unstale":"2024-09-19T06:49:52.189469+0000","last_undegraded":"2024-09-19T06:49:52.189469+0000","last_fullsized":"2024-09-19T06:49:52.189469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:31:44.270457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.192797+0000","last_change":"2024-09-19T06:49:10.374149+0000","last_active":"2024-09-19T06:49:52.192797+0000","last_peered":"2024-09-19T06:49:52.192797+0000","last_clean":"2024-09-19T06:49:52.192797+0000","last_became_active":"2024-09-19T06:49:10.372570+0000","last_became_peered":"2024-09-19T06:49:10.372570+0000","last_unstale":"2024-09-19T06:49:52.192797+0000","last_undegraded":"2024-09-19T06:49:52.192797+0000","last_fullsized":"2024-09-19T06:49:52.192797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:55:56.086106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190791+0000","last_change":"2024-09-19T06:49:10.373383+0000","last_active":"2024-09-19T06:49:52.190791+0000","last_peered":"2024-09-19T06:49:52.190791+0000","last_clean":"2024-09-19T06:49:52.190791+0000","last_became_active":"2024-09-19T06:49:10.372144+0000","last_became_peered":"2024-09-19T06:49:10.372144+0000","last_unstale":"2024-09-19T06:49:52.190791+0000","last_undegraded":"2024-09-19T06:49:52.190791+0000","last_fullsized":"2024-09-19T06:49:52.190791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:08:34.000815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187118+0000","last_change":"2024-09-19T06:49:10.379720+0000","last_active":"2024-09-19T06:49:52.187118+0000","last_peered":"2024-09-19T06:49:52.187118+0000","last_clean":"2024-09-19T06:49:52.187118+0000","last_became_active":"2024-09-19T06:49:10.379325+0000","last_became_peered":"2024-09-19T06:49:10.379325+0000","last_unstale":"2024-09-19T06:49:52.187118+0000","last_undegraded":"2024-09-19T06:49:52.187118+0000","last_fullsized":"2024-09-19T06:49:52.187118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:17:58.471617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.194997+0000","last_change":"2024-09-19T06:49:52.194997+0000","last_active":"2024-09-19T06:49:52.194997+0000","last_peered":"2024-09-19T06:49:52.194997+0000","last_clean":"2024-09-19T06:49:52.194997+0000","last_became_active":"2024-09-19T06:49:52.194662+0000","last_became_peered":"2024-09-19T06:49:52.194662+0000","last_unstale":"2024-09-19T06:49:52.194997+0000","last_undegraded":"2024-09-19T06:49:52.194997+0000","last_fullsized":"2024-09-19T06:49:52.194997+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:18.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190025+0000","last_change":"2024-09-19T06:49:10.375868+0000","last_active":"2024-09-19T06:49:52.190025+0000","last_peered":"2024-09-19T06:49:52.190025+0000","last_clean":"2024-09-19T06:49:52.190025+0000","last_became_active":"2024-09-19T06:49:10.375552+0000","last_became_peered":"2024-09-19T06:49:10.375552+0000","last_unstale":"2024-09-19T06:49:52.190025+0000","last_undegraded":"2024-09-19T06:49:52.190025+0000","last_fullsized":"2024-09-19T06:49:52.190025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:55:22.292330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191333+0000","last_change":"2024-09-19T06:49:10.365959+0000","last_active":"2024-09-19T06:49:52.191333+0000","last_peered":"2024-09-19T06:49:52.191333+0000","last_clean":"2024-09-19T06:49:52.191333+0000","last_became_active":"2024-09-19T06:49:10.365018+0000","last_became_peered":"2024-09-19T06:49:10.365018+0000","last_unstale":"2024-09-19T06:49:52.191333+0000","last_undegraded":"2024-09-19T06:49:52.191333+0000","last_fullsized":"2024-09-19T06:49:52.191333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:45:50.417270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.228424+0000","last_change":"2024-09-19T06:49:10.372988+0000","last_active":"2024-09-19T06:49:52.228424+0000","last_peered":"2024-09-19T06:49:52.228424+0000","last_clean":"2024-09-19T06:49:52.228424+0000","last_became_active":"2024-09-19T06:49:10.368995+0000","last_became_peered":"2024-09-19T06:49:10.368995+0000","last_unstale":"2024-09-19T06:49:52.228424+0000","last_undegraded":"2024-09-19T06:49:52.228424+0000","last_fullsized":"2024-09-19T06:49:52.228424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:29:04.939254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.229096+0000","last_change":"2024-09-19T06:49:52.228705+0000","last_active":"2024-09-19T06:49:52.229096+0000","last_peered":"2024-09-19T06:49:52.229096+0000","last_clean":"2024-09-19T06:49:52.229096+0000","last_became_active":"2024-09-19T06:49:52.228371+0000","last_became_peered":"2024-09-19T06:49:52.228371+0000","last_unstale":"2024-09-19T06:49:52.229096+0000","last_undegraded":"2024-09-19T06:49:52.229096+0000","last_fullsized":"2024-09-19T06:49:52.229096+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:24:58.959892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227671+0000","last_change":"2024-09-19T06:49:10.372657+0000","last_active":"2024-09-19T06:49:52.227671+0000","last_peered":"2024-09-19T06:49:52.227671+0000","last_clean":"2024-09-19T06:49:52.227671+0000","last_became_active":"2024-09-19T06:49:10.371641+0000","last_became_peered":"2024-09-19T06:49:10.371641+0000","last_unstale":"2024-09-19T06:49:52.227671+0000","last_undegraded":"2024-09-19T06:49:52.227671+0000","last_fullsized":"2024-09-19T06:49:52.227671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:54:09.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410805+0000","last_change":"2024-09-19T06:49:10.370765+0000","last_active":"2024-09-19T06:49:52.410805+0000","last_peered":"2024-09-19T06:49:52.410805+0000","last_clean":"2024-09-19T06:49:52.410805+0000","last_became_active":"2024-09-19T06:49:10.369741+0000","last_became_peered":"2024-09-19T06:49:10.369741+0000","last_unstale":"2024-09-19T06:49:52.410805+0000","last_undegraded":"2024-09-19T06:49:52.410805+0000","last_fullsized":"2024-09-19T06:49:52.410805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:35:31.377034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189842+0000","last_change":"2024-09-19T06:49:10.386328+0000","last_active":"2024-09-19T06:49:52.189842+0000","last_peered":"2024-09-19T06:49:52.189842+0000","last_clean":"2024-09-19T06:49:52.189842+0000","last_became_active":"2024-09-19T06:49:10.385970+0000","last_became_peered":"2024-09-19T06:49:10.385970+0000","last_unstale":"2024-09-19T06:49:52.189842+0000","last_undegraded":"2024-09-19T06:49:52.189842+0000","last_fullsized":"2024-09-19T06:49:52.189842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:52:27.404825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190032+0000","last_change":"2024-09-19T06:49:10.372047+0000","last_active":"2024-09-19T06:49:52.190032+0000","last_peered":"2024-09-19T06:49:52.190032+0000","last_clean":"2024-09-19T06:49:52.190032+0000","last_became_active":"2024-09-19T06:49:10.371743+0000","last_became_peered":"2024-09-19T06:49:10.371743+0000","last_unstale":"2024-09-19T06:49:52.190032+0000","last_undegraded":"2024-09-19T06:49:52.190032+0000","last_fullsized":"2024-09-19T06:49:52.190032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:06.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191151+0000","last_change":"2024-09-19T06:49:10.373221+0000","last_active":"2024-09-19T06:49:52.191151+0000","last_peered":"2024-09-19T06:49:52.191151+0000","last_clean":"2024-09-19T06:49:52.191151+0000","last_became_active":"2024-09-19T06:49:10.372859+0000","last_became_peered":"2024-09-19T06:49:10.372859+0000","last_unstale":"2024-09-19T06:49:52.191151+0000","last_undegraded":"2024-09-19T06:49:52.191151+0000","last_fullsized":"2024-09-19T06:49:52.191151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:53:19.333758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.191568+0000","last_change":"2024-09-19T06:49:10.371359+0000","last_active":"2024-09-19T06:49:52.191568+0000","last_peered":"2024-09-19T06:49:52.191568+0000","last_clean":"2024-09-19T06:49:52.191568+0000","last_became_active":"2024-09-19T06:49:10.371057+0000","last_became_peered":"2024-09-19T06:49:10.371057+0000","last_unstale":"2024-09-19T06:49:52.191568+0000","last_undegraded":"2024-09-19T06:49:52.191568+0000","last_fullsized":"2024-09-19T06:49:52.191568+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:18:40.387791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190196+0000","last_change":"2024-09-19T06:49:10.381371+0000","last_active":"2024-09-19T06:49:52.190196+0000","last_peered":"2024-09-19T06:49:52.190196+0000","last_clean":"2024-09-19T06:49:52.190196+0000","last_became_active":"2024-09-19T06:49:10.381017+0000","last_became_peered":"2024-09-19T06:49:10.381017+0000","last_unstale":"2024-09-19T06:49:52.190196+0000","last_undegraded":"2024-09-19T06:49:52.190196+0000","last_fullsized":"2024-09-19T06:49:52.190196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:17:54.998597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.410189+0000","last_change":"2024-09-19T06:49:10.377596+0000","last_active":"2024-09-19T06:49:52.410189+0000","last_peered":"2024-09-19T06:49:52.410189+0000","last_clean":"2024-09-19T06:49:52.410189+0000","last_became_active":"2024-09-19T06:49:10.377271+0000","last_became_peered":"2024-09-19T06:49:10.377271+0000","last_unstale":"2024-09-19T06:49:52.410189+0000","last_undegraded":"2024-09-19T06:49:52.410189+0000","last_fullsized":"2024-09-19T06:49:52.410189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:32:00.797866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.194675+0000","last_change":"2024-09-19T06:49:52.194675+0000","last_active":"2024-09-19T06:49:52.194675+0000","last_peered":"2024-09-19T06:49:52.194675+0000","last_clean":"2024-09-19T06:49:52.194675+0000","last_became_active":"2024-09-19T06:49:52.194007+0000","last_became_peered":"2024-09-19T06:49:52.194007+0000","last_unstale":"2024-09-19T06:49:52.194675+0000","last_undegraded":"2024-09-19T06:49:52.194675+0000","last_fullsized":"2024-09-19T06:49:52.194675+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:45:33.109454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.186170+0000","last_change":"2024-09-19T06:49:10.387416+0000","last_active":"2024-09-19T06:49:52.186170+0000","last_peered":"2024-09-19T06:49:52.186170+0000","last_clean":"2024-09-19T06:49:52.186170+0000","last_became_active":"2024-09-19T06:49:10.384083+0000","last_became_peered":"2024-09-19T06:49:10.384083+0000","last_unstale":"2024-09-19T06:49:52.186170+0000","last_undegraded":"2024-09-19T06:49:52.186170+0000","last_fullsized":"2024-09-19T06:49:52.186170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:07:05.691663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189758+0000","last_change":"2024-09-19T06:49:10.375636+0000","last_active":"2024-09-19T06:49:52.189758+0000","last_peered":"2024-09-19T06:49:52.189758+0000","last_clean":"2024-09-19T06:49:52.189758+0000","last_became_active":"2024-09-19T06:49:10.375227+0000","last_became_peered":"2024-09-19T06:49:10.375227+0000","last_unstale":"2024-09-19T06:49:52.189758+0000","last_undegraded":"2024-09-19T06:49:52.189758+0000","last_fullsized":"2024-09-19T06:49:52.189758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:41:59.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185687+0000","last_change":"2024-09-19T06:49:10.383453+0000","last_active":"2024-09-19T06:49:52.185687+0000","last_peered":"2024-09-19T06:49:52.185687+0000","last_clean":"2024-09-19T06:49:52.185687+0000","last_became_active":"2024-09-19T06:49:10.383075+0000","last_became_peered":"2024-09-19T06:49:10.383075+0000","last_unstale":"2024-09-19T06:49:52.185687+0000","last_undegraded":"2024-09-19T06:49:52.185687+0000","last_fullsized":"2024-09-19T06:49:52.185687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:18:24.775578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187494+0000","last_change":"2024-09-19T06:49:10.378733+0000","last_active":"2024-09-19T06:49:52.187494+0000","last_peered":"2024-09-19T06:49:52.187494+0000","last_clean":"2024-09-19T06:49:52.187494+0000","last_became_active":"2024-09-19T06:49:10.378231+0000","last_became_peered":"2024-09-19T06:49:10.378231+0000","last_unstale":"2024-09-19T06:49:52.187494+0000","last_undegraded":"2024-09-19T06:49:52.187494+0000","last_fullsized":"2024-09-19T06:49:52.187494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:53:14.959853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190301+0000","last_change":"2024-09-19T06:49:10.379775+0000","last_active":"2024-09-19T06:49:52.190301+0000","last_peered":"2024-09-19T06:49:52.190301+0000","last_clean":"2024-09-19T06:49:52.190301+0000","last_became_active":"2024-09-19T06:49:10.379130+0000","last_became_peered":"2024-09-19T06:49:10.379130+0000","last_unstale":"2024-09-19T06:49:52.190301+0000","last_undegraded":"2024-09-19T06:49:52.190301+0000","last_fullsized":"2024-09-19T06:49:52.190301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:05.914354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.411100+0000","last_change":"2024-09-19T06:49:10.376239+0000","last_active":"2024-09-19T06:49:52.411100+0000","last_peered":"2024-09-19T06:49:52.411100+0000","last_clean":"2024-09-19T06:49:52.411100+0000","last_became_active":"2024-09-19T06:49:10.375913+0000","last_became_peered":"2024-09-19T06:49:10.375913+0000","last_unstale":"2024-09-19T06:49:52.411100+0000","last_undegraded":"2024-09-19T06:49:52.411100+0000","last_fullsized":"2024-09-19T06:49:52.411100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:14:26.373766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.227710+0000","last_change":"2024-09-19T06:49:10.375479+0000","last_active":"2024-09-19T06:49:52.227710+0000","last_peered":"2024-09-19T06:49:52.227710+0000","last_clean":"2024-09-19T06:49:52.227710+0000","last_became_active":"2024-09-19T06:49:10.375201+0000","last_became_peered":"2024-09-19T06:49:10.375201+0000","last_unstale":"2024-09-19T06:49:52.227710+0000","last_undegraded":"2024-09-19T06:49:52.227710+0000","last_fullsized":"2024-09-19T06:49:52.227710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:44:56.421410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.186709+0000","last_change":"2024-09-19T06:49:10.374495+0000","last_active":"2024-09-19T06:49:52.186709+0000","last_peered":"2024-09-19T06:49:52.186709+0000","last_clean":"2024-09-19T06:49:52.186709+0000","last_became_active":"2024-09-19T06:49:10.374110+0000","last_became_peered":"2024-09-19T06:49:10.374110+0000","last_unstale":"2024-09-19T06:49:52.186709+0000","last_undegraded":"2024-09-19T06:49:52.186709+0000","last_fullsized":"2024-09-19T06:49:52.186709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:51:27.641785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189468+0000","last_change":"2024-09-19T06:49:10.373548+0000","last_active":"2024-09-19T06:49:52.189468+0000","last_peered":"2024-09-19T06:49:52.189468+0000","last_clean":"2024-09-19T06:49:52.189468+0000","last_became_active":"2024-09-19T06:49:10.373251+0000","last_became_peered":"2024-09-19T06:49:10.373251+0000","last_unstale":"2024-09-19T06:49:52.189468+0000","last_undegraded":"2024-09-19T06:49:52.189468+0000","last_fullsized":"2024-09-19T06:49:52.189468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:59:35.353223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.190651+0000","last_change":"2024-09-19T06:49:10.383000+0000","last_active":"2024-09-19T06:49:52.190651+0000","last_peered":"2024-09-19T06:49:52.190651+0000","last_clean":"2024-09-19T06:49:52.190651+0000","last_became_active":"2024-09-19T06:49:10.382653+0000","last_became_peered":"2024-09-19T06:49:10.382653+0000","last_unstale":"2024-09-19T06:49:52.190651+0000","last_undegraded":"2024-09-19T06:49:52.190651+0000","last_fullsized":"2024-09-19T06:49:52.190651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:31.317042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.185622+0000","last_change":"2024-09-19T06:49:10.380239+0000","last_active":"2024-09-19T06:49:52.185622+0000","last_peered":"2024-09-19T06:49:52.185622+0000","last_clean":"2024-09-19T06:49:52.185622+0000","last_became_active":"2024-09-19T06:49:10.378491+0000","last_became_peered":"2024-09-19T06:49:10.378491+0000","last_unstale":"2024-09-19T06:49:52.185622+0000","last_undegraded":"2024-09-19T06:49:52.185622+0000","last_fullsized":"2024-09-19T06:49:52.185622+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:56:14.016962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187477+0000","last_change":"2024-09-19T06:49:10.376989+0000","last_active":"2024-09-19T06:49:52.187477+0000","last_peered":"2024-09-19T06:49:52.187477+0000","last_clean":"2024-09-19T06:49:52.187477+0000","last_became_active":"2024-09-19T06:49:10.376701+0000","last_became_peered":"2024-09-19T06:49:10.376701+0000","last_unstale":"2024-09-19T06:49:52.187477+0000","last_undegraded":"2024-09-19T06:49:52.187477+0000","last_fullsized":"2024-09-19T06:49:52.187477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:13:34.153487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.187990+0000","last_change":"2024-09-19T06:49:10.382585+0000","last_active":"2024-09-19T06:49:52.187990+0000","last_peered":"2024-09-19T06:49:52.187990+0000","last_clean":"2024-09-19T06:49:52.187990+0000","last_became_active":"2024-09-19T06:49:10.381573+0000","last_became_peered":"2024-09-19T06:49:10.381573+0000","last_unstale":"2024-09-19T06:49:52.187990+0000","last_undegraded":"2024-09-19T06:49:52.187990+0000","last_fullsized":"2024-09-19T06:49:52.187990+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:29:08.324309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-19T06:49:52.189975+0000","last_change":"2024-09-19T06:49:10.381681+0000","last_active":"2024-09-19T06:49:52.189975+0000","last_peered":"2024-09-19T06:49:52.189975+0000","last_clean":"2024-09-19T06:49:52.189975+0000","last_became_active":"2024-09-19T06:49:10.381344+0000","last_became_peered":"2024-09-19T06:49:10.381344+0000","last_unstale":"2024-09-19T06:49:52.189975+0000","last_undegraded":"2024-09-19T06:49:52.189975+0000","last_fullsized":"2024-09-19T06:49:52.189975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:39:01.539679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64073,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":196,"num_read_kb":171,"num_write":243,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":192219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215885,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449114,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":522313,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":71,"seq":304942678046,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1171456,"data_stored":1166668,"data_compressed":59753,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841573,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1056768,"data_stored":526227,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1597,"internal_metadata":31914435},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103210,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":522397,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1119008,"data_compressed":59753,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495667,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":517635,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757307,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":1114197,"data_compressed":59753,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":522836,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182468,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":517343,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":562634,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":575923,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":519091,"data_compressed":23497,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34605,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":59010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":58496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":59387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T06:50:19.196 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-19T06:50:19.196 INFO:tasks.cephadm:Setting up client nodes... 2024-09-19T06:50:19.197 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-19T06:50:19.567 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 -- 172.21.15.114:0/2094337991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 msgr2=0x7f3178104200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 --2- 172.21.15.114:0/2094337991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 0x7f3178104200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3168009900 tx=0x7f316802a400 comp rx=0 tx=0).stop 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 -- 172.21.15.114:0/2094337991 shutdown_connections 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 --2- 172.21.15.114:0/2094337991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3178105920 0x7f317810c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 --2- 172.21.15.114:0/2094337991 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3178104f60 0x7f31781053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 --2- 172.21.15.114:0/2094337991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 0x7f3178104200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 -- 172.21.15.114:0/2094337991 >> 172.21.15.114:0/2094337991 conn(0x7f31780ff5b0 msgr2=0x7f31781019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 -- 172.21.15.114:0/2094337991 shutdown_connections 2024-09-19T06:50:20.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.138+0000 7f317de00640 1 -- 172.21.15.114:0/2094337991 wait complete. 2024-09-19T06:50:20.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.139+0000 7f317de00640 1 Processor -- start 2024-09-19T06:50:20.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.139+0000 7f317de00640 1 -- start start 2024-09-19T06:50:20.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.139+0000 7f317de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 0x7f317819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:20.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.139+0000 7f317de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3178104f60 0x7f317819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:20.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f317de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3178105920 0x7f31781a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f317de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f317810fab0 con 0x7f3178103e00 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f317de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f317810f930 con 0x7f3178104f60 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f317de00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f317810fc30 con 0x7f3178105920 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f3176a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 0x7f317819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f316e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3178104f60 0x7f317819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f3176a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 0x7f317819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36918/0 (socket says 172.21.15.114:36918) 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f316e000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3178104f60 0x7f317819a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:42886/0 (socket says 172.21.15.114:42886) 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f3176a00640 1 -- 172.21.15.114:0/2449793018 learned_addr learned my addr 172.21.15.114:0/2449793018 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f3177400640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3178105920 0x7f31781a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f316e000640 1 -- 172.21.15.114:0/2449793018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3178105920 msgr2=0x7f31781a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f316e000640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3178105920 0x7f31781a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f316e000640 1 -- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 msgr2=0x7f317819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:20.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f316e000640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 0x7f317819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.140+0000 7f316e000640 1 -- 172.21.15.114:0/2449793018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31781a1d30 con 0x7f3178104f60 2024-09-19T06:50:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.141+0000 7f316e000640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3178104f60 0x7f317819a640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f316000e9c0 tx=0x7f316000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.141+0000 7f317ca00640 1 -- 172.21.15.114:0/2449793018 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316000ccc0 con 0x7f3178104f60 2024-09-19T06:50:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.141+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31781a1fc0 con 0x7f3178104f60 2024-09-19T06:50:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.141+0000 7f317ca00640 1 -- 172.21.15.114:0/2449793018 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31600035f0 con 0x7f3178104f60 2024-09-19T06:50:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.141+0000 7f317ca00640 1 -- 172.21.15.114:0/2449793018 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316000ce60 con 0x7f3178104f60 2024-09-19T06:50:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.141+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31781a24d0 con 0x7f3178104f60 2024-09-19T06:50:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.142+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f317810b240 con 0x7f3178104f60 2024-09-19T06:50:20.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.144+0000 7f317ca00640 1 -- 172.21.15.114:0/2449793018 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f31600037a0 con 0x7f3178104f60 2024-09-19T06:50:20.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.145+0000 7f317ca00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f315c0a3380 0x7f315c0a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:20.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.146+0000 7f317ca00640 1 -- 172.21.15.114:0/2449793018 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f31600d0280 con 0x7f3178104f60 2024-09-19T06:50:20.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.146+0000 7f3176a00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f315c0a3380 0x7f315c0a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:20.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.146+0000 7f3176a00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f315c0a3380 0x7f315c0a5840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3168031440 tx=0x7f31680315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:20.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.148+0000 7f317ca00640 1 -- 172.21.15.114:0/2449793018 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f316009ca80 con 0x7f3178104f60 2024-09-19T06:50:20.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.387+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f317802d9c0 con 0x7f3178104f60 2024-09-19T06:50:20.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.393+0000 7f317ca00640 1 -- 172.21.15.114:0/2449793018 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f316009c380 con 0x7f3178104f60 2024-09-19T06:50:20.394 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-09-19T06:50:20.394 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQCsyetmH6M6FxAAg2Bsl0LRsK9pZYYyl0lO4g== 2024-09-19T06:50:20.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f315c0a3380 msgr2=0x7f315c0a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:20.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f315c0a3380 0x7f315c0a5840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3168031440 tx=0x7f31680315b0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3178104f60 msgr2=0x7f317819a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3178104f60 0x7f317819a640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f316000e9c0 tx=0x7f316000edc0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 shutdown_connections 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f315c0a3380 0x7f315c0a5840 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3178105920 0x7f31781a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3178104f60 0x7f317819a640 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 --2- 172.21.15.114:0/2449793018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3178103e00 0x7f317819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 >> 172.21.15.114:0/2449793018 conn(0x7f31780ff5b0 msgr2=0x7f3178108410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:20.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 shutdown_connections 2024-09-19T06:50:20.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:20.395+0000 7f317de00640 1 -- 172.21.15.114:0/2449793018 wait complete. 2024-09-19T06:50:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:20 smithi114 ceph-mon[30820]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:50:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:20 smithi114 ceph-mon[30820]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-09-19T06:50:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:20 smithi116 ceph-mon[35411]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:50:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:20 smithi116 ceph-mon[35411]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-09-19T06:50:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:20 smithi118 ceph-mon[35527]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:50:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:20 smithi118 ceph-mon[35527]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.3 KiB/s wr, 11 op/s 2024-09-19T06:50:21.165 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:50:21.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-19T06:50:21.165 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-19T06:50:21.246 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-19T06:50:21.246 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-19T06:50:21.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-19T06:50:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:21 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2449793018' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T06:50:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:21 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T06:50:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:21 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-19T06:50:21.653 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:21 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2449793018' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T06:50:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:21 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T06:50:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:21 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-19T06:50:21.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:21 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2449793018' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T06:50:21.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:21 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-19T06:50:21.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:21 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-19T06:50:22.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 -- 172.21.15.114:0/2620332676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d88104010 msgr2=0x7f2d881043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:22.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 --2- 172.21.15.114:0/2620332676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d88104010 0x7f2d881043f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009860 tx=0x7f2d7c02a340 comp rx=0 tx=0).stop 2024-09-19T06:50:22.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 -- 172.21.15.114:0/2620332676 shutdown_connections 2024-09-19T06:50:22.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 --2- 172.21.15.114:0/2620332676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d88109140 0x7f2d88110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 --2- 172.21.15.114:0/2620332676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d881049c0 0x7f2d88108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 --2- 172.21.15.114:0/2620332676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d88104010 0x7f2d881043f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 -- 172.21.15.114:0/2620332676 >> 172.21.15.114:0/2620332676 conn(0x7f2d880ffa50 msgr2=0x7f2d88101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:22.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 -- 172.21.15.114:0/2620332676 shutdown_connections 2024-09-19T06:50:22.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.385+0000 7f2d90800640 1 -- 172.21.15.114:0/2620332676 wait complete. 2024-09-19T06:50:22.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.386+0000 7f2d90800640 1 Processor -- start 2024-09-19T06:50:22.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d90800640 1 -- start start 2024-09-19T06:50:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d90800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d88104010 0x7f2d8819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d90800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d881049c0 0x7f2d8819ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d90800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d88109140 0x7f2d8819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d90800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2d88114160 con 0x7f2d88109140 2024-09-19T06:50:22.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d90800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d88113fe0 con 0x7f2d881049c0 2024-09-19T06:50:22.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d90800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2d881142e0 con 0x7f2d88104010 2024-09-19T06:50:22.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d8d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d881049c0 0x7f2d8819ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:22.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d8d400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d881049c0 0x7f2d8819ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:42904/0 (socket says 172.21.15.114:42904) 2024-09-19T06:50:22.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d8d400640 1 -- 172.21.15.114:0/705832996 learned_addr learned my addr 172.21.15.114:0/705832996 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:22.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.387+0000 7f2d8de00640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d88104010 0x7f2d8819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:22.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.388+0000 7f2d8e800640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d88109140 0x7f2d8819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:22.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.388+0000 7f2d8d400640 1 -- 172.21.15.114:0/705832996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d88104010 msgr2=0x7f2d8819a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:22.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.388+0000 7f2d8d400640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d88104010 0x7f2d8819a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.388+0000 7f2d8d400640 1 -- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d88109140 msgr2=0x7f2d8819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:22.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.388+0000 7f2d8d400640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d88109140 0x7f2d8819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.388+0000 7f2d8d400640 1 -- 172.21.15.114:0/705832996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8819f5f0 con 0x7f2d881049c0 2024-09-19T06:50:22.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.388+0000 7f2d8d400640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d881049c0 0x7f2d8819ab80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800e9f0 tx=0x7f2d7800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:22.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.388+0000 7f2d82a00640 1 -- 172.21.15.114:0/705832996 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7800ccc0 con 0x7f2d881049c0 2024-09-19T06:50:22.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.389+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8819f8e0 con 0x7f2d881049c0 2024-09-19T06:50:22.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.389+0000 7f2d82a00640 1 -- 172.21.15.114:0/705832996 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d78005500 con 0x7f2d881049c0 2024-09-19T06:50:22.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.389+0000 7f2d82a00640 1 -- 172.21.15.114:0/705832996 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78010690 con 0x7f2d881049c0 2024-09-19T06:50:22.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.389+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d881a71c0 con 0x7f2d881049c0 2024-09-19T06:50:22.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.390+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d881043f0 con 0x7f2d881049c0 2024-09-19T06:50:22.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.392+0000 7f2d82a00640 1 -- 172.21.15.114:0/705832996 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f2d780046c0 con 0x7f2d881049c0 2024-09-19T06:50:22.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.393+0000 7f2d82a00640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d700a32d0 0x7f2d700a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:22.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.394+0000 7f2d82a00640 1 -- 172.21.15.114:0/705832996 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2d780d03f0 con 0x7f2d881049c0 2024-09-19T06:50:22.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.394+0000 7f2d8de00640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d700a32d0 0x7f2d700a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:22.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.394+0000 7f2d8de00640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d700a32d0 0x7f2d700a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c02a930 tx=0x7f2d7c0056d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:22.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.396+0000 7f2d82a00640 1 -- 172.21.15.114:0/705832996 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2d7809cbf0 con 0x7f2d881049c0 2024-09-19T06:50:22.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.618+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f2d8819b9a0 con 0x7f2d881049c0 2024-09-19T06:50:22.620 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:22 smithi114 ceph-mon[30820]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.0 KiB/s wr, 14 op/s 2024-09-19T06:50:22.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.623+0000 7f2d82a00640 1 -- 172.21.15.114:0/705832996 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264058 (secure 0 0 0) 0x7f2d8819b9a0 con 0x7f2d881049c0 2024-09-19T06:50:22.630 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:50:22.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d700a32d0 msgr2=0x7f2d700a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:22.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d700a32d0 0x7f2d700a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c02a930 tx=0x7f2d7c0056d0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d881049c0 msgr2=0x7f2d8819ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:22.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d881049c0 0x7f2d8819ab80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800e9f0 tx=0x7f2d7800eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 shutdown_connections 2024-09-19T06:50:22.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2d700a32d0 0x7f2d700a5790 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d88109140 0x7f2d8819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d881049c0 0x7f2d8819ab80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 --2- 172.21.15.114:0/705832996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d88104010 0x7f2d8819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:22.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 >> 172.21.15.114:0/705832996 conn(0x7f2d880ffa50 msgr2=0x7f2d88100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:22.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 shutdown_connections 2024-09-19T06:50:22.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:22.627+0000 7f2d90800640 1 -- 172.21.15.114:0/705832996 wait complete. 2024-09-19T06:50:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:22 smithi116 ceph-mon[35411]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.0 KiB/s wr, 14 op/s 2024-09-19T06:50:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:22 smithi118 ceph-mon[35527]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 4.0 KiB/s wr, 14 op/s 2024-09-19T06:50:23.418 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3586930729},{"type":"v1","addr":"172.21.15.114:6801","nonce":3586930729}]},"active_addr":"172.21.15.114:6801/3586930729","active_change":"2024-09-19T06:41:50.994674+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2845998069}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2470917878}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2992591613}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":666492271}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1595452022}]}]} 2024-09-19T06:50:23.425 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-19T06:50:23.425 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-19T06:50:23.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:50:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:23 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/705832996' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-19T06:50:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:23 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/705832996' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-19T06:50:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:23 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/705832996' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-19T06:50:23.790 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:24.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 -- 172.21.15.114:0/2942253587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0103e30 msgr2=0x7f20f0104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:24.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 --2- 172.21.15.114:0/2942253587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0103e30 0x7f20f0104210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009900 tx=0x7f20e002a410 comp rx=0 tx=0).stop 2024-09-19T06:50:24.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 -- 172.21.15.114:0/2942253587 shutdown_connections 2024-09-19T06:50:24.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 --2- 172.21.15.114:0/2942253587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20f0108f60 0x7f20f0110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 --2- 172.21.15.114:0/2942253587 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20f01047e0 0x7f20f0108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 --2- 172.21.15.114:0/2942253587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0103e30 0x7f20f0104210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 -- 172.21.15.114:0/2942253587 >> 172.21.15.114:0/2942253587 conn(0x7f20f00ff8d0 msgr2=0x7f20f0101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 -- 172.21.15.114:0/2942253587 shutdown_connections 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.357+0000 7f20f4e00640 1 -- 172.21.15.114:0/2942253587 wait complete. 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.358+0000 7f20f4e00640 1 Processor -- start 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.358+0000 7f20f4e00640 1 -- start start 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.358+0000 7f20f4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20f0103e30 0x7f20f019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.358+0000 7f20f4e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20f01047e0 0x7f20f019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.358+0000 7f20f4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0108f60 0x7f20f019ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.358+0000 7f20f4e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0114010 con 0x7f20f0108f60 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.358+0000 7f20f4e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0113e90 con 0x7f20f0103e30 2024-09-19T06:50:24.379 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.358+0000 7f20f4e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0114190 con 0x7f20f01047e0 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0108f60 0x7f20f019ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0108f60 0x7f20f019ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36960/0 (socket says 172.21.15.114:36960) 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 -- 172.21.15.114:0/61362208 learned_addr learned my addr 172.21.15.114:0/61362208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20e7e00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20f01047e0 0x7f20f019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20ef400640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20f0103e30 0x7f20f019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 -- 172.21.15.114:0/61362208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20f01047e0 msgr2=0x7f20f019a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20f01047e0 0x7f20f019a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 -- 172.21.15.114:0/61362208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20f0103e30 msgr2=0x7f20f019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20f0103e30 0x7f20f019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 -- 172.21.15.114:0/61362208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f019f400 con 0x7f20f0108f60 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.359+0000 7f20efe00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0108f60 0x7f20f019ed20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f20e800eb10 tx=0x7f20e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.360+0000 7f20ecc00640 1 -- 172.21.15.114:0/61362208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e8004b20 con 0x7f20f0108f60 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.360+0000 7f20ecc00640 1 -- 172.21.15.114:0/61362208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20e8004cc0 con 0x7f20f0108f60 2024-09-19T06:50:24.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.360+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f019f6f0 con 0x7f20f0108f60 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.360+0000 7f20ecc00640 1 -- 172.21.15.114:0/61362208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e8010760 con 0x7f20f0108f60 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.360+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f01a6fd0 con 0x7f20f0108f60 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.361+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f0104210 con 0x7f20f0108f60 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.363+0000 7f20ecc00640 1 -- 172.21.15.114:0/61362208 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f20e8005020 con 0x7f20f0108f60 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.364+0000 7f20ecc00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f20d40a3350 0x7f20d40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.364+0000 7f20ecc00640 1 -- 172.21.15.114:0/61362208 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f20e80d02f0 con 0x7f20f0108f60 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.365+0000 7f20ef400640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f20d40a3350 0x7f20d40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.365+0000 7f20ef400640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f20d40a3350 0x7f20d40a5810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000c00 tx=0x7f20e0005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:24.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.369+0000 7f20ecc00640 1 -- 172.21.15.114:0/61362208 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f20e809caf0 con 0x7f20f0108f60 2024-09-19T06:50:24.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.551+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20f010bbb0 con 0x7f20f0108f60 2024-09-19T06:50:24.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:24 smithi114 ceph-mon[30820]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-19T06:50:24.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.553+0000 7f20ecc00640 1 -- 172.21.15.114:0/61362208 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24854 (secure 0 0 0) 0x7f20e809c3f0 con 0x7f20f0108f60 2024-09-19T06:50:24.555 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:50:24.555 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":93,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:49:52.180171+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:50:24.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f20d40a3350 msgr2=0x7f20d40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:24.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f20d40a3350 0x7f20d40a5810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000c00 tx=0x7f20e0005b50 comp rx=0 tx=0).stop 2024-09-19T06:50:24.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0108f60 msgr2=0x7f20f019ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:24.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0108f60 0x7f20f019ed20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f20e800eb10 tx=0x7f20e800c2d0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 shutdown_connections 2024-09-19T06:50:24.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f20d40a3350 0x7f20d40a5810 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20f0108f60 0x7f20f019ed20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20f01047e0 0x7f20f019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 --2- 172.21.15.114:0/61362208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20f0103e30 0x7f20f019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:24.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 >> 172.21.15.114:0/61362208 conn(0x7f20f00ff8d0 msgr2=0x7f20f0100b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:24.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 shutdown_connections 2024-09-19T06:50:24.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:24.556+0000 7f20f4e00640 1 -- 172.21.15.114:0/61362208 wait complete. 2024-09-19T06:50:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:24 smithi116 ceph-mon[35411]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-19T06:50:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:24 smithi118 ceph-mon[35527]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-19T06:50:25.315 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-19T06:50:25.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:50:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:25 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/61362208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:25 smithi114 ceph-mon[30820]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-19T06:50:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:25 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/61362208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:25 smithi116 ceph-mon[35411]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-19T06:50:25.681 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:25 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/61362208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:25 smithi118 ceph-mon[35527]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-19T06:50:26.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 -- 172.21.15.114:0/2425074127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0101de0 msgr2=0x7f76b01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:26.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2425074127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0101de0 0x7f76b01021c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f76a0009900 tx=0x7f76a002a410 comp rx=0 tx=0).stop 2024-09-19T06:50:26.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 -- 172.21.15.114:0/2425074127 shutdown_connections 2024-09-19T06:50:26.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2425074127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76b0106f10 0x7f76b010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2425074127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76b0102790 0x7f76b01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2425074127 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0101de0 0x7f76b01021c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 -- 172.21.15.114:0/2425074127 >> 172.21.15.114:0/2425074127 conn(0x7f76b00fd8c0 msgr2=0x7f76b00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:26.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 -- 172.21.15.114:0/2425074127 shutdown_connections 2024-09-19T06:50:26.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.241+0000 7f76b4e00640 1 -- 172.21.15.114:0/2425074127 wait complete. 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.242+0000 7f76b4e00640 1 Processor -- start 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76b4e00640 1 -- start start 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76b4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76b0101de0 0x7f76b019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76b4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76b0102790 0x7f76b019ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76b4e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0106f10 0x7f76b019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76b4e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f76b0111fc0 con 0x7f76b0102790 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76b4e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76b0111e40 con 0x7f76b0101de0 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76b4e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f76b0112140 con 0x7f76b0106f10 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76a5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76b0102790 0x7f76b019ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:26.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76ade00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76b0101de0 0x7f76b019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76ae800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0106f10 0x7f76b019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76a5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76b0102790 0x7f76b019ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36972/0 (socket says 172.21.15.114:36972) 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76ade00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76b0101de0 0x7f76b019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:42928/0 (socket says 172.21.15.114:42928) 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.243+0000 7f76a5400640 1 -- 172.21.15.114:0/2316929147 learned_addr learned my addr 172.21.15.114:0/2316929147 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76ade00640 1 -- 172.21.15.114:0/2316929147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0106f10 msgr2=0x7f76b019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76ade00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0106f10 0x7f76b019ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76ade00640 1 -- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76b0102790 msgr2=0x7f76b019ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76ade00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76b0102790 0x7f76b019ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76ade00640 1 -- 172.21.15.114:0/2316929147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b019f5f0 con 0x7f76b0101de0 2024-09-19T06:50:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76ae800640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0106f10 0x7f76b019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76ade00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76b0101de0 0x7f76b019a640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f76a002a930 tx=0x7f76a0003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76a7400640 1 -- 172.21.15.114:0/2316929147 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a003f070 con 0x7f76b0101de0 2024-09-19T06:50:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.244+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b019f880 con 0x7f76b0101de0 2024-09-19T06:50:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.245+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b01a71c0 con 0x7f76b0101de0 2024-09-19T06:50:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.245+0000 7f76a7400640 1 -- 172.21.15.114:0/2316929147 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a0031c90 con 0x7f76b0101de0 2024-09-19T06:50:26.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.245+0000 7f76a7400640 1 -- 172.21.15.114:0/2316929147 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a0049060 con 0x7f76b0101de0 2024-09-19T06:50:26.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.246+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76b01021c0 con 0x7f76b0101de0 2024-09-19T06:50:26.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.248+0000 7f76a7400640 1 -- 172.21.15.114:0/2316929147 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f76a0042070 con 0x7f76b0101de0 2024-09-19T06:50:26.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.249+0000 7f76a7400640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76940a3350 0x7f76940a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:26.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.249+0000 7f76a7400640 1 -- 172.21.15.114:0/2316929147 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f76a00f0230 con 0x7f76b0101de0 2024-09-19T06:50:26.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.249+0000 7f76a5400640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76940a3350 0x7f76940a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:26.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.250+0000 7f76a5400640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76940a3350 0x7f76940a5810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f76b00ff6f0 tx=0x7f7698005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:26.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.251+0000 7f76a7400640 1 -- 172.21.15.114:0/2316929147 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f76a00bc3e0 con 0x7f76b0101de0 2024-09-19T06:50:26.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.426+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76b01a7390 con 0x7f76b0101de0 2024-09-19T06:50:26.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.428+0000 7f76a7400640 1 -- 172.21.15.114:0/2316929147 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24854 (secure 0 0 0) 0x7f76a00c06e0 con 0x7f76b0101de0 2024-09-19T06:50:26.429 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:50:26.430 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":93,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:49:52.180171+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:50:26.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76940a3350 msgr2=0x7f76940a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:26.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76940a3350 0x7f76940a5810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f76b00ff6f0 tx=0x7f7698005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:26.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76b0101de0 msgr2=0x7f76b019a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76b0101de0 0x7f76b019a640 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f76a002a930 tx=0x7f76a0003b20 comp rx=0 tx=0).stop 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 shutdown_connections 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76940a3350 0x7f76940a5810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76b0106f10 0x7f76b019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f76b0102790 0x7f76b019ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 --2- 172.21.15.114:0/2316929147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76b0101de0 0x7f76b019a640 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 >> 172.21.15.114:0/2316929147 conn(0x7f76b00fd8c0 msgr2=0x7f76b00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 shutdown_connections 2024-09-19T06:50:26.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:26.431+0000 7f76b4e00640 1 -- 172.21.15.114:0/2316929147 wait complete. 2024-09-19T06:50:26.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:26 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2316929147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:26.768 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:26 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2316929147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:26 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2316929147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:50:27.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-19T06:50:27.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-19T06:50:27.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-19T06:50:27.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-19T06:50:27.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-19T06:50:27.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-19T06:50:27.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-19T06:50:27.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-19T06:50:27.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-19T06:50:27.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-19T06:50:27.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-19T06:50:27.153 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-19T06:50:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:27 smithi114 ceph-mon[30820]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-09-19T06:50:27.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:27 smithi118 ceph-mon[35527]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-09-19T06:50:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:27 smithi116 ceph-mon[35411]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-09-19T06:50:28.102 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 -- 172.21.15.114:0/2059683592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 msgr2=0x7f6a200fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 --2- 172.21.15.114:0/2059683592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 0x7f6a200fffc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009900 tx=0x7f6a0c02a410 comp rx=0 tx=0).stop 2024-09-19T06:50:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 -- 172.21.15.114:0/2059683592 shutdown_connections 2024-09-19T06:50:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 --2- 172.21.15.114:0/2059683592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a20100500 0x7f6a2010ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:28.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 --2- 172.21.15.114:0/2059683592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 0x7f6a200fffc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 --2- 172.21.15.114:0/2059683592 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6a20106b80 0x7f6a20106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 -- 172.21.15.114:0/2059683592 >> 172.21.15.114:0/2059683592 conn(0x7f6a200fba90 msgr2=0x7f6a200fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 -- 172.21.15.114:0/2059683592 shutdown_connections 2024-09-19T06:50:28.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.768+0000 7f6a24e00640 1 -- 172.21.15.114:0/2059683592 wait complete. 2024-09-19T06:50:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.769+0000 7f6a24e00640 1 Processor -- start 2024-09-19T06:50:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.769+0000 7f6a24e00640 1 -- start start 2024-09-19T06:50:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a24e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 0x7f6a201015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a24e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a20100500 0x7f6a20101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a24e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6a20106b80 0x7f6a20102050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:28.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a24e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6a20111ff0 con 0x7f6a200ffb60 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a24e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a20111e70 con 0x7f6a20100500 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a24e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f6a20112170 con 0x7f6a20106b80 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a1de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 0x7f6a201015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a1de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 0x7f6a201015d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37062/0 (socket says 172.21.15.114:37062) 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a1de00640 1 -- 172.21.15.114:0/3381232021 learned_addr learned my addr 172.21.15.114:0/3381232021 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a1d400640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a20100500 0x7f6a20101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.770+0000 7f6a1e800640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6a20106b80 0x7f6a20102050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a1de00640 1 -- 172.21.15.114:0/3381232021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6a20106b80 msgr2=0x7f6a20102050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a1de00640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6a20106b80 0x7f6a20102050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a1de00640 1 -- 172.21.15.114:0/3381232021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a20100500 msgr2=0x7f6a20101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a1de00640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a20100500 0x7f6a20101b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:28.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a1de00640 1 -- 172.21.15.114:0/3381232021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a201a6910 con 0x7f6a200ffb60 2024-09-19T06:50:28.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a1de00640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 0x7f6a201015d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000e9f0 tx=0x7f6a1000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:28.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1000cd80 con 0x7f6a200ffb60 2024-09-19T06:50:28.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a201a6c00 con 0x7f6a200ffb60 2024-09-19T06:50:28.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a10010910 con 0x7f6a200ffb60 2024-09-19T06:50:28.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a10018990 con 0x7f6a200ffb60 2024-09-19T06:50:28.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.771+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a201a7110 con 0x7f6a200ffb60 2024-09-19T06:50:28.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.772+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6a200fcd20 con 0x7f6a200ffb60 2024-09-19T06:50:28.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.774+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f6a10010ac0 con 0x7f6a200ffb60 2024-09-19T06:50:28.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.775+0000 7f6a16a00640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6a040a32d0 0x7f6a040a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:28.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.776+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f6a100d07a0 con 0x7f6a200ffb60 2024-09-19T06:50:28.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.776+0000 7f6a1d400640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6a040a32d0 0x7f6a040a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:28.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.776+0000 7f6a16a00640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] conn(0x7f6a040ae4a0 0x7f6a040b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:28.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.776+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 --> [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a10010760 con 0x7f6a040ae4a0 2024-09-19T06:50:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.776+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f6a100d0b50 con 0x7f6a200ffb60 2024-09-19T06:50:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.776+0000 7f6a1e800640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] conn(0x7f6a040ae4a0 0x7f6a040b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.777+0000 7f6a1d400640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6a040a32d0 0x7f6a040a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c000c00 tx=0x7f6a0c005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:28.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.777+0000 7f6a1e800640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] conn(0x7f6a040ae4a0 0x7f6a040b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:28.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.778+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 <== osd.3 v2:172.21.15.114:6810/21390467 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6a10010760 con 0x7f6a040ae4a0 2024-09-19T06:50:28.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.792+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 --> [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6a20101080 con 0x7f6a040ae4a0 2024-09-19T06:50:28.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.792+0000 7f6a16a00640 1 -- 172.21.15.114:0/3381232021 <== osd.3 v2:172.21.15.114:6810/21390467 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6a20101080 con 0x7f6a040ae4a0 2024-09-19T06:50:28.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.792+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] conn(0x7f6a040ae4a0 msgr2=0x7f6a040b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:28.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.792+0000 7f6a24e00640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] conn(0x7f6a040ae4a0 0x7f6a040b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:28.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.792+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6a040a32d0 msgr2=0x7f6a040a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:28.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.792+0000 7f6a24e00640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f6a040a32d0 0x7f6a040a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c000c00 tx=0x7f6a0c005b50 comp rx=0 tx=0).stop 2024-09-19T06:50:28.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.792+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 msgr2=0x7f6a201015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:28.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.792+0000 7f6a24e00640 1 --2- 172.21.15.114:0/3381232021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a200ffb60 0x7f6a201015d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000e9f0 tx=0x7f6a1000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:28.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.793+0000 7f6a1e800640 1 -- 172.21.15.114:0/3381232021 reap_dead start 2024-09-19T06:50:28.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.793+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 shutdown_connections 2024-09-19T06:50:28.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.793+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 >> 172.21.15.114:0/3381232021 conn(0x7f6a200fba90 msgr2=0x7f6a200fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:28.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.793+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 shutdown_connections 2024-09-19T06:50:28.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:28.793+0000 7f6a24e00640 1 -- 172.21.15.114:0/3381232021 wait complete. 2024-09-19T06:50:29.627 INFO:teuthology.orchestra.run.smithi114.stdout:107374182471 2024-09-19T06:50:29.627 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-19T06:50:29.669 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.671 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.673 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.673 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.675 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.682 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.685 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.689 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.700 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.704 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:29.709 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:30 smithi114 ceph-mon[30820]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-19T06:50:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:30 smithi116 ceph-mon[35411]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-19T06:50:30.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:30 smithi118 ceph-mon[35527]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 -- 172.21.15.114:0/758869042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe44c102790 msgr2=0x7fe44c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 --2- 172.21.15.114:0/758869042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe44c102790 0x7fe44c1067e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe438009860 tx=0x7fe43802a340 comp rx=0 tx=0).stop 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 -- 172.21.15.114:0/758869042 shutdown_connections 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 --2- 172.21.15.114:0/758869042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe44c106f10 0x7fe44c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 --2- 172.21.15.114:0/758869042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe44c102790 0x7fe44c1067e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 --2- 172.21.15.114:0/758869042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe44c101de0 0x7fe44c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 -- 172.21.15.114:0/758869042 >> 172.21.15.114:0/758869042 conn(0x7fe44c0fd8c0 msgr2=0x7fe44c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 -- 172.21.15.114:0/758869042 shutdown_connections 2024-09-19T06:50:31.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.787+0000 7fe453c00640 1 -- 172.21.15.114:0/758869042 wait complete. 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.789+0000 7fe453c00640 1 Processor -- start 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.789+0000 7fe453c00640 1 -- start start 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.789+0000 7fe453c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe44c101de0 0x7fe44c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.789+0000 7fe453c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe44c102790 0x7fe44c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.789+0000 7fe453c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe44c106f10 0x7fe44c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.789+0000 7fe453c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c111f70 con 0x7fe44c106f10 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.789+0000 7fe453c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c111df0 con 0x7fe44c101de0 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.789+0000 7fe453c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe44c1120f0 con 0x7fe44c102790 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe44c102790 0x7fe44c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe451c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe44c106f10 0x7fe44c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe44c102790 0x7fe44c198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:41460/0 (socket says 172.21.15.114:41460) 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 -- 172.21.15.114:0/346554503 learned_addr learned my addr 172.21.15.114:0/346554503 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe451200640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe44c101de0 0x7fe44c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 -- 172.21.15.114:0/346554503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe44c101de0 msgr2=0x7fe44c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe44c101de0 0x7fe44c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 -- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe44c106f10 msgr2=0x7fe44c19cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe44c106f10 0x7fe44c19cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 -- 172.21.15.114:0/346554503 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe44c19d3e0 con 0x7fe44c102790 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.790+0000 7fe44be00640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe44c102790 0x7fe44c198970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe438035040 tx=0x7fe438033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.791+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43803f020 con 0x7fe44c102790 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.791+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe438032770 con 0x7fe44c102790 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.791+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe44c19d670 con 0x7fe44c102790 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.791+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe438037040 con 0x7fe44c102790 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.791+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe44c074420 con 0x7fe44c102790 2024-09-19T06:50:31.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.792+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe44c062980 con 0x7fe44c102790 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.794+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fe438042020 con 0x7fe44c102790 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.796+0000 7fe449600640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe4340a3350 0x7fe4340a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.796+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe4380f01e0 con 0x7fe44c102790 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.796+0000 7fe451200640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe4340a3350 0x7fe4340a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.797+0000 7fe449600640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] conn(0x7fe4340ae3f0 0x7fe4340b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.797+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 --> [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe438001d30 con 0x7fe4340ae3f0 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.797+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fe4380f0590 con 0x7fe44c102790 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.797+0000 7fe451c00640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] conn(0x7fe4340ae3f0 0x7fe4340b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.797+0000 7fe451200640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe4340a3350 0x7fe4340a5810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe440005fd0 tx=0x7fe440009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:31.036 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.798+0000 7fe451c00640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] conn(0x7fe4340ae3f0 0x7fe4340b0890 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.798+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 <== osd.9 v2:172.21.15.114:6826/2129482086 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe438001d30 con 0x7fe4340ae3f0 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.814+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 --> [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe44c062d80 con 0x7fe4340ae3f0 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.814+0000 7fe449600640 1 -- 172.21.15.114:0/346554503 <== osd.9 v2:172.21.15.114:6826/2129482086 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe44c062d80 con 0x7fe4340ae3f0 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] conn(0x7fe4340ae3f0 msgr2=0x7fe4340b0890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] conn(0x7fe4340ae3f0 0x7fe4340b0890 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe4340a3350 msgr2=0x7fe4340a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe4340a3350 0x7fe4340a5810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe440005fd0 tx=0x7fe440009260 comp rx=0 tx=0).stop 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe44c102790 msgr2=0x7fe44c198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 --2- 172.21.15.114:0/346554503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe44c102790 0x7fe44c198970 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe438035040 tx=0x7fe438033d90 comp rx=0 tx=0).stop 2024-09-19T06:50:31.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe451c00640 1 -- 172.21.15.114:0/346554503 reap_dead start 2024-09-19T06:50:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 shutdown_connections 2024-09-19T06:50:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 >> 172.21.15.114:0/346554503 conn(0x7fe44c0fd8c0 msgr2=0x7fe44c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 shutdown_connections 2024-09-19T06:50:31.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:30.815+0000 7fe453c00640 1 -- 172.21.15.114:0/346554503 wait complete. 2024-09-19T06:50:31.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 -- 172.21.15.114:0/962378102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e8103e30 msgr2=0x7f52e8104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 --2- 172.21.15.114:0/962378102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e8103e30 0x7f52e8104210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009900 tx=0x7f52d802a3f0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 -- 172.21.15.114:0/962378102 shutdown_connections 2024-09-19T06:50:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 --2- 172.21.15.114:0/962378102 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52e8108f60 0x7f52e8110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 --2- 172.21.15.114:0/962378102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e81047e0 0x7f52e8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 --2- 172.21.15.114:0/962378102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e8103e30 0x7f52e8104210 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 -- 172.21.15.114:0/962378102 >> 172.21.15.114:0/962378102 conn(0x7f52e80ff8d0 msgr2=0x7f52e8101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 -- 172.21.15.114:0/962378102 shutdown_connections 2024-09-19T06:50:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.658+0000 7f52eca00640 1 -- 172.21.15.114:0/962378102 wait complete. 2024-09-19T06:50:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.659+0000 7f52eca00640 1 Processor -- start 2024-09-19T06:50:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52eca00640 1 -- start start 2024-09-19T06:50:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52eca00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e8103e30 0x7f52e819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52eca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e81047e0 0x7f52e819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52eca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52e8108f60 0x7f52e819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52eca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8114010 con 0x7f52e81047e0 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52eca00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8113e90 con 0x7f52e8103e30 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52eca00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8114190 con 0x7f52e8108f60 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52dde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e81047e0 0x7f52e819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52e6800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e8103e30 0x7f52e819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52dde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e81047e0 0x7f52e819aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37088/0 (socket says 172.21.15.114:37088) 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52dde00640 1 -- 172.21.15.114:0/26382624 learned_addr learned my addr 172.21.15.114:0/26382624 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.660+0000 7f52e7200640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52e8108f60 0x7f52e819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52e6800640 1 -- 172.21.15.114:0/26382624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52e8108f60 msgr2=0x7f52e819edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52e6800640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f52e8108f60 0x7f52e819edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52e6800640 1 -- 172.21.15.114:0/26382624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e81047e0 msgr2=0x7f52e819aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52e6800640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f52e81047e0 0x7f52e819aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52e6800640 1 -- 172.21.15.114:0/26382624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e819f4a0 con 0x7f52e8103e30 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52e6800640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e8103e30 0x7f52e819a4f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f52d802a910 tx=0x7f52d8003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d803f070 con 0x7f52e8103e30 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e819f730 con 0x7f52e8103e30 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.661+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52d8031c90 con 0x7f52e8103e30 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.662+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8035030 con 0x7f52e8103e30 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.662+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e81a7070 con 0x7f52e8103e30 2024-09-19T06:50:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.662+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f52e8062900 con 0x7f52e8103e30 2024-09-19T06:50:31.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.665+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f52d8042070 con 0x7f52e8103e30 2024-09-19T06:50:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.666+0000 7f52dfe00640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f52cc0a32d0 0x7f52cc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.666+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f52d80f0910 con 0x7f52e8103e30 2024-09-19T06:50:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.666+0000 7f52dde00640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f52cc0a32d0 0x7f52cc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.666+0000 7f52dfe00640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] conn(0x7f52cc0ae420 0x7f52cc0b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.666+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 --> [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52d8034e50 con 0x7f52cc0ae420 2024-09-19T06:50:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.666+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f52d80f0cc0 con 0x7f52e8103e30 2024-09-19T06:50:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.667+0000 7f52e7200640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] conn(0x7f52cc0ae420 0x7f52cc0b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.667+0000 7f52dde00640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f52cc0a32d0 0x7f52cc0a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f52e80ffd80 tx=0x7f52d0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.667+0000 7f52e7200640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] conn(0x7f52cc0ae420 0x7f52cc0b08c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:31.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.668+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 <== osd.1 v2:172.21.15.116:6800/2939029063 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f52d8034e50 con 0x7f52cc0ae420 2024-09-19T06:50:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.682+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 --> [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f52e8062d00 con 0x7f52cc0ae420 2024-09-19T06:50:31.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.683+0000 7f52dfe00640 1 -- 172.21.15.114:0/26382624 <== osd.1 v2:172.21.15.116:6800/2939029063 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f52e8062d00 con 0x7f52cc0ae420 2024-09-19T06:50:31.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 >> [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] conn(0x7f52cc0ae420 msgr2=0x7f52cc0b08c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] conn(0x7f52cc0ae420 0x7f52cc0b08c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f52cc0a32d0 msgr2=0x7f52cc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f52cc0a32d0 0x7f52cc0a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f52e80ffd80 tx=0x7f52d0005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e8103e30 msgr2=0x7f52e819a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 --2- 172.21.15.114:0/26382624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52e8103e30 0x7f52e819a4f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f52d802a910 tx=0x7f52d8003b20 comp rx=0 tx=0).stop 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52e7200640 1 -- 172.21.15.114:0/26382624 reap_dead start 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 shutdown_connections 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 >> 172.21.15.114:0/26382624 conn(0x7f52e80ff8d0 msgr2=0x7f52e8100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 shutdown_connections 2024-09-19T06:50:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:31.684+0000 7f52eca00640 1 -- 172.21.15.114:0/26382624 wait complete. 2024-09-19T06:50:32.250 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:32 smithi114 ceph-mon[30820]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:50:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:32 smithi116 ceph-mon[35411]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:50:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:32 smithi118 ceph-mon[35527]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.508+0000 7fbcef200640 1 -- 172.21.15.114:0/3698626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf01047e0 msgr2=0x7fbcf0108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.508+0000 7fbcef200640 1 --2- 172.21.15.114:0/3698626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf01047e0 0x7fbcf0108830 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009900 tx=0x7fbcdc02a400 comp rx=0 tx=0).stop 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.508+0000 7fbcef200640 1 -- 172.21.15.114:0/3698626 shutdown_connections 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.508+0000 7fbcef200640 1 --2- 172.21.15.114:0/3698626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf0108f60 0x7fbcf0110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.508+0000 7fbcef200640 1 --2- 172.21.15.114:0/3698626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf01047e0 0x7fbcf0108830 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.508+0000 7fbcef200640 1 --2- 172.21.15.114:0/3698626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcf0103e30 0x7fbcf0104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.508+0000 7fbcef200640 1 -- 172.21.15.114:0/3698626 >> 172.21.15.114:0/3698626 conn(0x7fbcf00ff910 msgr2=0x7fbcf0101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.509+0000 7fbcef200640 1 -- 172.21.15.114:0/3698626 shutdown_connections 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.509+0000 7fbcef200640 1 -- 172.21.15.114:0/3698626 wait complete. 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.510+0000 7fbcef200640 1 Processor -- start 2024-09-19T06:50:32.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.511+0000 7fbcef200640 1 -- start start 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.511+0000 7fbcef200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf0103e30 0x7fbcf019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.511+0000 7fbcef200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcf01047e0 0x7fbcf019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.511+0000 7fbcef200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf0108f60 0x7fbcf019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.511+0000 7fbcef200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf0113f80 con 0x7fbcf0103e30 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.511+0000 7fbcef200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf0113e00 con 0x7fbcf01047e0 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.511+0000 7fbcef200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf0114100 con 0x7fbcf0108f60 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.511+0000 7fbcee800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf0108f60 0x7fbcf019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbcee800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf0108f60 0x7fbcf019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:41490/0 (socket says 172.21.15.114:41490) 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbcee800640 1 -- 172.21.15.114:0/441238825 learned_addr learned my addr 172.21.15.114:0/441238825 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbcede00640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf0103e30 0x7fbcf019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbced400640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcf01047e0 0x7fbcf019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbcee800640 1 -- 172.21.15.114:0/441238825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcf01047e0 msgr2=0x7fbcf019a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:32.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbcee800640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcf01047e0 0x7fbcf019a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbcee800640 1 -- 172.21.15.114:0/441238825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf0103e30 msgr2=0x7fbcf019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbcee800640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbcf0103e30 0x7fbcf019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.512+0000 7fbcee800640 1 -- 172.21.15.114:0/441238825 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf019f430 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.513+0000 7fbcee800640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf0108f60 0x7fbcf019ed50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbce800efd0 tx=0x7fbce800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.513+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8003b70 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.513+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbce8003d10 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.513+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce80109f0 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.513+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf019f720 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.513+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf01a7000 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.514+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbcf0062980 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.518+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fbce8010b90 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.519+0000 7fbce6a00640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbcd40a3350 0x7fbcd40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.519+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fbce80d0420 con 0x7fbcf0108f60 2024-09-19T06:50:32.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.519+0000 7fbcede00640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbcd40a3350 0x7fbcd40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.520+0000 7fbce6a00640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] conn(0x7fbcd40ae520 0x7fbcd40b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.520+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 --> [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbce8043e50 con 0x7fbcd40ae520 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.520+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fbce80d07d0 con 0x7fbcf0108f60 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.520+0000 7fbced400640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] conn(0x7fbcd40ae520 0x7fbcd40b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.520+0000 7fbcede00640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbcd40a3350 0x7fbcd40a5810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbce000bbe0 tx=0x7fbce000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.521+0000 7fbced400640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] conn(0x7fbcd40ae520 0x7fbcd40b09c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.522+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 <== osd.8 v2:172.21.15.118:6816/3182934697 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbce8043e50 con 0x7fbcd40ae520 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.536+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 --> [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbcf0062e50 con 0x7fbcd40ae520 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.536+0000 7fbce6a00640 1 -- 172.21.15.114:0/441238825 <== osd.8 v2:172.21.15.118:6816/3182934697 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbcf0062e50 con 0x7fbcd40ae520 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 >> [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] conn(0x7fbcd40ae520 msgr2=0x7fbcd40b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:32.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] conn(0x7fbcd40ae520 0x7fbcd40b09c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbcd40a3350 msgr2=0x7fbcd40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbcd40a3350 0x7fbcd40a5810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbce000bbe0 tx=0x7fbce000b390 comp rx=0 tx=0).stop 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf0108f60 msgr2=0x7fbcf019ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 --2- 172.21.15.114:0/441238825 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbcf0108f60 0x7fbcf019ed50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbce800efd0 tx=0x7fbce800c6a0 comp rx=0 tx=0).stop 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcee800640 1 -- 172.21.15.114:0/441238825 reap_dead start 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 shutdown_connections 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 >> 172.21.15.114:0/441238825 conn(0x7fbcf00ff910 msgr2=0x7fbcf0100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 shutdown_connections 2024-09-19T06:50:32.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.537+0000 7fbcef200640 1 -- 172.21.15.114:0/441238825 wait complete. 2024-09-19T06:50:33.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 -- 172.21.15.114:0/2762638364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1194103ff0 msgr2=0x7f11941043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 --2- 172.21.15.114:0/2762638364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1194103ff0 0x7f11941043d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1184009900 tx=0x7f118402a3f0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 -- 172.21.15.114:0/2762638364 shutdown_connections 2024-09-19T06:50:33.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 --2- 172.21.15.114:0/2762638364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1194109120 0x7f1194110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 --2- 172.21.15.114:0/2762638364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f11941049a0 0x7f11941089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 --2- 172.21.15.114:0/2762638364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1194103ff0 0x7f11941043d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 -- 172.21.15.114:0/2762638364 >> 172.21.15.114:0/2762638364 conn(0x7f11940ffa90 msgr2=0x7f1194101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:33.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 -- 172.21.15.114:0/2762638364 shutdown_connections 2024-09-19T06:50:33.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.106+0000 7f1198e00640 1 -- 172.21.15.114:0/2762638364 wait complete. 2024-09-19T06:50:33.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.107+0000 7f1198e00640 1 Processor -- start 2024-09-19T06:50:33.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.107+0000 7f1198e00640 1 -- start start 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1198e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1194103ff0 0x7f119419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1198e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11941049a0 0x7f119419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1198e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1194109120 0x7f119419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1198e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1194114190 con 0x7f11941049a0 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1198e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1194114010 con 0x7f1194109120 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1198e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1194114310 con 0x7f1194103ff0 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1191400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11941049a0 0x7f119419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1191400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11941049a0 0x7f119419abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37128/0 (socket says 172.21.15.114:37128) 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1191400640 1 -- 172.21.15.114:0/3923282845 learned_addr learned my addr 172.21.15.114:0/3923282845 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1191e00640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1194103ff0 0x7f119419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.108+0000 7f1192800640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1194109120 0x7f119419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f1191400640 1 -- 172.21.15.114:0/3923282845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1194103ff0 msgr2=0x7f119419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f1191400640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1194103ff0 0x7f119419a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f1191400640 1 -- 172.21.15.114:0/3923282845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1194109120 msgr2=0x7f119419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f1191400640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1194109120 0x7f119419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f1191400640 1 -- 172.21.15.114:0/3923282845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119419f640 con 0x7f11941049a0 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f1191400640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11941049a0 0x7f119419abd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f118000d770 tx=0x7f118000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1180017070 con 0x7f11941049a0 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1180005500 con 0x7f11941049a0 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119419f930 con 0x7f11941049a0 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1180004e00 con 0x7f11941049a0 2024-09-19T06:50:33.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.109+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11941a7210 con 0x7f11941049a0 2024-09-19T06:50:33.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.110+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1194062900 con 0x7f11941049a0 2024-09-19T06:50:33.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.113+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f1180004fa0 con 0x7f11941049a0 2024-09-19T06:50:33.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.114+0000 7f118aa00640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f11780a3350 0x7f11780a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.114+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f11800cfb30 con 0x7f11941049a0 2024-09-19T06:50:33.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.114+0000 7f1191e00640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f11780a3350 0x7f11780a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.114+0000 7f118aa00640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] conn(0x7f11780ae520 0x7f11780b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.114+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 --> [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1180001d30 con 0x7f11780ae520 2024-09-19T06:50:33.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.114+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f11800cff70 con 0x7f11941049a0 2024-09-19T06:50:33.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.115+0000 7f1192800640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] conn(0x7f11780ae520 0x7f11780b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.115+0000 7f1191e00640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f11780a3350 0x7f11780a5810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1184000c00 tx=0x7f1184005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:33.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.115+0000 7f1192800640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] conn(0x7f11780ae520 0x7f11780b09c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:33.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.116+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 <== osd.2 v2:172.21.15.118:6800/657498996 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1180001d30 con 0x7f11780ae520 2024-09-19T06:50:33.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.130+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 --> [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1194062d00 con 0x7f11780ae520 2024-09-19T06:50:33.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f118aa00640 1 -- 172.21.15.114:0/3923282845 <== osd.2 v2:172.21.15.118:6800/657498996 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1194062d00 con 0x7f11780ae520 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 >> [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] conn(0x7f11780ae520 msgr2=0x7f11780b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] conn(0x7f11780ae520 0x7f11780b09c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f11780a3350 msgr2=0x7f11780a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f11780a3350 0x7f11780a5810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1184000c00 tx=0x7f1184005b50 comp rx=0 tx=0).stop 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11941049a0 msgr2=0x7f119419abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 --2- 172.21.15.114:0/3923282845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11941049a0 0x7f119419abd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f118000d770 tx=0x7f118000dc30 comp rx=0 tx=0).stop 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1192800640 1 -- 172.21.15.114:0/3923282845 reap_dead start 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 shutdown_connections 2024-09-19T06:50:33.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 >> 172.21.15.114:0/3923282845 conn(0x7f11940ffa90 msgr2=0x7f1194100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:33.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.131+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 shutdown_connections 2024-09-19T06:50:33.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.132+0000 7f1198e00640 1 -- 172.21.15.114:0/3923282845 wait complete. 2024-09-19T06:50:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.317+0000 7feb51a00640 1 -- 172.21.15.114:0/1621395665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c104010 msgr2=0x7feb4c1043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.317+0000 7feb51a00640 1 --2- 172.21.15.114:0/1621395665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c104010 0x7feb4c1043f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009900 tx=0x7feb3c02a410 comp rx=0 tx=0).stop 2024-09-19T06:50:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.317+0000 7feb51a00640 1 -- 172.21.15.114:0/1621395665 shutdown_connections 2024-09-19T06:50:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.317+0000 7feb51a00640 1 --2- 172.21.15.114:0/1621395665 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb4c109140 0x7feb4c110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.317+0000 7feb51a00640 1 --2- 172.21.15.114:0/1621395665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c1049c0 0x7feb4c108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.317+0000 7feb51a00640 1 --2- 172.21.15.114:0/1621395665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c104010 0x7feb4c1043f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.317+0000 7feb51a00640 1 -- 172.21.15.114:0/1621395665 >> 172.21.15.114:0/1621395665 conn(0x7feb4c0ffa50 msgr2=0x7feb4c101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:33.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.318+0000 7feb51a00640 1 -- 172.21.15.114:0/1621395665 shutdown_connections 2024-09-19T06:50:33.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.318+0000 7feb51a00640 1 -- 172.21.15.114:0/1621395665 wait complete. 2024-09-19T06:50:33.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.319+0000 7feb51a00640 1 Processor -- start 2024-09-19T06:50:33.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.319+0000 7feb51a00640 1 -- start start 2024-09-19T06:50:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb51a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c104010 0x7feb4c110860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb51a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb4c1049c0 0x7feb4c10e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb51a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c109140 0x7feb4c10eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb51a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c076c40 con 0x7feb4c109140 2024-09-19T06:50:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb51a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c076ac0 con 0x7feb4c104010 2024-09-19T06:50:33.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb51a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c076dc0 con 0x7feb4c1049c0 2024-09-19T06:50:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb4a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb4c1049c0 0x7feb4c10e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb4a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb4c1049c0 0x7feb4c10e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:41544/0 (socket says 172.21.15.114:41544) 2024-09-19T06:50:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb4a000640 1 -- 172.21.15.114:0/2319128908 learned_addr learned my addr 172.21.15.114:0/2319128908 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.320+0000 7feb4b400640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c109140 0x7feb4c10eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.321+0000 7feb4aa00640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c104010 0x7feb4c110860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.321+0000 7feb4a000640 1 -- 172.21.15.114:0/2319128908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c104010 msgr2=0x7feb4c110860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.321+0000 7feb4a000640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c104010 0x7feb4c110860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.321+0000 7feb4a000640 1 -- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c109140 msgr2=0x7feb4c10eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.321+0000 7feb4a000640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c109140 0x7feb4c10eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.321+0000 7feb4a000640 1 -- 172.21.15.114:0/2319128908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c10f520 con 0x7feb4c1049c0 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.321+0000 7feb4aa00640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb4c104010 0x7feb4c110860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.322+0000 7feb4b400640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb4c109140 0x7feb4c10eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.322+0000 7feb4a000640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb4c1049c0 0x7feb4c10e970 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feb3800d950 tx=0x7feb3800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.322+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38017070 con 0x7feb4c1049c0 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.322+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb38009c30 con 0x7feb4c1049c0 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.322+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3800f6b0 con 0x7feb4c1049c0 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.322+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4c1ab140 con 0x7feb4c1049c0 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.322+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4c1ab4e0 con 0x7feb4c1049c0 2024-09-19T06:50:33.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.323+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feb4c0628c0 con 0x7feb4c1049c0 2024-09-19T06:50:33.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.326+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7feb380109d0 con 0x7feb4c1049c0 2024-09-19T06:50:33.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.328+0000 7feb43400640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feb300a3380 0x7feb300a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.328+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7feb380cfa60 con 0x7feb4c1049c0 2024-09-19T06:50:33.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.328+0000 7feb4aa00640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feb300a3380 0x7feb300a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.328+0000 7feb43400640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] conn(0x7feb300ae550 0x7feb300b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:33.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.328+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 --> [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb300b10c0 con 0x7feb300ae550 2024-09-19T06:50:33.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.328+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7feb380cfe10 con 0x7feb4c1049c0 2024-09-19T06:50:33.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.328+0000 7feb4b400640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] conn(0x7feb300ae550 0x7feb300b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:33.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.329+0000 7feb4aa00640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feb300a3380 0x7feb300a5840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feb3c000c00 tx=0x7feb3c005a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:33.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.329+0000 7feb4b400640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] conn(0x7feb300ae550 0x7feb300b09f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:33.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.330+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 <== osd.6 v2:172.21.15.114:6818/1336611047 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb300b10c0 con 0x7feb300ae550 2024-09-19T06:50:33.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.344+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 --> [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feb4c062d90 con 0x7feb300ae550 2024-09-19T06:50:33.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb43400640 1 -- 172.21.15.114:0/2319128908 <== osd.6 v2:172.21.15.114:6818/1336611047 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7feb4c062d90 con 0x7feb300ae550 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] conn(0x7feb300ae550 msgr2=0x7feb300b09f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] conn(0x7feb300ae550 0x7feb300b09f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feb300a3380 msgr2=0x7feb300a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7feb300a3380 0x7feb300a5840 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feb3c000c00 tx=0x7feb3c005a40 comp rx=0 tx=0).stop 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb4c1049c0 msgr2=0x7feb4c10e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 --2- 172.21.15.114:0/2319128908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feb4c1049c0 0x7feb4c10e970 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feb3800d950 tx=0x7feb3800de10 comp rx=0 tx=0).stop 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb4b400640 1 -- 172.21.15.114:0/2319128908 reap_dead start 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 shutdown_connections 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 >> 172.21.15.114:0/2319128908 conn(0x7feb4c0ffa50 msgr2=0x7feb4c100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.345+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 shutdown_connections 2024-09-19T06:50:33.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:33.346+0000 7feb51a00640 1 -- 172.21.15.114:0/2319128908 wait complete. 2024-09-19T06:50:34.307 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:34 smithi114 ceph-mon[30820]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:34 smithi116 ceph-mon[35411]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:34 smithi118 ceph-mon[35527]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:34.897 INFO:teuthology.orchestra.run.smithi114.stdout:197568495671 2024-09-19T06:50:34.897 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-19T06:50:35.463 INFO:teuthology.orchestra.run.smithi114.stdout:55834574930 2024-09-19T06:50:35.463 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-19T06:50:35.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.619+0000 7f2bd7e00640 1 -- 172.21.15.114:0/1176748575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd8102370 msgr2=0x7f2bd810e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:35.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.619+0000 7f2bd7e00640 1 --2- 172.21.15.114:0/1176748575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd8102370 0x7f2bd810e8a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009cd0 tx=0x7f2bd002a8e0 comp rx=0 tx=0).stop 2024-09-19T06:50:35.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.619+0000 7f2bd7e00640 1 -- 172.21.15.114:0/1176748575 shutdown_connections 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.619+0000 7f2bd7e00640 1 --2- 172.21.15.114:0/1176748575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd8102370 0x7f2bd810e8a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.619+0000 7f2bd7e00640 1 --2- 172.21.15.114:0/1176748575 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd81019d0 0x7f2bd8101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.619+0000 7f2bd7e00640 1 --2- 172.21.15.114:0/1176748575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bd81079d0 0x7f2bd8107db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.619+0000 7f2bd7e00640 1 -- 172.21.15.114:0/1176748575 >> 172.21.15.114:0/1176748575 conn(0x7f2bd80fd710 msgr2=0x7f2bd80ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.620+0000 7f2bd7e00640 1 -- 172.21.15.114:0/1176748575 shutdown_connections 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.620+0000 7f2bd7e00640 1 -- 172.21.15.114:0/1176748575 wait complete. 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.621+0000 7f2bd7e00640 1 Processor -- start 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.621+0000 7f2bd7e00640 1 -- start start 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.621+0000 7f2bd7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd81019d0 0x7f2bd80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.621+0000 7f2bd7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bd8102370 0x7f2bd8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.621+0000 7f2bd7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd81079d0 0x7f2bd8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:35.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.621+0000 7f2bd7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd8111dd0 con 0x7f2bd8102370 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.621+0000 7f2bd7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd8111c50 con 0x7f2bd81079d0 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.621+0000 7f2bd7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd8111f50 con 0x7f2bd81019d0 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bd8102370 0x7f2bd8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd81019d0 0x7f2bd80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd81079d0 0x7f2bd8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd81019d0 0x7f2bd80744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:41508/0 (socket says 172.21.15.114:41508) 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 -- 172.21.15.114:0/3911300853 learned_addr learned my addr 172.21.15.114:0/3911300853 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6000640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bd8102370 0x7f2bd8074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37112/0 (socket says 172.21.15.114:37112) 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 -- 172.21.15.114:0/3911300853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd81079d0 msgr2=0x7f2bd8074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd81079d0 0x7f2bd8074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 -- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bd8102370 msgr2=0x7f2bd8074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bd8102370 0x7f2bd8074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:35.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 -- 172.21.15.114:0/3911300853 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd81a6710 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6000640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bd8102370 0x7f2bd8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.622+0000 7f2bd6a00640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd81019d0 0x7f2bd80744f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800e9f0 tx=0x7f2bc800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.623+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc800cd80 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.623+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc8010910 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.623+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd81a6a00 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.623+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8018990 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.623+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd81a6f90 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.624+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bd8062900 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.626+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f2bc8010ac0 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.627+0000 7f2bdca00640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2bbc0a3350 0x7f2bbc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.628+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2bc80d06d0 con 0x7f2bd81019d0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.628+0000 7f2bd6000640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2bbc0a3350 0x7f2bbc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.628+0000 7f2bdca00640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] conn(0x7f2bbc0ae520 0x7f2bbc0b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.628+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 --> [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bbc0b1090 con 0x7f2bbc0ae520 2024-09-19T06:50:35.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.628+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f2bc80d0a80 con 0x7f2bd81019d0 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.628+0000 7f2bd6000640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2bbc0a3350 0x7f2bbc0a5810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8075f80 tx=0x7f2bc4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.628+0000 7f2bd7400640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] conn(0x7f2bbc0ae520 0x7f2bbc0b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.629+0000 7f2bd7400640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] conn(0x7f2bbc0ae520 0x7f2bbc0b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.630+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 <== osd.7 v2:172.21.15.116:6816/2839705948 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2bbc0b1090 con 0x7f2bbc0ae520 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.644+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 --> [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bd8062d00 con 0x7f2bbc0ae520 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.645+0000 7f2bdca00640 1 -- 172.21.15.114:0/3911300853 <== osd.7 v2:172.21.15.116:6816/2839705948 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2bd8062d00 con 0x7f2bbc0ae520 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.645+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 >> [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] conn(0x7f2bbc0ae520 msgr2=0x7f2bbc0b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.645+0000 7f2bd7e00640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] conn(0x7f2bbc0ae520 0x7f2bbc0b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:35.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.645+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2bbc0a3350 msgr2=0x7f2bbc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.645+0000 7f2bd7e00640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2bbc0a3350 0x7f2bbc0a5810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8075f80 tx=0x7f2bc4005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.645+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd81019d0 msgr2=0x7f2bd80744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.645+0000 7f2bd7e00640 1 --2- 172.21.15.114:0/3911300853 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2bd81019d0 0x7f2bd80744f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800e9f0 tx=0x7f2bc800eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.645+0000 7f2bd7400640 1 -- 172.21.15.114:0/3911300853 reap_dead start 2024-09-19T06:50:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.646+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 shutdown_connections 2024-09-19T06:50:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.646+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 >> 172.21.15.114:0/3911300853 conn(0x7f2bd80fd710 msgr2=0x7f2bd80fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.646+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 shutdown_connections 2024-09-19T06:50:35.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:32.646+0000 7f2bd7e00640 1 -- 172.21.15.114:0/3911300853 wait complete. 2024-09-19T06:50:36.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:36 smithi114 ceph-mon[30820]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 -- 172.21.15.114:0/3776580090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7904108f60 msgr2=0x7f7904110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 --2- 172.21.15.114:0/3776580090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7904108f60 0x7f7904110ae0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009ca0 tx=0x7f78fc02a8c0 comp rx=0 tx=0).stop 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 -- 172.21.15.114:0/3776580090 shutdown_connections 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 --2- 172.21.15.114:0/3776580090 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7904108f60 0x7f7904110ae0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 --2- 172.21.15.114:0/3776580090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f79041047e0 0x7f7904108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 --2- 172.21.15.114:0/3776580090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7904103e30 0x7f7904104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 -- 172.21.15.114:0/3776580090 >> 172.21.15.114:0/3776580090 conn(0x7f79040ff8d0 msgr2=0x7f7904101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 -- 172.21.15.114:0/3776580090 shutdown_connections 2024-09-19T06:50:36.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.419+0000 7f790ac00640 1 -- 172.21.15.114:0/3776580090 wait complete. 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.420+0000 7f790ac00640 1 Processor -- start 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790ac00640 1 -- start start 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790ac00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7904103e30 0x7f790419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790ac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79041047e0 0x7f790419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790ac00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7904108f60 0x7f790419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790ac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7904113fc0 con 0x7f79041047e0 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790ac00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7904113e40 con 0x7f7904103e30 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790ac00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7904114140 con 0x7f7904108f60 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f7908e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79041047e0 0x7f790419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790a200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7904108f60 0x7f790419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f7909800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7904103e30 0x7f790419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:36.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790a200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7904108f60 0x7f790419ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:41554/0 (socket says 172.21.15.114:41554) 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.421+0000 7f790a200640 1 -- 172.21.15.114:0/929820942 learned_addr learned my addr 172.21.15.114:0/929820942 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f790a200640 1 -- 172.21.15.114:0/929820942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7904103e30 msgr2=0x7f790419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f790a200640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7904103e30 0x7f790419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f790a200640 1 -- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79041047e0 msgr2=0x7f790419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f790a200640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79041047e0 0x7f790419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f790a200640 1 -- 172.21.15.114:0/929820942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790419f3b0 con 0x7f7904108f60 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f7909800640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7904103e30 0x7f790419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f790a200640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7904108f60 0x7f790419ecd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f78fc031a20 tx=0x7f78fc031a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc003b70 con 0x7f7904108f60 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.422+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f790419f640 con 0x7f7904108f60 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.423+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78fc003d10 con 0x7f7904108f60 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.423+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc02fc40 con 0x7f7904108f60 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.423+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79041a6f80 con 0x7f7904108f60 2024-09-19T06:50:36.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.424+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7904062980 con 0x7f7904108f60 2024-09-19T06:50:36.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.426+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f78fc004050 con 0x7f7904108f60 2024-09-19T06:50:36.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.427+0000 7f7902000640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78ec0a3350 0x7f78ec0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:36.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.427+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f78fc0f0ff0 con 0x7f7904108f60 2024-09-19T06:50:36.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.427+0000 7f7909800640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78ec0a3350 0x7f78ec0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:36.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.427+0000 7f7902000640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] conn(0x7f78ec0ae520 0x7f78ec0b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:36.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.427+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 --> [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78fc036e70 con 0x7f78ec0ae520 2024-09-19T06:50:36.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.427+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f78fc0f1420 con 0x7f7904108f60 2024-09-19T06:50:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:36 smithi116 ceph-mon[35411]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:50:36.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.428+0000 7f7908e00640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] conn(0x7f78ec0ae520 0x7f78ec0b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:36.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.428+0000 7f7909800640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78ec0a3350 0x7f78ec0a5810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009fc0 tx=0x7f78f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:36.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.428+0000 7f7908e00640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] conn(0x7f78ec0ae520 0x7f78ec0b09c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:36.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.429+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 <== osd.0 v2:172.21.15.114:6802/1641241463 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f78fc036e70 con 0x7f78ec0ae520 2024-09-19T06:50:36.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.446+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 --> [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7904062d80 con 0x7f78ec0ae520 2024-09-19T06:50:36.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.447+0000 7f7902000640 1 -- 172.21.15.114:0/929820942 <== osd.0 v2:172.21.15.114:6802/1641241463 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7904062d80 con 0x7f78ec0ae520 2024-09-19T06:50:36.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.447+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] conn(0x7f78ec0ae520 msgr2=0x7f78ec0b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:36.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.447+0000 7f790ac00640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] conn(0x7f78ec0ae520 0x7f78ec0b09c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:36.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78ec0a3350 msgr2=0x7f78ec0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:36.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790ac00640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f78ec0a3350 0x7f78ec0a5810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009fc0 tx=0x7f78f8005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:36.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7904108f60 msgr2=0x7f790419ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:36.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790ac00640 1 --2- 172.21.15.114:0/929820942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7904108f60 0x7f790419ecd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f78fc031a20 tx=0x7f78fc031a50 comp rx=0 tx=0).stop 2024-09-19T06:50:36.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790a200640 1 -- 172.21.15.114:0/929820942 reap_dead start 2024-09-19T06:50:36.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 shutdown_connections 2024-09-19T06:50:36.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 >> 172.21.15.114:0/929820942 conn(0x7f79040ff8d0 msgr2=0x7f7904100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:36.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 shutdown_connections 2024-09-19T06:50:36.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.448+0000 7f790ac00640 1 -- 172.21.15.114:0/929820942 wait complete. 2024-09-19T06:50:36.509 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:36 smithi118 ceph-mon[35527]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:50:37.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 -- 172.21.15.114:0/4095336411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 msgr2=0x7faaf8110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 --2- 172.21.15.114:0/4095336411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 0x7faaf8110ae0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009ca0 tx=0x7faaf002a8c0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 -- 172.21.15.114:0/4095336411 shutdown_connections 2024-09-19T06:50:37.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 --2- 172.21.15.114:0/4095336411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 0x7faaf8110ae0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 --2- 172.21.15.114:0/4095336411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf81047e0 0x7faaf8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 --2- 172.21.15.114:0/4095336411 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf8103e30 0x7faaf8104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 -- 172.21.15.114:0/4095336411 >> 172.21.15.114:0/4095336411 conn(0x7faaf80ff910 msgr2=0x7faaf8101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 -- 172.21.15.114:0/4095336411 shutdown_connections 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.760+0000 7faafe400640 1 -- 172.21.15.114:0/4095336411 wait complete. 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.761+0000 7faafe400640 1 Processor -- start 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.761+0000 7faafe400640 1 -- start start 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafe400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf8103e30 0x7faaf819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafe400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf81047e0 0x7faaf819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafe400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 0x7faaf819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafe400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8113fc0 con 0x7faaf8108f60 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafe400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8113e40 con 0x7faaf8103e30 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafe400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8114140 con 0x7faaf81047e0 2024-09-19T06:50:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafda00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 0x7faaf819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafda00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 0x7faaf819ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37194/0 (socket says 172.21.15.114:37194) 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafd000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf8103e30 0x7faaf819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafda00640 1 -- 172.21.15.114:0/343963803 learned_addr learned my addr 172.21.15.114:0/343963803 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faaf7e00640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf81047e0 0x7faaf819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafda00640 1 -- 172.21.15.114:0/343963803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf81047e0 msgr2=0x7faaf819a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafda00640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faaf81047e0 0x7faaf819a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafda00640 1 -- 172.21.15.114:0/343963803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf8103e30 msgr2=0x7faaf819a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafda00640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faaf8103e30 0x7faaf819a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.762+0000 7faafda00640 1 -- 172.21.15.114:0/343963803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf819f3d0 con 0x7faaf8108f60 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.763+0000 7faafda00640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 0x7faaf819ecf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7faaf0031a20 tx=0x7faaf0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.763+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0003b70 con 0x7faaf8108f60 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.763+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf0003d10 con 0x7faaf8108f60 2024-09-19T06:50:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.763+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf819f660 con 0x7faaf8108f60 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.763+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf002fc40 con 0x7faaf8108f60 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.763+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf81a6fa0 con 0x7faaf8108f60 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.764+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faaf8062980 con 0x7faaf8108f60 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.766+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7faaf0004050 con 0x7faaf8108f60 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.767+0000 7faaf5600640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7faae00a3350 0x7faae00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.767+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7faaf00f0ff0 con 0x7faaf8108f60 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.767+0000 7faafd000640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7faae00a3350 0x7faae00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.767+0000 7faaf5600640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] conn(0x7faae00ae520 0x7faae00b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.768+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 --> [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaf0036e70 con 0x7faae00ae520 2024-09-19T06:50:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.768+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7faaf00f1420 con 0x7faaf8108f60 2024-09-19T06:50:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.768+0000 7faaf7e00640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] conn(0x7faae00ae520 0x7faae00b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.768+0000 7faafd000640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7faae00a3350 0x7faae00a5810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faaec009bd0 tx=0x7faaec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.768+0000 7faaf7e00640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] conn(0x7faae00ae520 0x7faae00b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.769+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 <== osd.11 v2:172.21.15.118:6824/694640175 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faaf0036e70 con 0x7faae00ae520 2024-09-19T06:50:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.783+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 --> [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faaf8062d80 con 0x7faae00ae520 2024-09-19T06:50:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faaf5600640 1 -- 172.21.15.114:0/343963803 <== osd.11 v2:172.21.15.118:6824/694640175 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7faaf8062d80 con 0x7faae00ae520 2024-09-19T06:50:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 >> [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] conn(0x7faae00ae520 msgr2=0x7faae00b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafe400640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] conn(0x7faae00ae520 0x7faae00b09c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7faae00a3350 msgr2=0x7faae00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafe400640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7faae00a3350 0x7faae00a5810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faaec009bd0 tx=0x7faaec005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 msgr2=0x7faaf819ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafe400640 1 --2- 172.21.15.114:0/343963803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faaf8108f60 0x7faaf819ecf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7faaf0031a20 tx=0x7faaf0031a50 comp rx=0 tx=0).stop 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafda00640 1 -- 172.21.15.114:0/343963803 reap_dead start 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 shutdown_connections 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.784+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 >> 172.21.15.114:0/343963803 conn(0x7faaf80ff910 msgr2=0x7faaf8100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.785+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 shutdown_connections 2024-09-19T06:50:37.378 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.785+0000 7faafe400640 1 -- 172.21.15.114:0/343963803 wait complete. 2024-09-19T06:50:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 -- 172.21.15.114:0/974572291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20103e30 msgr2=0x7f0a20104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 --2- 172.21.15.114:0/974572291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20103e30 0x7f0a20104210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009900 tx=0x7f0a1002a410 comp rx=0 tx=0).stop 2024-09-19T06:50:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 -- 172.21.15.114:0/974572291 shutdown_connections 2024-09-19T06:50:37.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 --2- 172.21.15.114:0/974572291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a20108f60 0x7f0a20110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 --2- 172.21.15.114:0/974572291 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a201047e0 0x7f0a20108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 --2- 172.21.15.114:0/974572291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20103e30 0x7f0a20104210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 -- 172.21.15.114:0/974572291 >> 172.21.15.114:0/974572291 conn(0x7f0a200ff8d0 msgr2=0x7f0a20101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 -- 172.21.15.114:0/974572291 shutdown_connections 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.980+0000 7f0a24a00640 1 -- 172.21.15.114:0/974572291 wait complete. 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.981+0000 7f0a24a00640 1 Processor -- start 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a24a00640 1 -- start start 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a24a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a20103e30 0x7f0a2019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a24a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a201047e0 0x7f0a2019a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a24a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20108f60 0x7f0a2019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a24a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20113fc0 con 0x7f0a20108f60 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a24a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20113e40 con 0x7f0a20103e30 2024-09-19T06:50:37.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a24a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20114140 con 0x7f0a201047e0 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a1fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20108f60 0x7f0a2019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a1fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20108f60 0x7f0a2019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48372/0 (socket says 172.21.15.114:48372) 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a17e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a201047e0 0x7f0a2019a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.982+0000 7f0a1fe00640 1 -- 172.21.15.114:0/3600451152 learned_addr learned my addr 172.21.15.114:0/3600451152 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1f400640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a20103e30 0x7f0a2019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1fe00640 1 -- 172.21.15.114:0/3600451152 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a201047e0 msgr2=0x7f0a2019a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1fe00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a201047e0 0x7f0a2019a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1fe00640 1 -- 172.21.15.114:0/3600451152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a20103e30 msgr2=0x7f0a2019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1fe00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a20103e30 0x7f0a2019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1fe00640 1 -- 172.21.15.114:0/3600451152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2019f430 con 0x7f0a20108f60 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a17e00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a201047e0 0x7f0a2019a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:37.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1fe00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20108f60 0x7f0a2019ed50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800eb10 tx=0x7f0a1800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18004b20 con 0x7f0a20108f60 2024-09-19T06:50:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a18004cc0 con 0x7f0a20108f60 2024-09-19T06:50:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.983+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18010760 con 0x7f0a20108f60 2024-09-19T06:50:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.984+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2019f720 con 0x7f0a20108f60 2024-09-19T06:50:37.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.984+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a201a7000 con 0x7f0a20108f60 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.984+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0a20062980 con 0x7f0a20108f60 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.987+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f0a18005020 con 0x7f0a20108f60 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.988+0000 7f0a1cc00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0a040a3350 0x7f0a040a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.988+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f0a180d0710 con 0x7f0a20108f60 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.988+0000 7f0a1f400640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0a040a3350 0x7f0a040a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.988+0000 7f0a1cc00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] conn(0x7f0a040ae520 0x7f0a040b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.988+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 --> [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a040b1090 con 0x7f0a040ae520 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.988+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f0a180d0ac0 con 0x7f0a20108f60 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.989+0000 7f0a17e00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] conn(0x7f0a040ae520 0x7f0a040b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.989+0000 7f0a1f400640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0a040a3350 0x7f0a040a5810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009230 tx=0x7f0a100058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:37.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.989+0000 7f0a17e00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] conn(0x7f0a040ae520 0x7f0a040b09c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:36.990+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 <== osd.4 v2:172.21.15.116:6808/2553198252 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0a040b1090 con 0x7f0a040ae520 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.004+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 --> [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0a20062e00 con 0x7f0a040ae520 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a1cc00640 1 -- 172.21.15.114:0/3600451152 <== osd.4 v2:172.21.15.116:6808/2553198252 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0a20062e00 con 0x7f0a040ae520 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 >> [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] conn(0x7f0a040ae520 msgr2=0x7f0a040b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] conn(0x7f0a040ae520 0x7f0a040b09c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0a040a3350 msgr2=0x7f0a040a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0a040a3350 0x7f0a040a5810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009230 tx=0x7f0a100058d0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20108f60 msgr2=0x7f0a2019ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 --2- 172.21.15.114:0/3600451152 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0a20108f60 0x7f0a2019ed50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800eb10 tx=0x7f0a1800c2d0 comp rx=0 tx=0).stop 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a1fe00640 1 -- 172.21.15.114:0/3600451152 reap_dead start 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 shutdown_connections 2024-09-19T06:50:37.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 >> 172.21.15.114:0/3600451152 conn(0x7f0a200ff8d0 msgr2=0x7f0a20100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:37.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 shutdown_connections 2024-09-19T06:50:37.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:37.005+0000 7f0a24a00640 1 -- 172.21.15.114:0/3600451152 wait complete. 2024-09-19T06:50:38.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:38 smithi114 ceph-mon[30820]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:50:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:38 smithi116 ceph-mon[35411]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:50:38.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:38 smithi118 ceph-mon[35527]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:50:38.837 INFO:teuthology.orchestra.run.smithi114.stdout:335007449119 2024-09-19T06:50:38.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-19T06:50:39.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 -- 172.21.15.114:0/1316523754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40101de0 msgr2=0x7f4e401021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:39.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 --2- 172.21.15.114:0/1316523754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40101de0 0x7f4e401021c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4e30009900 tx=0x7f4e3002a3f0 comp rx=0 tx=0).stop 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 -- 172.21.15.114:0/1316523754 shutdown_connections 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 --2- 172.21.15.114:0/1316523754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e40106f10 0x7f4e4010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 --2- 172.21.15.114:0/1316523754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e40102790 0x7f4e401067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 --2- 172.21.15.114:0/1316523754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40101de0 0x7f4e401021c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 -- 172.21.15.114:0/1316523754 >> 172.21.15.114:0/1316523754 conn(0x7f4e400fd8c0 msgr2=0x7f4e400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 -- 172.21.15.114:0/1316523754 shutdown_connections 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.239+0000 7f4e47000640 1 -- 172.21.15.114:0/1316523754 wait complete. 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.240+0000 7f4e47000640 1 Processor -- start 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.240+0000 7f4e47000640 1 -- start start 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.240+0000 7f4e47000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e40101de0 0x7f4e400744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.240+0000 7f4e47000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40102790 0x7f4e40074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.240+0000 7f4e47000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e40106f10 0x7f4e40074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.240+0000 7f4e47000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4e40112000 con 0x7f4e40102790 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.240+0000 7f4e47000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e40111e80 con 0x7f4e40101de0 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.240+0000 7f4e47000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4e40112180 con 0x7f4e40106f10 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e40101de0 0x7f4e400744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e45000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e40106f10 0x7f4e40074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e40101de0 0x7f4e400744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:56754/0 (socket says 172.21.15.114:56754) 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 -- 172.21.15.114:0/1733822520 learned_addr learned my addr 172.21.15.114:0/1733822520 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e37e00640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40102790 0x7f4e40074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 -- 172.21.15.114:0/1733822520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e40106f10 msgr2=0x7f4e40074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e40106f10 0x7f4e40074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 -- 172.21.15.114:0/1733822520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40102790 msgr2=0x7f4e40074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4e40102790 0x7f4e40074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 -- 172.21.15.114:0/1733822520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e401a6980 con 0x7f4e40101de0 2024-09-19T06:50:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.241+0000 7f4e3fe00640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e40101de0 0x7f4e400744f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e30004770 tx=0x7f4e30003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.242+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3003f070 con 0x7f4e40101de0 2024-09-19T06:50:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.242+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e401a6c10 con 0x7f4e40101de0 2024-09-19T06:50:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.242+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e30031ca0 con 0x7f4e40101de0 2024-09-19T06:50:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.242+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e30035030 con 0x7f4e40101de0 2024-09-19T06:50:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.242+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e401a7140 con 0x7f4e40101de0 2024-09-19T06:50:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.243+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4e400628c0 con 0x7f4e40101de0 2024-09-19T06:50:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.245+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f4e30042070 con 0x7f4e40101de0 2024-09-19T06:50:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.246+0000 7f4e3d600640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e240a32d0 0x7f4e240a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.246+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4e300f0910 con 0x7f4e40101de0 2024-09-19T06:50:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.246+0000 7f4e37e00640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e240a32d0 0x7f4e240a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.246+0000 7f4e3d600640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] conn(0x7f4e240ae4a0 0x7f4e240b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.246+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 --> [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e30038e90 con 0x7f4e240ae4a0 2024-09-19T06:50:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.246+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f4e300f0cc0 con 0x7f4e40101de0 2024-09-19T06:50:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.247+0000 7f4e45000640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] conn(0x7f4e240ae4a0 0x7f4e240b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.247+0000 7f4e37e00640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e240a32d0 0x7f4e240a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4e40075f80 tx=0x7f4e2800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.247+0000 7f4e45000640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] conn(0x7f4e240ae4a0 0x7f4e240b0940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.248+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 <== osd.10 v2:172.21.15.116:6824/1825955684 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4e30038e90 con 0x7f4e240ae4a0 2024-09-19T06:50:39.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.263+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 --> [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4e40062cc0 con 0x7f4e240ae4a0 2024-09-19T06:50:39.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.263+0000 7f4e3d600640 1 -- 172.21.15.114:0/1733822520 <== osd.10 v2:172.21.15.116:6824/1825955684 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4e40062cc0 con 0x7f4e240ae4a0 2024-09-19T06:50:39.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 >> [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] conn(0x7f4e240ae4a0 msgr2=0x7f4e240b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:39.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] conn(0x7f4e240ae4a0 0x7f4e240b0940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e240a32d0 msgr2=0x7f4e240a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4e240a32d0 0x7f4e240a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4e40075f80 tx=0x7f4e2800b390 comp rx=0 tx=0).stop 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e40101de0 msgr2=0x7f4e400744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 --2- 172.21.15.114:0/1733822520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e40101de0 0x7f4e400744f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e30004770 tx=0x7f4e30003c10 comp rx=0 tx=0).stop 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e45000640 1 -- 172.21.15.114:0/1733822520 reap_dead start 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 shutdown_connections 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 >> 172.21.15.114:0/1733822520 conn(0x7f4e400fd8c0 msgr2=0x7f4e400febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 shutdown_connections 2024-09-19T06:50:39.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.264+0000 7f4e47000640 1 -- 172.21.15.114:0/1733822520 wait complete. 2024-09-19T06:50:39.960 INFO:teuthology.orchestra.run.smithi114.stdout:77309411407 2024-09-19T06:50:39.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-19T06:50:40.105 INFO:teuthology.orchestra.run.smithi114.stdout:34359738458 2024-09-19T06:50:40.105 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-19T06:50:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:40 smithi116 ceph-mon[35411]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:50:40.461 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:40 smithi114 ceph-mon[30820]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:50:40.464 INFO:teuthology.orchestra.run.smithi114.stdout:219043332147 2024-09-19T06:50:40.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-19T06:50:40.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:40 smithi118 ceph-mon[35527]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:50:40.961 INFO:teuthology.orchestra.run.smithi114.stdout:141733920835 2024-09-19T06:50:40.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-19T06:50:41.415 INFO:teuthology.orchestra.run.smithi114.stdout:283467841577 2024-09-19T06:50:41.416 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-19T06:50:41.737 INFO:teuthology.orchestra.run.smithi114.stdout:249108103214 2024-09-19T06:50:41.737 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-19T06:50:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 -- 172.21.15.114:0/2889241917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 msgr2=0x7f4cb80748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 --2- 172.21.15.114:0/2889241917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 0x7f4cb80748f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8009900 tx=0x7f4ca802a480 comp rx=0 tx=0).stop 2024-09-19T06:50:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 -- 172.21.15.114:0/2889241917 shutdown_connections 2024-09-19T06:50:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 --2- 172.21.15.114:0/2889241917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb810e280 0x7f4cb8110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 --2- 172.21.15.114:0/2889241917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb8074e30 0x7f4cb810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 --2- 172.21.15.114:0/2889241917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 0x7f4cb80748f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:41.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 -- 172.21.15.114:0/2889241917 >> 172.21.15.114:0/2889241917 conn(0x7f4cb80fd740 msgr2=0x7f4cb80ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 -- 172.21.15.114:0/2889241917 shutdown_connections 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.487+0000 7f4cbd000640 1 -- 172.21.15.114:0/2889241917 wait complete. 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cbd000640 1 Processor -- start 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cbd000640 1 -- start start 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cbd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 0x7f4cb819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cbd000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb8074e30 0x7f4cb819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cbd000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb810e280 0x7f4cb819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cbd000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8113fb0 con 0x7f4cb8074510 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cbd000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8113e30 con 0x7f4cb8074e30 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cbd000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb8114130 con 0x7f4cb810e280 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cb7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 0x7f4cb819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:41.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cb7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 0x7f4cb819a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48404/0 (socket says 172.21.15.114:48404) 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.488+0000 7f4cb7400640 1 -- 172.21.15.114:0/2850493527 learned_addr learned my addr 172.21.15.114:0/2850493527 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cb7e00640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb810e280 0x7f4cb819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cb6a00640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb8074e30 0x7f4cb819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cb7400640 1 -- 172.21.15.114:0/2850493527 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb810e280 msgr2=0x7f4cb819ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cb7400640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4cb810e280 0x7f4cb819ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cb7400640 1 -- 172.21.15.114:0/2850493527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb8074e30 msgr2=0x7f4cb819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cb7400640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cb8074e30 0x7f4cb819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cb7400640 1 -- 172.21.15.114:0/2850493527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb819f520 con 0x7f4cb8074510 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cb7400640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 0x7f4cb819a570 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8006010 tx=0x7f4ca8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.489+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca803f030 con 0x7f4cb8074510 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.490+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca8004440 con 0x7f4cb8074510 2024-09-19T06:50:41.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.490+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8035070 con 0x7f4cb8074510 2024-09-19T06:50:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.490+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb819f7b0 con 0x7f4cb8074510 2024-09-19T06:50:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.490+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb819fc40 con 0x7f4cb8074510 2024-09-19T06:50:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.491+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4cb8062900 con 0x7f4cb8074510 2024-09-19T06:50:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.493+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f4ca8042030 con 0x7f4cb8074510 2024-09-19T06:50:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.494+0000 7f4cafe00640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4c9c0a32d0 0x7f4c9c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.494+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4ca80f08e0 con 0x7f4cb8074510 2024-09-19T06:50:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.494+0000 7f4cb6a00640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4c9c0a32d0 0x7f4c9c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:41.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.494+0000 7f4cafe00640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] conn(0x7f4c9c0ae4a0 0x7f4c9c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.494+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 --> [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c9c0b1010 con 0x7f4c9c0ae4a0 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.494+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f4ca80f0d30 con 0x7f4cb8074510 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.495+0000 7f4cb7e00640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] conn(0x7f4c9c0ae4a0 0x7f4c9c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.495+0000 7f4cb6a00640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4c9c0a32d0 0x7f4c9c0a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80ffd60 tx=0x7f4ca4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.495+0000 7f4cb7e00640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] conn(0x7f4c9c0ae4a0 0x7f4c9c0b0940 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.496+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 <== osd.5 v2:172.21.15.118:6808/2474489127 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c9c0b1010 con 0x7f4c9c0ae4a0 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.511+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 --> [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4cb8062d80 con 0x7f4c9c0ae4a0 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.511+0000 7f4cafe00640 1 -- 172.21.15.114:0/2850493527 <== osd.5 v2:172.21.15.118:6808/2474489127 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4cb8062d80 con 0x7f4c9c0ae4a0 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 >> [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] conn(0x7f4c9c0ae4a0 msgr2=0x7f4c9c0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] conn(0x7f4c9c0ae4a0 0x7f4c9c0b0940 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4c9c0a32d0 msgr2=0x7f4c9c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4c9c0a32d0 0x7f4c9c0a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80ffd60 tx=0x7f4ca4005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:41.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 msgr2=0x7f4cb819a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 --2- 172.21.15.114:0/2850493527 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4cb8074510 0x7f4cb819a570 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8006010 tx=0x7f4ca8004040 comp rx=0 tx=0).stop 2024-09-19T06:50:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cb7e00640 1 -- 172.21.15.114:0/2850493527 reap_dead start 2024-09-19T06:50:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 shutdown_connections 2024-09-19T06:50:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 >> 172.21.15.114:0/2850493527 conn(0x7f4cb80fd740 msgr2=0x7f4cb80feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 shutdown_connections 2024-09-19T06:50:41.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:39.512+0000 7f4cbd000640 1 -- 172.21.15.114:0/2850493527 wait complete. 2024-09-19T06:50:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:42 smithi114 ceph-mon[30820]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:50:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:42 smithi118 ceph-mon[35527]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:50:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:42 smithi116 ceph-mon[35411]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:50:42.802 INFO:teuthology.orchestra.run.smithi114.stdout:304942678051 2024-09-19T06:50:42.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-19T06:50:43.115 INFO:teuthology.orchestra.run.smithi114.stdout:163208757313 2024-09-19T06:50:43.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-19T06:50:43.144 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:43.235 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:43.308 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:43.309 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:44.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 -- 172.21.15.114:0/2487738351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc10e260 msgr2=0x7fb5cc110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:44.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2487738351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc10e260 0x7fb5cc110650 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009ca0 tx=0x7fb5c402a950 comp rx=0 tx=0).stop 2024-09-19T06:50:44.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 -- 172.21.15.114:0/2487738351 shutdown_connections 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2487738351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc10e260 0x7fb5cc110650 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2487738351 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5cc074ee0 0x7fb5cc10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2487738351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc0745c0 0x7fb5cc0749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 -- 172.21.15.114:0/2487738351 >> 172.21.15.114:0/2487738351 conn(0x7fb5cc0fd740 msgr2=0x7fb5cc0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 -- 172.21.15.114:0/2487738351 shutdown_connections 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.139+0000 7fb5d1800640 1 -- 172.21.15.114:0/2487738351 wait complete. 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.140+0000 7fb5d1800640 1 Processor -- start 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.141+0000 7fb5d1800640 1 -- start start 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.141+0000 7fb5d1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc0745c0 0x7fb5cc19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.141+0000 7fb5d1800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5cc074ee0 0x7fb5cc19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.141+0000 7fb5d1800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc10e260 0x7fb5cc19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:44.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.141+0000 7fb5d1800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc113f70 con 0x7fb5cc0745c0 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.141+0000 7fb5d1800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc113df0 con 0x7fb5cc10e260 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.141+0000 7fb5d1800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc1140f0 con 0x7fb5cc074ee0 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5cc074ee0 0x7fb5cc19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5cbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc0745c0 0x7fb5cc19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5cc074ee0 0x7fb5cc19aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:51270/0 (socket says 172.21.15.114:51270) 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 -- 172.21.15.114:0/2099121662 learned_addr learned my addr 172.21.15.114:0/2099121662 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5d0e00640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc10e260 0x7fb5cc19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 -- 172.21.15.114:0/2099121662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc10e260 msgr2=0x7fb5cc19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc10e260 0x7fb5cc19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 -- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc0745c0 msgr2=0x7fb5cc19a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc0745c0 0x7fb5cc19a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.146 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 -- 172.21.15.114:0/2099121662 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc19f4e0 con 0x7fb5cc074ee0 2024-09-19T06:50:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.142+0000 7fb5c3e00640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5cc074ee0 0x7fb5cc19aa70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400e9f0 tx=0x7fb5b400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.143+0000 7fb5c9600640 1 -- 172.21.15.114:0/2099121662 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400cd80 con 0x7fb5cc074ee0 2024-09-19T06:50:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.143+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc19f7d0 con 0x7fb5cc074ee0 2024-09-19T06:50:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.143+0000 7fb5c9600640 1 -- 172.21.15.114:0/2099121662 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5b4010910 con 0x7fb5cc074ee0 2024-09-19T06:50:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.143+0000 7fb5c9600640 1 -- 172.21.15.114:0/2099121662 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b40188f0 con 0x7fb5cc074ee0 2024-09-19T06:50:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.143+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc1a70b0 con 0x7fb5cc074ee0 2024-09-19T06:50:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.144+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5cc0749a0 con 0x7fb5cc074ee0 2024-09-19T06:50:44.147 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.146+0000 7fb5c9600640 1 -- 172.21.15.114:0/2099121662 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fb5b4010ac0 con 0x7fb5cc074ee0 2024-09-19T06:50:44.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.147+0000 7fb5c9600640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb5b00a32d0 0x7fb5b00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:44.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.147+0000 7fb5c9600640 1 -- 172.21.15.114:0/2099121662 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb5b40d0570 con 0x7fb5cc074ee0 2024-09-19T06:50:44.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.147+0000 7fb5cbe00640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb5b00a32d0 0x7fb5b00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:44.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.148+0000 7fb5cbe00640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb5b00a32d0 0x7fb5b00a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0095f0 tx=0x7fb5bc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:44.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.149+0000 7fb5c9600640 1 -- 172.21.15.114:0/2099121662 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fb5b409c720 con 0x7fb5cc074ee0 2024-09-19T06:50:44.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.337+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb5cc19b890 con 0x7fb5cc074ee0 2024-09-19T06:50:44.366 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:44 smithi114 ceph-mon[30820]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:44.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.337+0000 7fb5c9600640 1 -- 172.21.15.114:0/2099121662 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb5b40a0a20 con 0x7fb5cc074ee0 2024-09-19T06:50:44.367 INFO:teuthology.orchestra.run.smithi114.stdout:197568495673 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb5b00a32d0 msgr2=0x7fb5b00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb5b00a32d0 0x7fb5b00a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0095f0 tx=0x7fb5bc005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5cc074ee0 msgr2=0x7fb5cc19aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5cc074ee0 0x7fb5cc19aa70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400e9f0 tx=0x7fb5b400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 shutdown_connections 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fb5b00a32d0 0x7fb5b00a5790 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5cc10e260 0x7fb5cc19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5cc074ee0 0x7fb5cc19aa70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 --2- 172.21.15.114:0/2099121662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb5cc0745c0 0x7fb5cc19a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:44.368 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 >> 172.21.15.114:0/2099121662 conn(0x7fb5cc0fd740 msgr2=0x7fb5cc0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:44.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 shutdown_connections 2024-09-19T06:50:44.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:44.340+0000 7fb5d1800640 1 -- 172.21.15.114:0/2099121662 wait complete. 2024-09-19T06:50:44.436 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:44.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:44 smithi118 ceph-mon[35527]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:44 smithi116 ceph-mon[35411]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:44.803 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:44.803 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:44.804 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:44.804 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:44.805 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:45.392 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:45 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2099121662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 -- 172.21.15.114:0/1363743962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c0745c0 msgr2=0x7f558c0749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 --2- 172.21.15.114:0/1363743962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c0745c0 0x7f558c0749a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f557c009900 tx=0x7f557c02a480 comp rx=0 tx=0).stop 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 -- 172.21.15.114:0/1363743962 shutdown_connections 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 --2- 172.21.15.114:0/1363743962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f558c10e260 0x7f558c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 --2- 172.21.15.114:0/1363743962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c074ee0 0x7f558c10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 --2- 172.21.15.114:0/1363743962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c0745c0 0x7f558c0749a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 -- 172.21.15.114:0/1363743962 >> 172.21.15.114:0/1363743962 conn(0x7f558c0fd740 msgr2=0x7f558c0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 -- 172.21.15.114:0/1363743962 shutdown_connections 2024-09-19T06:50:45.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.399+0000 7f5591800640 1 -- 172.21.15.114:0/1363743962 wait complete. 2024-09-19T06:50:45.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.400+0000 7f5591800640 1 Processor -- start 2024-09-19T06:50:45.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.400+0000 7f5591800640 1 -- start start 2024-09-19T06:50:45.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5591800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c0745c0 0x7f558c19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:45.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5591800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f558c074ee0 0x7f558c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:45.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5591800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c10e260 0x7f558c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:45.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5591800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f558c113fb0 con 0x7f558c10e260 2024-09-19T06:50:45.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5591800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f558c113e30 con 0x7f558c074ee0 2024-09-19T06:50:45.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5591800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f558c114130 con 0x7f558c0745c0 2024-09-19T06:50:45.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5590e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c10e260 0x7f558c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:45.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5590e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c10e260 0x7f558c19ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48430/0 (socket says 172.21.15.114:48430) 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5590e00640 1 -- 172.21.15.114:0/472589511 learned_addr learned my addr 172.21.15.114:0/472589511 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f558be00640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c0745c0 0x7f558c19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.401+0000 7f5583400640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f558c074ee0 0x7f558c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.402+0000 7f558be00640 1 -- 172.21.15.114:0/472589511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f558c074ee0 msgr2=0x7f558c19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.402+0000 7f558be00640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f558c074ee0 0x7f558c19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.402+0000 7f558be00640 1 -- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c10e260 msgr2=0x7f558c19ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.402+0000 7f558be00640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c10e260 0x7f558c19ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.402+0000 7f558be00640 1 -- 172.21.15.114:0/472589511 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c19f460 con 0x7f558c0745c0 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.402+0000 7f5590e00640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c10e260 0x7f558c19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.402+0000 7f558be00640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c0745c0 0x7f558c19a4b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f557c005940 tx=0x7f557c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.402+0000 7f5589600640 1 -- 172.21.15.114:0/472589511 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c03f030 con 0x7f558c0745c0 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.403+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558c19f6f0 con 0x7f558c0745c0 2024-09-19T06:50:45.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.403+0000 7f5589600640 1 -- 172.21.15.114:0/472589511 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f557c003ad0 con 0x7f558c0745c0 2024-09-19T06:50:45.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.403+0000 7f5589600640 1 -- 172.21.15.114:0/472589511 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c035070 con 0x7f558c0745c0 2024-09-19T06:50:45.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.403+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f558c1a7030 con 0x7f558c0745c0 2024-09-19T06:50:45.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.404+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f558c075c70 con 0x7f558c0745c0 2024-09-19T06:50:45.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.406+0000 7f5589600640 1 -- 172.21.15.114:0/472589511 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f557c042030 con 0x7f558c0745c0 2024-09-19T06:50:45.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.407+0000 7f5589600640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55700a32d0 0x7f55700a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:45.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.407+0000 7f5589600640 1 -- 172.21.15.114:0/472589511 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f557c0f06f0 con 0x7f558c0745c0 2024-09-19T06:50:45.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.407+0000 7f5583400640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55700a32d0 0x7f55700a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:45.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.408+0000 7f5583400640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55700a32d0 0x7f55700a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f558c10f660 tx=0x7f5574005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:45.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.409+0000 7f5589600640 1 -- 172.21.15.114:0/472589511 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f557c0bc8a0 con 0x7f558c0745c0 2024-09-19T06:50:45.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:45 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2099121662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-19T06:50:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.594+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f558c1093e0 con 0x7f558c0745c0 2024-09-19T06:50:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.595+0000 7f5589600640 1 -- 172.21.15.114:0/472589511 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f557c0c0ba0 con 0x7f558c0745c0 2024-09-19T06:50:45.597 INFO:teuthology.orchestra.run.smithi114.stdout:141733920837 2024-09-19T06:50:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55700a32d0 msgr2=0x7f55700a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55700a32d0 0x7f55700a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f558c10f660 tx=0x7f5574005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c0745c0 msgr2=0x7f558c19a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c0745c0 0x7f558c19a4b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f557c005940 tx=0x7f557c004270 comp rx=0 tx=0).stop 2024-09-19T06:50:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 shutdown_connections 2024-09-19T06:50:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f55700a32d0 0x7f55700a5790 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f558c10e260 0x7f558c19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f558c074ee0 0x7f558c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 --2- 172.21.15.114:0/472589511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f558c0745c0 0x7f558c19a4b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 >> 172.21.15.114:0/472589511 conn(0x7f558c0fd740 msgr2=0x7f558c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 shutdown_connections 2024-09-19T06:50:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.598+0000 7f5591800640 1 -- 172.21.15.114:0/472589511 wait complete. 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 -- 172.21.15.114:0/690396915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc4102d30 msgr2=0x7effc4103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 --2- 172.21.15.114:0/690396915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc4102d30 0x7effc4103110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7effb8009930 tx=0x7effb802a420 comp rx=0 tx=0).stop 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 -- 172.21.15.114:0/690396915 shutdown_connections 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 --2- 172.21.15.114:0/690396915 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effc41082a0 0x7effc410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 --2- 172.21.15.114:0/690396915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc41036e0 0x7effc4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 --2- 172.21.15.114:0/690396915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc4102d30 0x7effc4103110 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 -- 172.21.15.114:0/690396915 >> 172.21.15.114:0/690396915 conn(0x7effc40fc7d0 msgr2=0x7effc40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 -- 172.21.15.114:0/690396915 shutdown_connections 2024-09-19T06:50:45.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.649+0000 7effcba00640 1 -- 172.21.15.114:0/690396915 wait complete. 2024-09-19T06:50:45.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.650+0000 7effcba00640 1 Processor -- start 2024-09-19T06:50:45.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effcba00640 1 -- start start 2024-09-19T06:50:45.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effcba00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effc4102d30 0x7effc41a48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:45.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effcba00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc41036e0 0x7effc41a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:45.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effcba00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc41082a0 0x7effc410de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:45.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effcba00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7effc41120e0 con 0x7effc41036e0 2024-09-19T06:50:45.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effcba00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7effc4111f60 con 0x7effc41082a0 2024-09-19T06:50:45.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effcba00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7effc4112260 con 0x7effc4102d30 2024-09-19T06:50:45.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effc9000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effc4102d30 0x7effc41a48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:45.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effc3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc41036e0 0x7effc41a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:45.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effc9000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effc4102d30 0x7effc41a48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:51318/0 (socket says 172.21.15.114:51318) 2024-09-19T06:50:45.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.651+0000 7effc9000640 1 -- 172.21.15.114:0/1862727398 learned_addr learned my addr 172.21.15.114:0/1862727398 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:45.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc9a00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc41082a0 0x7effc410de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:45.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc9000640 1 -- 172.21.15.114:0/1862727398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc41082a0 msgr2=0x7effc410de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc9000640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc41082a0 0x7effc410de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc9000640 1 -- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc41036e0 msgr2=0x7effc41a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc9000640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc41036e0 0x7effc41a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc9000640 1 -- 172.21.15.114:0/1862727398 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc410e640 con 0x7effc4102d30 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc3e00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc41036e0 0x7effc41a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc9000640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effc4102d30 0x7effc41a48f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7effb802a940 tx=0x7effb8003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effc1600640 1 -- 172.21.15.114:0/1862727398 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb803f070 con 0x7effc4102d30 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.652+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc410e8d0 con 0x7effc4102d30 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.653+0000 7effc1600640 1 -- 172.21.15.114:0/1862727398 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effb8031c90 con 0x7effc4102d30 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.653+0000 7effc1600640 1 -- 172.21.15.114:0/1862727398 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb8035030 con 0x7effc4102d30 2024-09-19T06:50:45.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.653+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc41ab6e0 con 0x7effc4102d30 2024-09-19T06:50:45.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.654+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc406a850 con 0x7effc4102d30 2024-09-19T06:50:45.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.656+0000 7effc1600640 1 -- 172.21.15.114:0/1862727398 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7effb8042070 con 0x7effc4102d30 2024-09-19T06:50:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.657+0000 7effc1600640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7effac0a3350 0x7effac0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.657+0000 7effc1600640 1 -- 172.21.15.114:0/1862727398 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7effb80f0c30 con 0x7effc4102d30 2024-09-19T06:50:45.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.657+0000 7effc3e00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7effac0a3350 0x7effac0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:45.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.658+0000 7effc3e00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7effac0a3350 0x7effac0a5810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7effb00095f0 tx=0x7effb0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:45.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.659+0000 7effc1600640 1 -- 172.21.15.114:0/1862727398 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7effb80bd430 con 0x7effc4102d30 2024-09-19T06:50:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:45 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2099121662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-19T06:50:45.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.841+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7effc4103110 con 0x7effc4102d30 2024-09-19T06:50:45.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.842+0000 7effc1600640 1 -- 172.21.15.114:0/1862727398 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7effb80bcd30 con 0x7effc4102d30 2024-09-19T06:50:45.843 INFO:teuthology.orchestra.run.smithi114.stdout:77309411410 2024-09-19T06:50:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7effac0a3350 msgr2=0x7effac0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7effac0a3350 0x7effac0a5810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7effb00095f0 tx=0x7effb0005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effc4102d30 msgr2=0x7effc41a48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:45.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effc4102d30 0x7effc41a48f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7effb802a940 tx=0x7effb8003b20 comp rx=0 tx=0).stop 2024-09-19T06:50:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 shutdown_connections 2024-09-19T06:50:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7effac0a3350 0x7effac0a5810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc41082a0 0x7effc410de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7effc41036e0 0x7effc41a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 --2- 172.21.15.114:0/1862727398 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7effc4102d30 0x7effc41a48f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 >> 172.21.15.114:0/1862727398 conn(0x7effc40fc7d0 msgr2=0x7effc40fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 shutdown_connections 2024-09-19T06:50:45.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:45.844+0000 7effcba00640 1 -- 172.21.15.114:0/1862727398 wait complete. 2024-09-19T06:50:46.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:46 smithi118 ceph-mon[35527]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:50:46.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:46 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/472589511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-19T06:50:46.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:46 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1862727398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-19T06:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:46 smithi114 ceph-mon[30820]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:46 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/472589511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-19T06:50:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:46 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1862727398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-19T06:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:46 smithi116 ceph-mon[35411]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:46 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/472589511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-19T06:50:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:46 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1862727398' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-19T06:50:47.531 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495671 got 197568495673 for osd.6 2024-09-19T06:50:47.531 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:47.789 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411410 for osd.2 2024-09-19T06:50:47.789 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:47.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 -- 172.21.15.114:0/1158482162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468101dd0 msgr2=0x7fe4681021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:47.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 --2- 172.21.15.114:0/1158482162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468101dd0 0x7fe4681021b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe458009900 tx=0x7fe45802a3f0 comp rx=0 tx=0).stop 2024-09-19T06:50:47.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 -- 172.21.15.114:0/1158482162 shutdown_connections 2024-09-19T06:50:47.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 --2- 172.21.15.114:0/1158482162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe468106f00 0x7fe46810ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:47.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 --2- 172.21.15.114:0/1158482162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe468102780 0x7fe4681067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:47.974 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 --2- 172.21.15.114:0/1158482162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468101dd0 0x7fe4681021b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 -- 172.21.15.114:0/1158482162 >> 172.21.15.114:0/1158482162 conn(0x7fe4680fd8b0 msgr2=0x7fe4680ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 -- 172.21.15.114:0/1158482162 shutdown_connections 2024-09-19T06:50:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.972+0000 7fe46f800640 1 -- 172.21.15.114:0/1158482162 wait complete. 2024-09-19T06:50:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.973+0000 7fe46f800640 1 Processor -- start 2024-09-19T06:50:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.973+0000 7fe46f800640 1 -- start start 2024-09-19T06:50:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46f800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe468101dd0 0x7fe46819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46f800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe468102780 0x7fe46819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46f800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468106f00 0x7fe46819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:47.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46f800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe468111f60 con 0x7fe468106f00 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46f800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe468111de0 con 0x7fe468102780 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46f800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe4681120e0 con 0x7fe468101dd0 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468106f00 0x7fe46819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe468101dd0 0x7fe46819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468106f00 0x7fe46819eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48362/0 (socket says 172.21.15.114:48362) 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe468101dd0 0x7fe46819a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:48452/0 (socket says 172.21.15.114:48452) 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe46d800640 1 -- 172.21.15.114:0/1248280195 learned_addr learned my addr 172.21.15.114:0/1248280195 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.974+0000 7fe45fe00640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe468102780 0x7fe46819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe45fe00640 1 -- 172.21.15.114:0/1248280195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe468101dd0 msgr2=0x7fe46819a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe45fe00640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe468101dd0 0x7fe46819a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe45fe00640 1 -- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468106f00 msgr2=0x7fe46819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe45fe00640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468106f00 0x7fe46819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe45fe00640 1 -- 172.21.15.114:0/1248280195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46819f5b0 con 0x7fe468102780 2024-09-19T06:50:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe46d800640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468106f00 0x7fe46819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe45fe00640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe468102780 0x7fe46819ab40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe45000e9f0 tx=0x7fe45000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe466000640 1 -- 172.21.15.114:0/1248280195 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45000ccc0 con 0x7fe468102780 2024-09-19T06:50:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe466000640 1 -- 172.21.15.114:0/1248280195 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe450010910 con 0x7fe468102780 2024-09-19T06:50:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe466000640 1 -- 172.21.15.114:0/1248280195 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450018960 con 0x7fe468102780 2024-09-19T06:50:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46819f8a0 con 0x7fe468102780 2024-09-19T06:50:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.975+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4681a7180 con 0x7fe468102780 2024-09-19T06:50:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.976+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4681021b0 con 0x7fe468102780 2024-09-19T06:50:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.979+0000 7fe466000640 1 -- 172.21.15.114:0/1248280195 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fe450010ac0 con 0x7fe468102780 2024-09-19T06:50:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.980+0000 7fe466000640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe44c0a32d0 0x7fe44c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.980+0000 7fe466000640 1 -- 172.21.15.114:0/1248280195 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe4500d0650 con 0x7fe468102780 2024-09-19T06:50:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.980+0000 7fe46ce00640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe44c0a32d0 0x7fe44c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.981+0000 7fe46ce00640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe44c0a32d0 0x7fe44c0a5790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe458009230 tx=0x7fe4580058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.982+0000 7fe466000640 1 -- 172.21.15.114:0/1248280195 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe45009c860 con 0x7fe468102780 2024-09-19T06:50:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.176+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe4681a73e0 con 0x7fe468102780 2024-09-19T06:50:48.177 INFO:teuthology.orchestra.run.smithi114.stdout:219043332150 2024-09-19T06:50:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.176+0000 7fe466000640 1 -- 172.21.15.114:0/1248280195 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe4500a0b60 con 0x7fe468102780 2024-09-19T06:50:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe44c0a32d0 msgr2=0x7fe44c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe44c0a32d0 0x7fe44c0a5790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe458009230 tx=0x7fe4580058d0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe468102780 msgr2=0x7fe46819ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:48.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe468102780 0x7fe46819ab40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe45000e9f0 tx=0x7fe45000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 shutdown_connections 2024-09-19T06:50:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe44c0a32d0 0x7fe44c0a5790 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe468106f00 0x7fe46819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe468102780 0x7fe46819ab40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 --2- 172.21.15.114:0/1248280195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe468101dd0 0x7fe46819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 >> 172.21.15.114:0/1248280195 conn(0x7fe4680fd8b0 msgr2=0x7fe4680feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 shutdown_connections 2024-09-19T06:50:48.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.179+0000 7fe46f800640 1 -- 172.21.15.114:0/1248280195 wait complete. 2024-09-19T06:50:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:48 smithi114 ceph-mon[30820]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:50:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:48 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3711835490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-19T06:50:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:48 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1248280195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-19T06:50:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:48 smithi118 ceph-mon[35527]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:50:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:48 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3711835490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-19T06:50:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:48 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1248280195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-19T06:50:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:48 smithi116 ceph-mon[35411]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:50:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:48 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3711835490' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-19T06:50:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:48 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1248280195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-19T06:50:48.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.873+0000 7f53d4600640 1 -- 172.21.15.114:0/4108611768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc10e3c0 msgr2=0x7f53cc110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:48.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.873+0000 7f53d4600640 1 --2- 172.21.15.114:0/4108611768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc10e3c0 0x7f53cc110810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009900 tx=0x7f53c402a480 comp rx=0 tx=0).stop 2024-09-19T06:50:48.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.873+0000 7f53d4600640 1 -- 172.21.15.114:0/4108611768 shutdown_connections 2024-09-19T06:50:48.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.873+0000 7f53d4600640 1 --2- 172.21.15.114:0/4108611768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc10e3c0 0x7f53cc110810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.873+0000 7f53d4600640 1 --2- 172.21.15.114:0/4108611768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53cc074f40 0x7f53cc10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.874+0000 7f53d4600640 1 --2- 172.21.15.114:0/4108611768 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53cc074620 0x7f53cc074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.874+0000 7f53d4600640 1 -- 172.21.15.114:0/4108611768 >> 172.21.15.114:0/4108611768 conn(0x7f53cc0fd8c0 msgr2=0x7f53cc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.874+0000 7f53d4600640 1 -- 172.21.15.114:0/4108611768 shutdown_connections 2024-09-19T06:50:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.874+0000 7f53d4600640 1 -- 172.21.15.114:0/4108611768 wait complete. 2024-09-19T06:50:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d4600640 1 Processor -- start 2024-09-19T06:50:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d4600640 1 -- start start 2024-09-19T06:50:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d4600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53cc074620 0x7f53cc19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d4600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc074f40 0x7f53cc19ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d4600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53cc10e3c0 0x7f53cc19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d4600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc1141a0 con 0x7f53cc074f40 2024-09-19T06:50:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d4600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc114020 con 0x7f53cc074620 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d4600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc114320 con 0x7f53cc10e3c0 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc074f40 0x7f53cc19ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d1c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53cc074620 0x7f53cc19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc074f40 0x7f53cc19ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48400/0 (socket says 172.21.15.114:48400) 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.875+0000 7f53d1200640 1 -- 172.21.15.114:0/1527985770 learned_addr learned my addr 172.21.15.114:0/1527985770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53d2600640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53cc10e3c0 0x7f53cc19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53d1c00640 1 -- 172.21.15.114:0/1527985770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53cc10e3c0 msgr2=0x7f53cc19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53d1c00640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53cc10e3c0 0x7f53cc19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53d1c00640 1 -- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc074f40 msgr2=0x7f53cc19ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53d1c00640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc074f40 0x7f53cc19ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53d1c00640 1 -- 172.21.15.114:0/1527985770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc19f5d0 con 0x7f53cc074620 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53d1200640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc074f40 0x7f53cc19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53d1c00640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53cc074620 0x7f53cc19a620 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53c000eb00 tx=0x7f53c000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:48.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53ca000640 1 -- 172.21.15.114:0/1527985770 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0010430 con 0x7f53cc074620 2024-09-19T06:50:48.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53ca000640 1 -- 172.21.15.114:0/1527985770 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c0010ab0 con 0x7f53cc074620 2024-09-19T06:50:48.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.876+0000 7f53ca000640 1 -- 172.21.15.114:0/1527985770 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0017bb0 con 0x7f53cc074620 2024-09-19T06:50:48.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.877+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53cc19f8c0 con 0x7f53cc074620 2024-09-19T06:50:48.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.877+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53cc1a71a0 con 0x7f53cc074620 2024-09-19T06:50:48.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.878+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53cc074a00 con 0x7f53cc074620 2024-09-19T06:50:48.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.880+0000 7f53ca000640 1 -- 172.21.15.114:0/1527985770 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f53c0010c60 con 0x7f53cc074620 2024-09-19T06:50:48.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.881+0000 7f53ca000640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f53b40a3350 0x7f53b40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:48.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.881+0000 7f53ca000640 1 -- 172.21.15.114:0/1527985770 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f53c00d09b0 con 0x7f53cc074620 2024-09-19T06:50:48.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.881+0000 7f53d1200640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f53b40a3350 0x7f53b40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:48.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.882+0000 7f53d1200640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f53b40a3350 0x7f53b40a5810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f53cc110210 tx=0x7f53b8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:48.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.883+0000 7f53ca000640 1 -- 172.21.15.114:0/1527985770 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f53c0054210 con 0x7f53cc074620 2024-09-19T06:50:49.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.062+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f53cc19b980 con 0x7f53cc074620 2024-09-19T06:50:49.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.063+0000 7f53ca000640 1 -- 172.21.15.114:0/1527985770 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f53c009d1b0 con 0x7f53cc074620 2024-09-19T06:50:49.064 INFO:teuthology.orchestra.run.smithi114.stdout:335007449121 2024-09-19T06:50:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f53b40a3350 msgr2=0x7f53b40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f53b40a3350 0x7f53b40a5810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f53cc110210 tx=0x7f53b8005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:49.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53cc074620 msgr2=0x7f53cc19a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53cc074620 0x7f53cc19a620 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53c000eb00 tx=0x7f53c000c2d0 comp rx=0 tx=0).stop 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 shutdown_connections 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f53b40a3350 0x7f53b40a5810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f53cc10e3c0 0x7f53cc19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53cc074f40 0x7f53cc19ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 --2- 172.21.15.114:0/1527985770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53cc074620 0x7f53cc19a620 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 >> 172.21.15.114:0/1527985770 conn(0x7f53cc0fd8c0 msgr2=0x7f53cc0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 shutdown_connections 2024-09-19T06:50:49.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:49.065+0000 7f53d4600640 1 -- 172.21.15.114:0/1527985770 wait complete. 2024-09-19T06:50:49.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920835 got 141733920837 for osd.4 2024-09-19T06:50:49.305 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:49 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3510798001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-19T06:50:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:49 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1527985770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-19T06:50:49.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:49 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3510798001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-19T06:50:49.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:49 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1527985770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-19T06:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:49 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3510798001' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-19T06:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:49 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1527985770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-19T06:50:50.212 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449119 got 335007449121 for osd.11 2024-09-19T06:50:50.212 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:50.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 -- 172.21.15.114:0/2080141364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8076720 msgr2=0x7f2dd8076b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 --2- 172.21.15.114:0/2080141364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8076720 0x7f2dd8076b00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009ca0 tx=0x7f2dcc02a8c0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 -- 172.21.15.114:0/2080141364 shutdown_connections 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 --2- 172.21.15.114:0/2080141364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd8112a30 0x7f2dd8114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 --2- 172.21.15.114:0/2080141364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2dd8077040 0x7f2dd81124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 --2- 172.21.15.114:0/2080141364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8076720 0x7f2dd8076b00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 -- 172.21.15.114:0/2080141364 >> 172.21.15.114:0/2080141364 conn(0x7f2dd80ffa90 msgr2=0x7f2dd8101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 -- 172.21.15.114:0/2080141364 shutdown_connections 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.486+0000 7f2de0600640 1 -- 172.21.15.114:0/2080141364 wait complete. 2024-09-19T06:50:50.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.487+0000 7f2de0600640 1 Processor -- start 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.487+0000 7f2de0600640 1 -- start start 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.487+0000 7f2de0600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2dd8076720 0x7f2dd81a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.487+0000 7f2de0600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8077040 0x7f2dd81a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.487+0000 7f2de0600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd8112a30 0x7f2dd819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.487+0000 7f2de0600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd81185c0 con 0x7f2dd8077040 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.487+0000 7f2de0600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8118440 con 0x7f2dd8112a30 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.487+0000 7f2de0600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8118740 con 0x7f2dd8076720 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2ddd200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8077040 0x7f2dd81a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2dde600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd8112a30 0x7f2dd819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2ddd200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8077040 0x7f2dd81a4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48342/0 (socket says 172.21.15.114:48342) 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2ddd200640 1 -- 172.21.15.114:0/3711835490 learned_addr learned my addr 172.21.15.114:0/3711835490 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2dddc00640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2dd8076720 0x7f2dd81a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:50.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2dde600640 1 -- 172.21.15.114:0/3711835490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2dd8076720 msgr2=0x7f2dd81a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2dde600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2dd8076720 0x7f2dd81a46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2dde600640 1 -- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8077040 msgr2=0x7f2dd81a4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2dde600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8077040 0x7f2dd81a4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2dde600640 1 -- 172.21.15.114:0/3711835490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd819efe0 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.488+0000 7f2dde600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd8112a30 0x7f2dd819e8b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2dd806b460 tx=0x7f2dd000db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.489+0000 7f2dd6000640 1 -- 172.21.15.114:0/3711835490 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0017070 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.489+0000 7f2dd6000640 1 -- 172.21.15.114:0/3711835490 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd0009cd0 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.489+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd819f2d0 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.489+0000 7f2dd6000640 1 -- 172.21.15.114:0/3711835490 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0003d60 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.489+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd81ab5c0 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.490+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd8076b00 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.492+0000 7f2dd6000640 1 -- 172.21.15.114:0/3711835490 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f2dd00097f0 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.494+0000 7f2dd6000640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2dc00a32d0 0x7f2dc00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.495+0000 7f2dd6000640 1 -- 172.21.15.114:0/3711835490 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2dd00cfab0 con 0x7f2dd8112a30 2024-09-19T06:50:50.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.495+0000 7f2dddc00640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2dc00a32d0 0x7f2dc00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.495+0000 7f2dddc00640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2dc00a32d0 0x7f2dc00a5790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc000f80 tx=0x7f2dcc0093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.496+0000 7f2dd6000640 1 -- 172.21.15.114:0/3711835490 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f2dd009c2b0 con 0x7f2dd8112a30 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.683+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2dd81ab860 con 0x7f2dd8112a30 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.684+0000 7f2dd6000640 1 -- 172.21.15.114:0/3711835490 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2dd009bbb0 con 0x7f2dd8112a30 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2dc00a32d0 msgr2=0x7f2dc00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2dc00a32d0 0x7f2dc00a5790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc000f80 tx=0x7f2dcc0093f0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd8112a30 msgr2=0x7f2dd819e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd8112a30 0x7f2dd819e8b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2dd806b460 tx=0x7f2dd000db40 comp rx=0 tx=0).stop 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 shutdown_connections 2024-09-19T06:50:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f2dc00a32d0 0x7f2dc00a5790 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.422 INFO:teuthology.orchestra.run.smithi114.stdout:107374182475 2024-09-19T06:50:50.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dd8112a30 0x7f2dd819e8b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2dd8077040 0x7f2dd81a4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 --2- 172.21.15.114:0/3711835490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2dd8076720 0x7f2dd81a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:50.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 >> 172.21.15.114:0/3711835490 conn(0x7f2dd80ffa90 msgr2=0x7f2dd8101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:50.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.686+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 shutdown_connections 2024-09-19T06:50:50.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:47.687+0000 7f2de0600640 1 -- 172.21.15.114:0/3711835490 wait complete. 2024-09-19T06:50:50.424 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:50 smithi114 ceph-mon[30820]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:50 smithi118 ceph-mon[35527]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:50 smithi116 ceph-mon[35411]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:50:50.739 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:51.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182471 got 107374182475 for osd.3 2024-09-19T06:50:51.079 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:51.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 -- 172.21.15.114:0/2068622073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 msgr2=0x7f770c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 --2- 172.21.15.114:0/2068622073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 0x7f770c1067e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f76f8009860 tx=0x7f76f802a340 comp rx=0 tx=0).stop 2024-09-19T06:50:51.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 -- 172.21.15.114:0/2068622073 shutdown_connections 2024-09-19T06:50:51.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 --2- 172.21.15.114:0/2068622073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c106f10 0x7f770c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 --2- 172.21.15.114:0/2068622073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 0x7f770c1067e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 --2- 172.21.15.114:0/2068622073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770c101de0 0x7f770c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 -- 172.21.15.114:0/2068622073 >> 172.21.15.114:0/2068622073 conn(0x7f770c0fd8c0 msgr2=0x7f770c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 -- 172.21.15.114:0/2068622073 shutdown_connections 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.224+0000 7f7711a00640 1 -- 172.21.15.114:0/2068622073 wait complete. 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 Processor -- start 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 -- start start 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770c101de0 0x7f770c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 0x7f770c19abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c106f10 0x7f770c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f770c111f30 con 0x7f770c101de0 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f770c111db0 con 0x7f770c106f10 2024-09-19T06:50:51.141 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1120b0 con 0x7f770c102790 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770c101de0 0x7f770c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770c101de0 0x7f770c19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48378/0 (socket says 172.21.15.114:48378) 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 -- 172.21.15.114:0/3510798001 learned_addr learned my addr 172.21.15.114:0/3510798001 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770a000640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 0x7f770c19abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770b400640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c106f10 0x7f770c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 -- 172.21.15.114:0/3510798001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 msgr2=0x7f770c19abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 0x7f770c19abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 -- 172.21.15.114:0/3510798001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c106f10 msgr2=0x7f770c19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c106f10 0x7f770c19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 -- 172.21.15.114:0/3510798001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c19f620 con 0x7f770c101de0 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770a000640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 0x7f770c19abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f770aa00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770c101de0 0x7f770c19a670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00e9c0 tx=0x7f76fc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:51.142 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7703400640 1 -- 172.21.15.114:0/3510798001 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc00cde0 con 0x7f770c101de0 2024-09-19T06:50:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7703400640 1 -- 172.21.15.114:0/3510798001 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76fc0035f0 con 0x7f770c101de0 2024-09-19T06:50:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c19f910 con 0x7f770c101de0 2024-09-19T06:50:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7703400640 1 -- 172.21.15.114:0/3510798001 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc010690 con 0x7f770c101de0 2024-09-19T06:50:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c1a71f0 con 0x7f770c101de0 2024-09-19T06:50:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.225+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f770c1021c0 con 0x7f770c101de0 2024-09-19T06:50:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.231+0000 7f7703400640 1 -- 172.21.15.114:0/3510798001 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f76fc0037a0 con 0x7f770c101de0 2024-09-19T06:50:51.143 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.232+0000 7f7703400640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76f00a3420 0x7f76f00a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:51.143 INFO:teuthology.orchestra.run.smithi114.stdout:55834574933 2024-09-19T06:50:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.232+0000 7f7703400640 1 -- 172.21.15.114:0/3510798001 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f76fc0d0150 con 0x7f770c101de0 2024-09-19T06:50:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.232+0000 7f770a000640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76f00a3420 0x7f76f00a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.232+0000 7f770a000640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76f00a3420 0x7f76f00a58e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f76f802a930 tx=0x7f76f80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.232+0000 7f7703400640 1 -- 172.21.15.114:0/3510798001 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f76fc09ca60 con 0x7f770c101de0 2024-09-19T06:50:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.431+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f770c19b9d0 con 0x7f770c101de0 2024-09-19T06:50:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.432+0000 7f7703400640 1 -- 172.21.15.114:0/3510798001 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f76fc09c360 con 0x7f770c101de0 2024-09-19T06:50:51.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76f00a3420 msgr2=0x7f76f00a58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76f00a3420 0x7f76f00a58e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f76f802a930 tx=0x7f76f80058d0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770c101de0 msgr2=0x7f770c19a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770c101de0 0x7f770c19a670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f76fc00e9c0 tx=0x7f76fc00ee80 comp rx=0 tx=0).stop 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 shutdown_connections 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f76f00a3420 0x7f76f00a58e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c106f10 0x7f770c19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c102790 0x7f770c19abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 --2- 172.21.15.114:0/3510798001 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770c101de0 0x7f770c19a670 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 >> 172.21.15.114:0/3510798001 conn(0x7f770c0fd8c0 msgr2=0x7f770c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 shutdown_connections 2024-09-19T06:50:51.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:48.434+0000 7f7711a00640 1 -- 172.21.15.114:0/3510798001 wait complete. 2024-09-19T06:50:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:51 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2364778529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-19T06:50:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-09-19T06:50:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-09-19T06:50:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-09-19T06:50:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-09-19T06:50:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 10]}]: dispatch 2024-09-19T06:50:51.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:51 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2364778529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-19T06:50:51.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-09-19T06:50:51.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-09-19T06:50:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-09-19T06:50:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-09-19T06:50:51.507 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:51 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 10]}]: dispatch 2024-09-19T06:50:51.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 -- 172.21.15.114:0/802018833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0106f20 msgr2=0x7fe7f010eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/802018833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0106f20 0x7fe7f010eaa0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009ca0 tx=0x7fe7e802a8c0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 -- 172.21.15.114:0/802018833 shutdown_connections 2024-09-19T06:50:51.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/802018833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0106f20 0x7fe7f010eaa0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/802018833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f01027a0 0x7fe7f01067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/802018833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7f0101df0 0x7fe7f01021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 -- 172.21.15.114:0/802018833 >> 172.21.15.114:0/802018833 conn(0x7fe7f00fd8b0 msgr2=0x7fe7f00ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:51.545 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 -- 172.21.15.114:0/802018833 shutdown_connections 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.420+0000 7fe7f5e00640 1 -- 172.21.15.114:0/802018833 wait complete. 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7f5e00640 1 Processor -- start 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7f5e00640 1 -- start start 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7f5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0101df0 0x7fe7f019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7f5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7f01027a0 0x7fe7f019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7f5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f0106f20 0x7fe7f019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7f5e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0111f80 con 0x7fe7f0101df0 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7f5e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0111e00 con 0x7fe7f0106f20 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7f5e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0112100 con 0x7fe7f01027a0 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.421+0000 7fe7eea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0101df0 0x7fe7f019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7eea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0101df0 0x7fe7f019a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48424/0 (socket says 172.21.15.114:48424) 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7eea00640 1 -- 172.21.15.114:0/2364778529 learned_addr learned my addr 172.21.15.114:0/2364778529 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:51.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7ee000640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7f01027a0 0x7fe7f019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7ef400640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f0106f20 0x7fe7f019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7eea00640 1 -- 172.21.15.114:0/2364778529 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7f01027a0 msgr2=0x7fe7f019aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7eea00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7f01027a0 0x7fe7f019aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7eea00640 1 -- 172.21.15.114:0/2364778529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f0106f20 msgr2=0x7fe7f019ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7eea00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f0106f20 0x7fe7f019ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7eea00640 1 -- 172.21.15.114:0/2364778529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f019f550 con 0x7fe7f0101df0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7eea00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0101df0 0x7fe7f019a5a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000e9f0 tx=0x7fe7e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7f4a00640 1 -- 172.21.15.114:0/2364778529 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e000cd80 con 0x7fe7f0101df0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7f4a00640 1 -- 172.21.15.114:0/2364778529 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e0010910 con 0x7fe7f0101df0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.422+0000 7fe7f4a00640 1 -- 172.21.15.114:0/2364778529 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0018990 con 0x7fe7f0101df0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.423+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f019f840 con 0x7fe7f0101df0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.423+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f01a7120 con 0x7fe7f0101df0 2024-09-19T06:50:51.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.424+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f01021d0 con 0x7fe7f0101df0 2024-09-19T06:50:51.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.426+0000 7fe7f4a00640 1 -- 172.21.15.114:0/2364778529 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fe7e0010ac0 con 0x7fe7f0101df0 2024-09-19T06:50:51.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.427+0000 7fe7f4a00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe7d40a32d0 0x7fe7d40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:51.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.427+0000 7fe7f4a00640 1 -- 172.21.15.114:0/2364778529 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe7e00d0170 con 0x7fe7f0101df0 2024-09-19T06:50:51.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.427+0000 7fe7ee000640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe7d40a32d0 0x7fe7d40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:51.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.428+0000 7fe7ee000640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe7d40a32d0 0x7fe7d40a5790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe7f010eec0 tx=0x7fe7dc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:51.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.431+0000 7fe7f4a00640 1 -- 172.21.15.114:0/2364778529 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fe7e009c320 con 0x7fe7f0101df0 2024-09-19T06:50:51.548 INFO:teuthology.orchestra.run.smithi114.stdout:304942678053 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.610+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fe7f01a72f0 con 0x7fe7f0101df0 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.611+0000 7fe7f4a00640 1 -- 172.21.15.114:0/2364778529 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fe7e00a0620 con 0x7fe7f0101df0 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe7d40a32d0 msgr2=0x7fe7d40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe7d40a32d0 0x7fe7d40a5790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe7f010eec0 tx=0x7fe7dc005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0101df0 msgr2=0x7fe7f019a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0101df0 0x7fe7f019a5a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000e9f0 tx=0x7fe7e000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 shutdown_connections 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fe7d40a32d0 0x7fe7d40a5790 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7f0106f20 0x7fe7f019ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7f01027a0 0x7fe7f019aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 --2- 172.21.15.114:0/2364778529 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7f0101df0 0x7fe7f019a5a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:51.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 >> 172.21.15.114:0/2364778529 conn(0x7fe7f00fd8b0 msgr2=0x7fe7f00fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:51.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 shutdown_connections 2024-09-19T06:50:51.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:50.613+0000 7fe7f5e00640 1 -- 172.21.15.114:0/2364778529 wait complete. 2024-09-19T06:50:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:51 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2364778529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-19T06:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-09-19T06:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-09-19T06:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-09-19T06:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-09-19T06:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 10]}]: dispatch 2024-09-19T06:50:52.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574930 got 55834574933 for osd.1 2024-09-19T06:50:52.233 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:52.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:52 smithi114 ceph-mon[30820]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:50:52.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-09-19T06:50:52.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-09-19T06:50:52.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-09-19T06:50:52.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-19T06:50:52.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 10]}]': finished 2024-09-19T06:50:52.502 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:52 smithi114 ceph-mon[30820]: osdmap e94: 12 total, 12 up, 12 in 2024-09-19T06:50:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:52 smithi118 ceph-mon[35527]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:50:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-09-19T06:50:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-09-19T06:50:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-09-19T06:50:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-19T06:50:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:52 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 10]}]': finished 2024-09-19T06:50:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:52 smithi118 ceph-mon[35527]: osdmap e94: 12 total, 12 up, 12 in 2024-09-19T06:50:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:52 smithi116 ceph-mon[35411]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:50:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-09-19T06:50:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-09-19T06:50:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-09-19T06:50:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-09-19T06:50:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 10]}]': finished 2024-09-19T06:50:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:52 smithi116 ceph-mon[35411]: osdmap e94: 12 total, 12 up, 12 in 2024-09-19T06:50:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:53 smithi114 ceph-mon[30820]: osdmap e95: 12 total, 12 up, 12 in 2024-09-19T06:50:53.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:53 smithi118 ceph-mon[35527]: osdmap e95: 12 total, 12 up, 12 in 2024-09-19T06:50:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:53 smithi116 ceph-mon[35411]: osdmap e95: 12 total, 12 up, 12 in 2024-09-19T06:50:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.729+0000 7f19afc00640 1 -- 172.21.15.114:0/464721456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8102790 msgr2=0x7f19a81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.729+0000 7f19afc00640 1 --2- 172.21.15.114:0/464721456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8102790 0x7f19a81067e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1990009900 tx=0x7f199002a410 comp rx=0 tx=0).stop 2024-09-19T06:50:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.729+0000 7f19afc00640 1 -- 172.21.15.114:0/464721456 shutdown_connections 2024-09-19T06:50:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.729+0000 7f19afc00640 1 --2- 172.21.15.114:0/464721456 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19a8106f10 0x7f19a810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.729+0000 7f19afc00640 1 --2- 172.21.15.114:0/464721456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8102790 0x7f19a81067e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.729+0000 7f19afc00640 1 --2- 172.21.15.114:0/464721456 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19a8101de0 0x7f19a81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.729+0000 7f19afc00640 1 -- 172.21.15.114:0/464721456 >> 172.21.15.114:0/464721456 conn(0x7f19a80fd8c0 msgr2=0x7f19a80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.729+0000 7f19afc00640 1 -- 172.21.15.114:0/464721456 shutdown_connections 2024-09-19T06:50:53.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.730+0000 7f19afc00640 1 -- 172.21.15.114:0/464721456 wait complete. 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19afc00640 1 Processor -- start 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19afc00640 1 -- start start 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19afc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8101de0 0x7f19a819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19afc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19a8102790 0x7f19a819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19afc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19a8106f10 0x7f19a819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19afc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f19a8111f80 con 0x7f19a8101de0 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19afc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19a8111e00 con 0x7f19a8102790 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19afc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f19a8112100 con 0x7f19a8106f10 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19ad200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8101de0 0x7f19a819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19ad200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8101de0 0x7f19a819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48444/0 (socket says 172.21.15.114:48444) 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.731+0000 7f19ad200640 1 -- 172.21.15.114:0/2009554506 learned_addr learned my addr 172.21.15.114:0/2009554506 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:53.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.732+0000 7f19adc00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19a8106f10 0x7f19a819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:53.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.732+0000 7f199fe00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19a8102790 0x7f19a819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:53.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.732+0000 7f19ad200640 1 -- 172.21.15.114:0/2009554506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19a8106f10 msgr2=0x7f19a819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.732+0000 7f19ad200640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19a8106f10 0x7f19a819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.732+0000 7f19ad200640 1 -- 172.21.15.114:0/2009554506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19a8102790 msgr2=0x7f19a819ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.732+0000 7f19ad200640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19a8102790 0x7f19a819ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.732+0000 7f19ad200640 1 -- 172.21.15.114:0/2009554506 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a819f690 con 0x7f19a8101de0 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.732+0000 7f19ad200640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8101de0 0x7f19a819a6e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f199800d950 tx=0x7f199800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.733+0000 7f19a6000640 1 -- 172.21.15.114:0/2009554506 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1998017070 con 0x7f19a8101de0 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.733+0000 7f19a6000640 1 -- 172.21.15.114:0/2009554506 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1998009c30 con 0x7f19a8101de0 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.733+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19a819f980 con 0x7f19a8101de0 2024-09-19T06:50:53.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.733+0000 7f19a6000640 1 -- 172.21.15.114:0/2009554506 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199800f6b0 con 0x7f19a8101de0 2024-09-19T06:50:53.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.733+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19a81a7260 con 0x7f19a8101de0 2024-09-19T06:50:53.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.734+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19a81021c0 con 0x7f19a8101de0 2024-09-19T06:50:53.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.736+0000 7f19a6000640 1 -- 172.21.15.114:0/2009554506 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f19980109d0 con 0x7f19a8101de0 2024-09-19T06:50:53.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.737+0000 7f19a6000640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f198c0a32d0 0x7f198c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:53.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.737+0000 7f19a6000640 1 -- 172.21.15.114:0/2009554506 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f19980cfa70 con 0x7f19a8101de0 2024-09-19T06:50:53.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.738+0000 7f199fe00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f198c0a32d0 0x7f198c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:53.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.739+0000 7f199fe00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f198c0a32d0 0x7f198c0a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1990000c00 tx=0x7f1990005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:53.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.742+0000 7f19a6000640 1 -- 172.21.15.114:0/2009554506 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1998009de0 con 0x7f19a8101de0 2024-09-19T06:50:53.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.919+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f19a81a7490 con 0x7f19a8101de0 2024-09-19T06:50:53.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.919+0000 7f19a6000640 1 -- 172.21.15.114:0/2009554506 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f199809bbf0 con 0x7f19a8101de0 2024-09-19T06:50:53.921 INFO:teuthology.orchestra.run.smithi114.stdout:283467841581 2024-09-19T06:50:53.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f198c0a32d0 msgr2=0x7f198c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:53.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f198c0a32d0 0x7f198c0a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1990000c00 tx=0x7f1990005b50 comp rx=0 tx=0).stop 2024-09-19T06:50:53.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8101de0 msgr2=0x7f19a819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:53.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8101de0 0x7f19a819a6e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f199800d950 tx=0x7f199800de10 comp rx=0 tx=0).stop 2024-09-19T06:50:53.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 shutdown_connections 2024-09-19T06:50:53.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f198c0a32d0 0x7f198c0a5790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19a8106f10 0x7f19a819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19a8102790 0x7f19a819ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 --2- 172.21.15.114:0/2009554506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19a8101de0 0x7f19a819a6e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:53.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 >> 172.21.15.114:0/2009554506 conn(0x7f19a80fd8c0 msgr2=0x7f19a80fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:53.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 shutdown_connections 2024-09-19T06:50:53.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:53.922+0000 7f19afc00640 1 -- 172.21.15.114:0/2009554506 wait complete. 2024-09-19T06:50:54.202 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332147 got 219043332150 for osd.7 2024-09-19T06:50:54.202 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:54 smithi114 ceph-mon[30820]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-19T06:50:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:54 smithi114 ceph-mon[30820]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-19T06:50:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:54 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2009554506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-19T06:50:54.509 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:54 smithi118 ceph-mon[35527]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-19T06:50:54.509 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:54 smithi118 ceph-mon[35527]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-19T06:50:54.509 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:54 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2009554506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-19T06:50:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:54 smithi116 ceph-mon[35411]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-09-19T06:50:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:54 smithi116 ceph-mon[35411]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-19T06:50:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:54 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2009554506' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-19T06:50:54.714 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678051 got 304942678053 for osd.10 2024-09-19T06:50:54.715 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.669+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2927038711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 msgr2=0x7f1adc076b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.669+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2927038711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 0x7f1adc076b00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009900 tx=0x7f1ad002a3f0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.670+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2927038711 shutdown_connections 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.670+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2927038711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1adc112870 0x7f1adc114c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.670+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2927038711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1adc077040 0x7f1adc112330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.670+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2927038711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 0x7f1adc076b00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.670+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2927038711 >> 172.21.15.114:0/2927038711 conn(0x7f1adc0ff8d0 msgr2=0x7f1adc101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.670+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2927038711 shutdown_connections 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.670+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2927038711 wait complete. 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2c00640 1 Processor -- start 2024-09-19T06:50:54.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2c00640 1 -- start start 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 0x7f1adc1a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1adc077040 0x7f1adc1a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1adc112870 0x7f1adc19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc118450 con 0x7f1adc076720 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1182d0 con 0x7f1adc077040 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc1185d0 con 0x7f1adc112870 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 0x7f1adc1a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 0x7f1adc1a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48470/0 (socket says 172.21.15.114:48470) 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae1800640 1 -- 172.21.15.114:0/2079644458 learned_addr learned my addr 172.21.15.114:0/2079644458 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae0e00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1adc077040 0x7f1adc1a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.671+0000 7f1ae2200640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1adc112870 0x7f1adc19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ae1800640 1 -- 172.21.15.114:0/2079644458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1adc112870 msgr2=0x7f1adc19e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:54.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ae1800640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1adc112870 0x7f1adc19e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ae1800640 1 -- 172.21.15.114:0/2079644458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1adc077040 msgr2=0x7f1adc1a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:54.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ae1800640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1adc077040 0x7f1adc1a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ae1800640 1 -- 172.21.15.114:0/2079644458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1adc19efb0 con 0x7f1adc076720 2024-09-19T06:50:54.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ae0e00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1adc077040 0x7f1adc1a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:50:54.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ae1800640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 0x7f1adc1a4670 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0004770 tx=0x7f1ad0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:54.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ada000640 1 -- 172.21.15.114:0/2079644458 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad003f070 con 0x7f1adc076720 2024-09-19T06:50:54.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ada000640 1 -- 172.21.15.114:0/2079644458 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad0031ca0 con 0x7f1adc076720 2024-09-19T06:50:54.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1adc19f240 con 0x7f1adc076720 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.672+0000 7f1ada000640 1 -- 172.21.15.114:0/2079644458 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0035030 con 0x7f1adc076720 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.673+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1adc1ab420 con 0x7f1adc076720 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.674+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1adc077e60 con 0x7f1adc076720 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.675+0000 7f1ada000640 1 -- 172.21.15.114:0/2079644458 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f1ad0042070 con 0x7f1adc076720 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.677+0000 7f1ada000640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f1ac40a3350 0x7f1ac40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.677+0000 7f1ada000640 1 -- 172.21.15.114:0/2079644458 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f1ad00f0400 con 0x7f1adc076720 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.677+0000 7f1ae0e00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f1ac40a3350 0x7f1ac40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.678+0000 7f1ae0e00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f1ac40a3350 0x7f1ac40a5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80095f0 tx=0x7f1ac8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:54.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.683+0000 7f1ada000640 1 -- 172.21.15.114:0/2079644458 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f1ad00bc5e0 con 0x7f1adc076720 2024-09-19T06:50:54.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.865+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f1adc02d7c0 con 0x7f1adc076720 2024-09-19T06:50:54.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.866+0000 7f1ada000640 1 -- 172.21.15.114:0/2079644458 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1ad00c08e0 con 0x7f1adc076720 2024-09-19T06:50:54.867 INFO:teuthology.orchestra.run.smithi114.stdout:249108103218 2024-09-19T06:50:54.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f1ac40a3350 msgr2=0x7f1ac40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:54.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f1ac40a3350 0x7f1ac40a5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80095f0 tx=0x7f1ac8005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 msgr2=0x7f1adc1a4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 0x7f1adc1a4670 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0004770 tx=0x7f1ad0003c10 comp rx=0 tx=0).stop 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 shutdown_connections 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f1ac40a3350 0x7f1ac40a5810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1adc112870 0x7f1adc19e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1adc077040 0x7f1adc1a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 --2- 172.21.15.114:0/2079644458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1adc076720 0x7f1adc1a4670 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.868+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 >> 172.21.15.114:0/2079644458 conn(0x7f1adc0ff8d0 msgr2=0x7f1adc101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.869+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 shutdown_connections 2024-09-19T06:50:54.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:54.869+0000 7f1ae2c00640 1 -- 172.21.15.114:0/2079644458 wait complete. 2024-09-19T06:50:55.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:55 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2079644458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-19T06:50:55.583 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:55 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2079644458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-19T06:50:55.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841577 got 283467841581 for osd.9 2024-09-19T06:50:55.620 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:55 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2079644458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 -- 172.21.15.114:0/1880998061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf98103e30 msgr2=0x7fbf98104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1880998061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf98103e30 0x7fbf98104210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009860 tx=0x7fbf8c02a340 comp rx=0 tx=0).stop 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 -- 172.21.15.114:0/1880998061 shutdown_connections 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1880998061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf98108f60 0x7fbf98110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1880998061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf981047e0 0x7fbf98108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1880998061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf98103e30 0x7fbf98104210 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 -- 172.21.15.114:0/1880998061 >> 172.21.15.114:0/1880998061 conn(0x7fbf980ff910 msgr2=0x7fbf98101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 -- 172.21.15.114:0/1880998061 shutdown_connections 2024-09-19T06:50:56.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.987+0000 7fbf9de00640 1 -- 172.21.15.114:0/1880998061 wait complete. 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.988+0000 7fbf9de00640 1 Processor -- start 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.988+0000 7fbf9de00640 1 -- start start 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.988+0000 7fbf9de00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf98103e30 0x7fbf9819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf9de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf981047e0 0x7fbf9819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf9de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf98108f60 0x7fbf9819ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf9de00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbf98113fc0 con 0x7fbf981047e0 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf9de00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf98113e40 con 0x7fbf98103e30 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf9de00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbf98114140 con 0x7fbf98108f60 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf97e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf981047e0 0x7fbf9819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf97e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf981047e0 0x7fbf9819a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48490/0 (socket says 172.21.15.114:48490) 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf97e00640 1 -- 172.21.15.114:0/1328174025 learned_addr learned my addr 172.21.15.114:0/1328174025 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf9ca00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf98103e30 0x7fbf9819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:56.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf9d400640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf98108f60 0x7fbf9819ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf97e00640 1 -- 172.21.15.114:0/1328174025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf98108f60 msgr2=0x7fbf9819ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf97e00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf98108f60 0x7fbf9819ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf97e00640 1 -- 172.21.15.114:0/1328174025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf98103e30 msgr2=0x7fbf9819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf97e00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf98103e30 0x7fbf9819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.989+0000 7fbf97e00640 1 -- 172.21.15.114:0/1328174025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9819f3f0 con 0x7fbf981047e0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.990+0000 7fbf97e00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf981047e0 0x7fbf9819a980 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400e9c0 tx=0x7fbf8400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.990+0000 7fbf95600640 1 -- 172.21.15.114:0/1328174025 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8400cde0 con 0x7fbf981047e0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.990+0000 7fbf95600640 1 -- 172.21.15.114:0/1328174025 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf840035f0 con 0x7fbf981047e0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.990+0000 7fbf95600640 1 -- 172.21.15.114:0/1328174025 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84010690 con 0x7fbf981047e0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.990+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9819f6e0 con 0x7fbf981047e0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.990+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf981a6fc0 con 0x7fbf981047e0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.991+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf98105310 con 0x7fbf981047e0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.993+0000 7fbf95600640 1 -- 172.21.15.114:0/1328174025 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fbf840037a0 con 0x7fbf981047e0 2024-09-19T06:50:56.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.995+0000 7fbf95600640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbf800a3350 0x7fbf800a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:56.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.995+0000 7fbf95600640 1 -- 172.21.15.114:0/1328174025 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fbf840d0170 con 0x7fbf981047e0 2024-09-19T06:50:56.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.995+0000 7fbf9ca00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbf800a3350 0x7fbf800a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:56.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.996+0000 7fbf9ca00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbf800a3350 0x7fbf800a5810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c02a930 tx=0x7fbf8c0056d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:56.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:55.999+0000 7fbf95600640 1 -- 172.21.15.114:0/1328174025 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbf8409c8e0 con 0x7fbf981047e0 2024-09-19T06:50:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.178+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fbf9810bbb0 con 0x7fbf981047e0 2024-09-19T06:50:56.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.179+0000 7fbf95600640 1 -- 172.21.15.114:0/1328174025 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbf840bae20 con 0x7fbf981047e0 2024-09-19T06:50:56.180 INFO:teuthology.orchestra.run.smithi114.stdout:34359738462 2024-09-19T06:50:56.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbf800a3350 msgr2=0x7fbf800a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbf800a3350 0x7fbf800a5810 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c02a930 tx=0x7fbf8c0056d0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf981047e0 msgr2=0x7fbf9819a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf981047e0 0x7fbf9819a980 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400e9c0 tx=0x7fbf8400ee80 comp rx=0 tx=0).stop 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 shutdown_connections 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbf800a3350 0x7fbf800a5810 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf98108f60 0x7fbf9819ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf981047e0 0x7fbf9819a980 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 --2- 172.21.15.114:0/1328174025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf98103e30 0x7fbf9819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 >> 172.21.15.114:0/1328174025 conn(0x7fbf980ff910 msgr2=0x7fbf98100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 shutdown_connections 2024-09-19T06:50:56.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:56.181+0000 7fbf9de00640 1 -- 172.21.15.114:0/1328174025 wait complete. 2024-09-19T06:50:56.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:56 smithi114 ceph-mon[30820]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-09-19T06:50:56.447 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:56 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1328174025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-19T06:50:56.483 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103214 got 249108103218 for osd.8 2024-09-19T06:50:56.483 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:56 smithi118 ceph-mon[35527]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-09-19T06:50:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:56 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1328174025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-19T06:50:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:56 smithi116 ceph-mon[35411]: pgmap v357: 129 pgs: 2 peering, 127 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-09-19T06:50:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:56 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1328174025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-19T06:50:56.746 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:57.256 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738458 got 34359738462 for osd.0 2024-09-19T06:50:57.256 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:57.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:57 smithi114 ceph-mon[30820]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-19T06:50:57.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:57 smithi114 ceph-mon[30820]: Cluster is now healthy 2024-09-19T06:50:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:57 smithi116 ceph-mon[35411]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-19T06:50:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:57 smithi116 ceph-mon[35411]: Cluster is now healthy 2024-09-19T06:50:57.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 -- 172.21.15.114:0/319542132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 msgr2=0x7f50a0104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:57.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 --2- 172.21.15.114:0/319542132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 0x7f50a0104210 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5090009860 tx=0x7f509002a340 comp rx=0 tx=0).stop 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 -- 172.21.15.114:0/319542132 shutdown_connections 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 --2- 172.21.15.114:0/319542132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50a0108f60 0x7f50a0110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 --2- 172.21.15.114:0/319542132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a01047e0 0x7f50a0108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 --2- 172.21.15.114:0/319542132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 0x7f50a0104210 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 -- 172.21.15.114:0/319542132 >> 172.21.15.114:0/319542132 conn(0x7f50a00ff910 msgr2=0x7f50a0101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 -- 172.21.15.114:0/319542132 shutdown_connections 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.712+0000 7f50a4e00640 1 -- 172.21.15.114:0/319542132 wait complete. 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.713+0000 7f50a4e00640 1 Processor -- start 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.713+0000 7f50a4e00640 1 -- start start 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.713+0000 7f50a4e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 0x7f50a019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.713+0000 7f50a4e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a01047e0 0x7f50a019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:57.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f50a4e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50a0108f60 0x7f50a019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f50a4e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f50a0113fc0 con 0x7f50a01047e0 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f50a4e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50a0113e40 con 0x7f50a0103e30 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f50a4e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f50a0114140 con 0x7f50a0108f60 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f5096a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a01047e0 0x7f50a019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f509f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 0x7f50a019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f5096a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a01047e0 0x7f50a019a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50250/0 (socket says 172.21.15.114:50250) 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f5096a00640 1 -- 172.21.15.114:0/1727220973 learned_addr learned my addr 172.21.15.114:0/1727220973 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f509fe00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50a0108f60 0x7f50a019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f509f400640 1 -- 172.21.15.114:0/1727220973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50a0108f60 msgr2=0x7f50a019ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f509f400640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50a0108f60 0x7f50a019ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f509f400640 1 -- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a01047e0 msgr2=0x7f50a019a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f509f400640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a01047e0 0x7f50a019a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:57.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f509f400640 1 -- 172.21.15.114:0/1727220973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a019f3f0 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.714+0000 7f509fe00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50a0108f60 0x7f50a019ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.715+0000 7f509f400640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 0x7f50a019a440 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f509002a860 tx=0x7f5090033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.715+0000 7f509cc00640 1 -- 172.21.15.114:0/1727220973 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090037040 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.715+0000 7f509cc00640 1 -- 172.21.15.114:0/1727220973 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5090033d50 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.715+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a019f680 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.715+0000 7f509cc00640 1 -- 172.21.15.114:0/1727220973 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090003960 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.715+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a01a6fc0 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.716+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a0105310 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.718+0000 7f509cc00640 1 -- 172.21.15.114:0/1727220973 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f5090042020 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.719+0000 7f509cc00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f50840a3350 0x7f50840a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.719+0000 7f509cc00640 1 -- 172.21.15.114:0/1727220973 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f50900ef8c0 con 0x7f50a0103e30 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.719+0000 7f5096a00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f50840a3350 0x7f50840a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.720+0000 7f5096a00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f50840a3350 0x7f50840a5810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f50a00ffd30 tx=0x7f5088009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:57.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.721+0000 7f509cc00640 1 -- 172.21.15.114:0/1727220973 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f5090049020 con 0x7f50a0103e30 2024-09-19T06:50:57.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:57 smithi118 ceph-mon[35527]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-19T06:50:57.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:57 smithi118 ceph-mon[35527]: Cluster is now healthy 2024-09-19T06:50:57.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.897+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f50a010c140 con 0x7f50a0103e30 2024-09-19T06:50:57.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.898+0000 7f509cc00640 1 -- 172.21.15.114:0/1727220973 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f50900bc140 con 0x7f50a0103e30 2024-09-19T06:50:57.899 INFO:teuthology.orchestra.run.smithi114.stdout:163208757316 2024-09-19T06:50:57.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.900+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f50840a3350 msgr2=0x7f50840a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:57.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.900+0000 7f50a4e00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f50840a3350 0x7f50840a5810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f50a00ffd30 tx=0x7f5088009260 comp rx=0 tx=0).stop 2024-09-19T06:50:57.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.900+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 msgr2=0x7f50a019a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:57.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.900+0000 7f50a4e00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 0x7f50a019a440 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f509002a860 tx=0x7f5090033b60 comp rx=0 tx=0).stop 2024-09-19T06:50:57.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.900+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 shutdown_connections 2024-09-19T06:50:57.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.900+0000 7f50a4e00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f50840a3350 0x7f50840a5810 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f50a00ffd30 tx=0x7f5088009260 comp rx=0 tx=0).stop 2024-09-19T06:50:57.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.900+0000 7f50a4e00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50a0108f60 0x7f50a019ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:57.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.901+0000 7f50a4e00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f50a01047e0 0x7f50a019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.901+0000 7f50a4e00640 1 --2- 172.21.15.114:0/1727220973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a0103e30 0x7f50a019a440 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.901+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 >> 172.21.15.114:0/1727220973 conn(0x7f50a00ff910 msgr2=0x7f50a0100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.901+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 shutdown_connections 2024-09-19T06:50:57.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:57.901+0000 7f50a4e00640 1 -- 172.21.15.114:0/1727220973 wait complete. 2024-09-19T06:50:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:58 smithi114 ceph-mon[30820]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-09-19T06:50:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:50:58 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1727220973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-19T06:50:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:58 smithi116 ceph-mon[35411]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-09-19T06:50:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:50:58 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1727220973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-19T06:50:58.707 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757313 got 163208757316 for osd.5 2024-09-19T06:50:58.707 DEBUG:teuthology.parallel:result is None 2024-09-19T06:50:58.707 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-19T06:50:58.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T06:50:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:58 smithi118 ceph-mon[35527]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-09-19T06:50:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:50:58 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1727220973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-19T06:50:59.077 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:50:59.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 -- 172.21.15.114:0/155833281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0109140 msgr2=0x7face0110cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:59.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 --2- 172.21.15.114:0/155833281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0109140 0x7face0110cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7facd8009ca0 tx=0x7facd802a8c0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 -- 172.21.15.114:0/155833281 shutdown_connections 2024-09-19T06:50:59.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 --2- 172.21.15.114:0/155833281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0109140 0x7face0110cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 --2- 172.21.15.114:0/155833281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7face01049c0 0x7face0108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 --2- 172.21.15.114:0/155833281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face0104010 0x7face01043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 -- 172.21.15.114:0/155833281 >> 172.21.15.114:0/155833281 conn(0x7face00ffa50 msgr2=0x7face0101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:59.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 -- 172.21.15.114:0/155833281 shutdown_connections 2024-09-19T06:50:59.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.622+0000 7face4c00640 1 -- 172.21.15.114:0/155833281 wait complete. 2024-09-19T06:50:59.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.623+0000 7face4c00640 1 Processor -- start 2024-09-19T06:50:59.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.623+0000 7face4c00640 1 -- start start 2024-09-19T06:50:59.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.623+0000 7face4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0104010 0x7face019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:59.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.623+0000 7face4c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7face01049c0 0x7face019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:59.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.623+0000 7face4c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face0109140 0x7face019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:59.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.623+0000 7face4c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7face01141c0 con 0x7face0104010 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.623+0000 7face4c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7face0114040 con 0x7face01049c0 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.623+0000 7face4c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7face0114340 con 0x7face0109140 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0104010 0x7face019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0104010 0x7face019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50272/0 (socket says 172.21.15.114:50272) 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 -- 172.21.15.114:0/3646999008 learned_addr learned my addr 172.21.15.114:0/3646999008 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facde800640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face0109140 0x7face019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdd400640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7face01049c0 0x7face019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 -- 172.21.15.114:0/3646999008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face0109140 msgr2=0x7face019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face0109140 0x7face019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 -- 172.21.15.114:0/3646999008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7face01049c0 msgr2=0x7face019ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7face01049c0 0x7face019ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 -- 172.21.15.114:0/3646999008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face019f670 con 0x7face0104010 2024-09-19T06:50:59.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.624+0000 7facdde00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0104010 0x7face019a6c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7facd000e9f0 tx=0x7facd000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:59.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.625+0000 7facd6a00640 1 -- 172.21.15.114:0/3646999008 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd000cd80 con 0x7face0104010 2024-09-19T06:50:59.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.625+0000 7facd6a00640 1 -- 172.21.15.114:0/3646999008 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facd0005500 con 0x7face0104010 2024-09-19T06:50:59.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.625+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7face019f960 con 0x7face0104010 2024-09-19T06:50:59.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.625+0000 7facd6a00640 1 -- 172.21.15.114:0/3646999008 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd0010690 con 0x7face0104010 2024-09-19T06:50:59.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.625+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7face01a7240 con 0x7face0104010 2024-09-19T06:50:59.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.626+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7face01043f0 con 0x7face0104010 2024-09-19T06:50:59.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.628+0000 7facd6a00640 1 -- 172.21.15.114:0/3646999008 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7facd00046c0 con 0x7face0104010 2024-09-19T06:50:59.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.629+0000 7facd6a00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7facc40a3350 0x7facc40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:50:59.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.629+0000 7facd6a00640 1 -- 172.21.15.114:0/3646999008 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7facd00d0270 con 0x7face0104010 2024-09-19T06:50:59.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.629+0000 7facdd400640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7facc40a3350 0x7facc40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:50:59.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.630+0000 7facdd400640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7facc40a3350 0x7facc40a5810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7face010aab0 tx=0x7faccc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:50:59.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.633+0000 7facd6a00640 1 -- 172.21.15.114:0/3646999008 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7facd009c9e0 con 0x7face0104010 2024-09-19T06:50:59.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.803+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7face0062630 con 0x7facc40a3350 2024-09-19T06:50:59.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.811+0000 7facd6a00640 1 -- 172.21.15.114:0/3646999008 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340646 (secure 0 0 0) 0x7face0062630 con 0x7facc40a3350 2024-09-19T06:50:59.817 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:50:59.818 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-19T06:50:59.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7facc40a3350 msgr2=0x7facc40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7facc40a3350 0x7facc40a5810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7face010aab0 tx=0x7faccc005f90 comp rx=0 tx=0).stop 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0104010 msgr2=0x7face019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0104010 0x7face019a6c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7facd000e9f0 tx=0x7facd000edf0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 shutdown_connections 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7facc40a3350 0x7facc40a5810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7face0109140 0x7face019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7face01049c0 0x7face019ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 --2- 172.21.15.114:0/3646999008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7face0104010 0x7face019a6c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 >> 172.21.15.114:0/3646999008 conn(0x7face00ffa50 msgr2=0x7face0100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.816+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 shutdown_connections 2024-09-19T06:50:59.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:50:59.817+0000 7face4c00640 1 -- 172.21.15.114:0/3646999008 wait complete. 2024-09-19T06:51:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:00 smithi114 ceph-mon[30820]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s; 11 B/s, 0 objects/s recovering 2024-09-19T06:51:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:00 smithi116 ceph-mon[35411]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s; 11 B/s, 0 objects/s recovering 2024-09-19T06:51:00.751 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-19T06:50:59.112081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660078,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":729,"num_read_kb":484,"num_write":431,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180826,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":352,"ondisk_log_size":352,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":345564,"kb_used_data":12884,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124596260,"statfs":{"total":1151940427776,"available":1151586570240,"internally_reserved":0,"allocated":13193216,"data_stored":8443772,"data_compressed":461784,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19139,"internal_metadata":340374845},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":23,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002089"},"pg_stats":[{"pgid":"2.1f","version":"93'13","reported_seq":111,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:54.333108+0000","last_change":"2024-09-19T06:49:07.909205+0000","last_active":"2024-09-19T06:50:54.333108+0000","last_peered":"2024-09-19T06:50:54.333108+0000","last_clean":"2024-09-19T06:50:54.333108+0000","last_became_active":"2024-09-19T06:49:07.907471+0000","last_became_peered":"2024-09-19T06:49:07.907471+0000","last_unstale":"2024-09-19T06:50:54.333108+0000","last_undegraded":"2024-09-19T06:50:54.333108+0000","last_fullsized":"2024-09-19T06:50:54.333108+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:55:00.655927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":58,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330051+0000","last_change":"2024-09-19T06:49:52.196586+0000","last_active":"2024-09-19T06:50:52.330051+0000","last_peered":"2024-09-19T06:50:52.330051+0000","last_clean":"2024-09-19T06:50:52.330051+0000","last_became_active":"2024-09-19T06:49:52.195680+0000","last_became_peered":"2024-09-19T06:49:52.195680+0000","last_unstale":"2024-09-19T06:50:52.330051+0000","last_undegraded":"2024-09-19T06:50:52.330051+0000","last_fullsized":"2024-09-19T06:50:52.330051+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:46:53.026767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330518+0000","last_change":"2024-09-19T06:49:20.424335+0000","last_active":"2024-09-19T06:50:52.330518+0000","last_peered":"2024-09-19T06:50:52.330518+0000","last_clean":"2024-09-19T06:50:52.330518+0000","last_became_active":"2024-09-19T06:49:20.423890+0000","last_became_peered":"2024-09-19T06:49:20.423890+0000","last_unstale":"2024-09-19T06:50:52.330518+0000","last_undegraded":"2024-09-19T06:50:52.330518+0000","last_fullsized":"2024-09-19T06:50:52.330518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:30:49.042220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218579+0000","last_change":"2024-09-19T06:49:08.185286+0000","last_active":"2024-09-19T06:50:52.218579+0000","last_peered":"2024-09-19T06:50:52.218579+0000","last_clean":"2024-09-19T06:50:52.218579+0000","last_became_active":"2024-09-19T06:49:08.184971+0000","last_became_peered":"2024-09-19T06:49:08.184971+0000","last_unstale":"2024-09-19T06:50:52.218579+0000","last_undegraded":"2024-09-19T06:50:52.218579+0000","last_fullsized":"2024-09-19T06:50:52.218579+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:26.532594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767343+0000","last_change":"2024-09-19T06:49:10.385824+0000","last_active":"2024-09-19T06:50:52.767343+0000","last_peered":"2024-09-19T06:50:52.767343+0000","last_clean":"2024-09-19T06:50:52.767343+0000","last_became_active":"2024-09-19T06:49:10.381448+0000","last_became_peered":"2024-09-19T06:49:10.381448+0000","last_unstale":"2024-09-19T06:50:52.767343+0000","last_undegraded":"2024-09-19T06:50:52.767343+0000","last_fullsized":"2024-09-19T06:50:52.767343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:39.192054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.334354+0000","last_change":"2024-09-19T06:50:52.334354+0000","last_active":"2024-09-19T06:50:52.334354+0000","last_peered":"2024-09-19T06:50:52.334354+0000","last_clean":"2024-09-19T06:50:52.334354+0000","last_became_active":"2024-09-19T06:50:52.333866+0000","last_became_peered":"2024-09-19T06:50:52.333866+0000","last_unstale":"2024-09-19T06:50:52.334354+0000","last_undegraded":"2024-09-19T06:50:52.334354+0000","last_fullsized":"2024-09-19T06:50:52.334354+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:08.379318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,10,3],"acting":[4,6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'12","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216299+0000","last_change":"2024-09-19T06:49:07.907830+0000","last_active":"2024-09-19T06:50:52.216299+0000","last_peered":"2024-09-19T06:50:52.216299+0000","last_clean":"2024-09-19T06:50:52.216299+0000","last_became_active":"2024-09-19T06:49:07.907510+0000","last_became_peered":"2024-09-19T06:49:07.907510+0000","last_unstale":"2024-09-19T06:50:52.216299+0000","last_undegraded":"2024-09-19T06:50:52.216299+0000","last_fullsized":"2024-09-19T06:50:52.216299+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:52:46.192671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414918+0000","last_change":"2024-09-19T06:49:10.371663+0000","last_active":"2024-09-19T06:50:52.414918+0000","last_peered":"2024-09-19T06:50:52.414918+0000","last_clean":"2024-09-19T06:50:52.414918+0000","last_became_active":"2024-09-19T06:49:10.371325+0000","last_became_peered":"2024-09-19T06:49:10.371325+0000","last_unstale":"2024-09-19T06:50:52.414918+0000","last_undegraded":"2024-09-19T06:50:52.414918+0000","last_fullsized":"2024-09-19T06:50:52.414918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:48:58.406933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217200+0000","last_change":"2024-09-19T06:49:20.429446+0000","last_active":"2024-09-19T06:50:52.217200+0000","last_peered":"2024-09-19T06:50:52.217200+0000","last_clean":"2024-09-19T06:50:52.217200+0000","last_became_active":"2024-09-19T06:49:20.428996+0000","last_became_peered":"2024-09-19T06:49:20.428996+0000","last_unstale":"2024-09-19T06:50:52.217200+0000","last_undegraded":"2024-09-19T06:50:52.217200+0000","last_fullsized":"2024-09-19T06:50:52.217200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:39.542347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"93'5","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216067+0000","last_change":"2024-09-19T06:49:07.906940+0000","last_active":"2024-09-19T06:50:52.216067+0000","last_peered":"2024-09-19T06:50:52.216067+0000","last_clean":"2024-09-19T06:50:52.216067+0000","last_became_active":"2024-09-19T06:49:07.906635+0000","last_became_peered":"2024-09-19T06:49:07.906635+0000","last_unstale":"2024-09-19T06:50:52.216067+0000","last_undegraded":"2024-09-19T06:50:52.216067+0000","last_fullsized":"2024-09-19T06:50:52.216067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:00.794049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223602+0000","last_change":"2024-09-19T06:49:52.411553+0000","last_active":"2024-09-19T06:50:52.223602+0000","last_peered":"2024-09-19T06:50:52.223602+0000","last_clean":"2024-09-19T06:50:52.223602+0000","last_became_active":"2024-09-19T06:49:52.411251+0000","last_became_peered":"2024-09-19T06:49:52.411251+0000","last_unstale":"2024-09-19T06:50:52.223602+0000","last_undegraded":"2024-09-19T06:50:52.223602+0000","last_fullsized":"2024-09-19T06:50:52.223602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:17.774698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216749+0000","last_change":"2024-09-19T06:49:20.427188+0000","last_active":"2024-09-19T06:50:52.216749+0000","last_peered":"2024-09-19T06:50:52.216749+0000","last_clean":"2024-09-19T06:50:52.216749+0000","last_became_active":"2024-09-19T06:49:20.426716+0000","last_became_peered":"2024-09-19T06:49:20.426716+0000","last_unstale":"2024-09-19T06:50:52.216749+0000","last_undegraded":"2024-09-19T06:50:52.216749+0000","last_fullsized":"2024-09-19T06:50:52.216749+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:25:09.086438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222719+0000","last_change":"2024-09-19T06:49:10.371433+0000","last_active":"2024-09-19T06:50:52.222719+0000","last_peered":"2024-09-19T06:50:52.222719+0000","last_clean":"2024-09-19T06:50:52.222719+0000","last_became_active":"2024-09-19T06:49:10.370947+0000","last_became_peered":"2024-09-19T06:49:10.370947+0000","last_unstale":"2024-09-19T06:50:52.222719+0000","last_undegraded":"2024-09-19T06:50:52.222719+0000","last_fullsized":"2024-09-19T06:50:52.222719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:40.954670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215809+0000","last_change":"2024-09-19T06:49:07.907972+0000","last_active":"2024-09-19T06:50:52.215809+0000","last_peered":"2024-09-19T06:50:52.215809+0000","last_clean":"2024-09-19T06:50:52.215809+0000","last_became_active":"2024-09-19T06:49:07.907631+0000","last_became_peered":"2024-09-19T06:49:07.907631+0000","last_unstale":"2024-09-19T06:50:52.215809+0000","last_undegraded":"2024-09-19T06:50:52.215809+0000","last_fullsized":"2024-09-19T06:50:52.215809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:12:16.298696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217079+0000","last_change":"2024-09-19T06:49:20.617446+0000","last_active":"2024-09-19T06:50:52.217079+0000","last_peered":"2024-09-19T06:50:52.217079+0000","last_clean":"2024-09-19T06:50:52.217079+0000","last_became_active":"2024-09-19T06:49:20.616950+0000","last_became_peered":"2024-09-19T06:49:20.616950+0000","last_unstale":"2024-09-19T06:50:52.217079+0000","last_undegraded":"2024-09-19T06:50:52.217079+0000","last_fullsized":"2024-09-19T06:50:52.217079+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:37:00.640364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.329931+0000","last_change":"2024-09-19T06:49:52.195937+0000","last_active":"2024-09-19T06:50:52.329931+0000","last_peered":"2024-09-19T06:50:52.329931+0000","last_clean":"2024-09-19T06:50:52.329931+0000","last_became_active":"2024-09-19T06:49:52.195615+0000","last_became_peered":"2024-09-19T06:49:52.195615+0000","last_unstale":"2024-09-19T06:50:52.329931+0000","last_undegraded":"2024-09-19T06:50:52.329931+0000","last_fullsized":"2024-09-19T06:50:52.329931+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:28:33.296849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.a","version":"93'3","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330730+0000","last_change":"2024-09-19T06:49:07.906081+0000","last_active":"2024-09-19T06:50:52.330730+0000","last_peered":"2024-09-19T06:50:52.330730+0000","last_clean":"2024-09-19T06:50:52.330730+0000","last_became_active":"2024-09-19T06:49:07.905559+0000","last_became_peered":"2024-09-19T06:49:07.905559+0000","last_unstale":"2024-09-19T06:50:52.330730+0000","last_undegraded":"2024-09-19T06:50:52.330730+0000","last_fullsized":"2024-09-19T06:50:52.330730+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:30:56.922084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217028+0000","last_change":"2024-09-19T06:49:20.431255+0000","last_active":"2024-09-19T06:50:52.217028+0000","last_peered":"2024-09-19T06:50:52.217028+0000","last_clean":"2024-09-19T06:50:52.217028+0000","last_became_active":"2024-09-19T06:49:20.430727+0000","last_became_peered":"2024-09-19T06:49:20.430727+0000","last_unstale":"2024-09-19T06:50:52.217028+0000","last_undegraded":"2024-09-19T06:50:52.217028+0000","last_fullsized":"2024-09-19T06:50:52.217028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:01.761121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220297+0000","last_change":"2024-09-19T06:49:10.382739+0000","last_active":"2024-09-19T06:50:52.220297+0000","last_peered":"2024-09-19T06:50:52.220297+0000","last_clean":"2024-09-19T06:50:52.220297+0000","last_became_active":"2024-09-19T06:49:10.382216+0000","last_became_peered":"2024-09-19T06:49:10.382216+0000","last_unstale":"2024-09-19T06:50:52.220297+0000","last_undegraded":"2024-09-19T06:50:52.220297+0000","last_fullsized":"2024-09-19T06:50:52.220297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:31:37.908345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"93'7","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218241+0000","last_change":"2024-09-19T06:49:07.906876+0000","last_active":"2024-09-19T06:50:52.218241+0000","last_peered":"2024-09-19T06:50:52.218241+0000","last_clean":"2024-09-19T06:50:52.218241+0000","last_became_active":"2024-09-19T06:49:07.906567+0000","last_became_peered":"2024-09-19T06:49:07.906567+0000","last_unstale":"2024-09-19T06:50:52.218241+0000","last_undegraded":"2024-09-19T06:50:52.218241+0000","last_fullsized":"2024-09-19T06:50:52.218241+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:21:39.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332912+0000","last_change":"2024-09-19T06:50:52.332343+0000","last_active":"2024-09-19T06:50:52.332912+0000","last_peered":"2024-09-19T06:50:52.332912+0000","last_clean":"2024-09-19T06:50:52.332912+0000","last_became_active":"2024-09-19T06:50:52.331869+0000","last_became_peered":"2024-09-19T06:50:52.331869+0000","last_unstale":"2024-09-19T06:50:52.332912+0000","last_undegraded":"2024-09-19T06:50:52.332912+0000","last_fullsized":"2024-09-19T06:50:52.332912+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:56:59.286185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215623+0000","last_change":"2024-09-19T06:49:10.369037+0000","last_active":"2024-09-19T06:50:52.215623+0000","last_peered":"2024-09-19T06:50:52.215623+0000","last_clean":"2024-09-19T06:50:52.215623+0000","last_became_active":"2024-09-19T06:49:10.368653+0000","last_became_peered":"2024-09-19T06:49:10.368653+0000","last_unstale":"2024-09-19T06:50:52.215623+0000","last_undegraded":"2024-09-19T06:50:52.215623+0000","last_fullsized":"2024-09-19T06:50:52.215623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:46:02.268843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414742+0000","last_change":"2024-09-19T06:49:08.185517+0000","last_active":"2024-09-19T06:50:52.414742+0000","last_peered":"2024-09-19T06:50:52.414742+0000","last_clean":"2024-09-19T06:50:52.414742+0000","last_became_active":"2024-09-19T06:49:08.185138+0000","last_became_peered":"2024-09-19T06:49:08.185138+0000","last_unstale":"2024-09-19T06:50:52.414742+0000","last_undegraded":"2024-09-19T06:50:52.414742+0000","last_fullsized":"2024-09-19T06:50:52.414742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:57:59.082821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217098+0000","last_change":"2024-09-19T06:49:20.617423+0000","last_active":"2024-09-19T06:50:52.217098+0000","last_peered":"2024-09-19T06:50:52.217098+0000","last_clean":"2024-09-19T06:50:52.217098+0000","last_became_active":"2024-09-19T06:49:20.616834+0000","last_became_peered":"2024-09-19T06:49:20.616834+0000","last_unstale":"2024-09-19T06:50:52.217098+0000","last_undegraded":"2024-09-19T06:50:52.217098+0000","last_fullsized":"2024-09-19T06:50:52.217098+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:53:47.384169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767310+0000","last_change":"2024-09-19T06:49:10.388955+0000","last_active":"2024-09-19T06:50:52.767310+0000","last_peered":"2024-09-19T06:50:52.767310+0000","last_clean":"2024-09-19T06:50:52.767310+0000","last_became_active":"2024-09-19T06:49:10.387947+0000","last_became_peered":"2024-09-19T06:49:10.387947+0000","last_unstale":"2024-09-19T06:50:52.767310+0000","last_undegraded":"2024-09-19T06:50:52.767310+0000","last_fullsized":"2024-09-19T06:50:52.767310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:44.556097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"93'7","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217734+0000","last_change":"2024-09-19T06:49:07.912897+0000","last_active":"2024-09-19T06:50:52.217734+0000","last_peered":"2024-09-19T06:50:52.217734+0000","last_clean":"2024-09-19T06:50:52.217734+0000","last_became_active":"2024-09-19T06:49:07.912520+0000","last_became_peered":"2024-09-19T06:49:07.912520+0000","last_unstale":"2024-09-19T06:50:52.217734+0000","last_undegraded":"2024-09-19T06:50:52.217734+0000","last_fullsized":"2024-09-19T06:50:52.217734+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:23:56.582808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330848+0000","last_change":"2024-09-19T06:49:20.420851+0000","last_active":"2024-09-19T06:50:52.330848+0000","last_peered":"2024-09-19T06:50:52.330848+0000","last_clean":"2024-09-19T06:50:52.330848+0000","last_became_active":"2024-09-19T06:49:20.420307+0000","last_became_peered":"2024-09-19T06:49:20.420307+0000","last_unstale":"2024-09-19T06:50:52.330848+0000","last_undegraded":"2024-09-19T06:50:52.330848+0000","last_fullsized":"2024-09-19T06:50:52.330848+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:20.633070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218467+0000","last_change":"2024-09-19T06:49:10.372935+0000","last_active":"2024-09-19T06:50:52.218467+0000","last_peered":"2024-09-19T06:50:52.218467+0000","last_clean":"2024-09-19T06:50:52.218467+0000","last_became_active":"2024-09-19T06:49:10.372641+0000","last_became_peered":"2024-09-19T06:49:10.372641+0000","last_unstale":"2024-09-19T06:50:52.218467+0000","last_undegraded":"2024-09-19T06:50:52.218467+0000","last_fullsized":"2024-09-19T06:50:52.218467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:13:07.917094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'16","reported_seq":56,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330389+0000","last_change":"2024-09-19T06:49:07.908184+0000","last_active":"2024-09-19T06:50:52.330389+0000","last_peered":"2024-09-19T06:50:52.330389+0000","last_clean":"2024-09-19T06:50:52.330389+0000","last_became_active":"2024-09-19T06:49:07.907722+0000","last_became_peered":"2024-09-19T06:49:07.907722+0000","last_unstale":"2024-09-19T06:50:52.330389+0000","last_undegraded":"2024-09-19T06:50:52.330389+0000","last_fullsized":"2024-09-19T06:50:52.330389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:24:25.957501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218316+0000","last_change":"2024-09-19T06:49:20.616966+0000","last_active":"2024-09-19T06:50:52.218316+0000","last_peered":"2024-09-19T06:50:52.218316+0000","last_clean":"2024-09-19T06:50:52.218316+0000","last_became_active":"2024-09-19T06:49:20.616402+0000","last_became_peered":"2024-09-19T06:49:20.616402+0000","last_unstale":"2024-09-19T06:50:52.218316+0000","last_undegraded":"2024-09-19T06:50:52.218316+0000","last_fullsized":"2024-09-19T06:50:52.218316+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:15:12.572214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.333101+0000","last_change":"2024-09-19T06:49:10.373048+0000","last_active":"2024-09-19T06:50:52.333101+0000","last_peered":"2024-09-19T06:50:52.333101+0000","last_clean":"2024-09-19T06:50:52.333101+0000","last_became_active":"2024-09-19T06:49:10.372257+0000","last_became_peered":"2024-09-19T06:49:10.372257+0000","last_unstale":"2024-09-19T06:50:52.333101+0000","last_undegraded":"2024-09-19T06:50:52.333101+0000","last_fullsized":"2024-09-19T06:50:52.333101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:30.707948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"93'4","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220484+0000","last_change":"2024-09-19T06:49:52.195314+0000","last_active":"2024-09-19T06:50:52.220484+0000","last_peered":"2024-09-19T06:50:52.220484+0000","last_clean":"2024-09-19T06:50:52.220484+0000","last_became_active":"2024-09-19T06:49:52.194849+0000","last_became_peered":"2024-09-19T06:49:52.194849+0000","last_unstale":"2024-09-19T06:50:52.220484+0000","last_undegraded":"2024-09-19T06:50:52.220484+0000","last_fullsized":"2024-09-19T06:50:52.220484+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:01:20.902382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220312+0000","last_change":"2024-09-19T06:49:20.422658+0000","last_active":"2024-09-19T06:50:52.220312+0000","last_peered":"2024-09-19T06:50:52.220312+0000","last_clean":"2024-09-19T06:50:52.220312+0000","last_became_active":"2024-09-19T06:49:20.422190+0000","last_became_peered":"2024-09-19T06:49:20.422190+0000","last_unstale":"2024-09-19T06:50:52.220312+0000","last_undegraded":"2024-09-19T06:50:52.220312+0000","last_fullsized":"2024-09-19T06:50:52.220312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:13:14.338388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267449+0000","last_change":"2024-09-19T06:49:10.376579+0000","last_active":"2024-09-19T06:50:52.267449+0000","last_peered":"2024-09-19T06:50:52.267449+0000","last_clean":"2024-09-19T06:50:52.267449+0000","last_became_active":"2024-09-19T06:49:10.375920+0000","last_became_peered":"2024-09-19T06:49:10.375920+0000","last_unstale":"2024-09-19T06:50:52.267449+0000","last_undegraded":"2024-09-19T06:50:52.267449+0000","last_fullsized":"2024-09-19T06:50:52.267449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:47:35.976117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"93'1","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330330+0000","last_change":"2024-09-19T06:49:08.187353+0000","last_active":"2024-09-19T06:50:52.330330+0000","last_peered":"2024-09-19T06:50:52.330330+0000","last_clean":"2024-09-19T06:50:52.330330+0000","last_became_active":"2024-09-19T06:49:08.186860+0000","last_became_peered":"2024-09-19T06:49:08.186860+0000","last_unstale":"2024-09-19T06:50:52.330330+0000","last_undegraded":"2024-09-19T06:50:52.330330+0000","last_fullsized":"2024-09-19T06:50:52.330330+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:45:06.236143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215603+0000","last_change":"2024-09-19T06:49:20.617028+0000","last_active":"2024-09-19T06:50:52.215603+0000","last_peered":"2024-09-19T06:50:52.215603+0000","last_clean":"2024-09-19T06:50:52.215603+0000","last_became_active":"2024-09-19T06:49:20.616443+0000","last_became_peered":"2024-09-19T06:49:20.616443+0000","last_unstale":"2024-09-19T06:50:52.215603+0000","last_undegraded":"2024-09-19T06:50:52.215603+0000","last_fullsized":"2024-09-19T06:50:52.215603+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:46:30.394051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267323+0000","last_change":"2024-09-19T06:49:10.386030+0000","last_active":"2024-09-19T06:50:52.267323+0000","last_peered":"2024-09-19T06:50:52.267323+0000","last_clean":"2024-09-19T06:50:52.267323+0000","last_became_active":"2024-09-19T06:49:10.385698+0000","last_became_peered":"2024-09-19T06:49:10.385698+0000","last_unstale":"2024-09-19T06:50:52.267323+0000","last_undegraded":"2024-09-19T06:50:52.267323+0000","last_fullsized":"2024-09-19T06:50:52.267323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:56:12.688507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'5","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.331196+0000","last_change":"2024-09-19T06:49:07.910172+0000","last_active":"2024-09-19T06:50:52.331196+0000","last_peered":"2024-09-19T06:50:52.331196+0000","last_clean":"2024-09-19T06:50:52.331196+0000","last_became_active":"2024-09-19T06:49:07.909798+0000","last_became_peered":"2024-09-19T06:49:07.909798+0000","last_unstale":"2024-09-19T06:50:52.331196+0000","last_undegraded":"2024-09-19T06:50:52.331196+0000","last_fullsized":"2024-09-19T06:50:52.331196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.592988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332850+0000","last_change":"2024-09-19T06:49:20.424233+0000","last_active":"2024-09-19T06:50:52.332850+0000","last_peered":"2024-09-19T06:50:52.332850+0000","last_clean":"2024-09-19T06:50:52.332850+0000","last_became_active":"2024-09-19T06:49:20.423797+0000","last_became_peered":"2024-09-19T06:49:20.423797+0000","last_unstale":"2024-09-19T06:50:52.332850+0000","last_undegraded":"2024-09-19T06:50:52.332850+0000","last_fullsized":"2024-09-19T06:50:52.332850+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:58:54.983250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222640+0000","last_change":"2024-09-19T06:49:10.376408+0000","last_active":"2024-09-19T06:50:52.222640+0000","last_peered":"2024-09-19T06:50:52.222640+0000","last_clean":"2024-09-19T06:50:52.222640+0000","last_became_active":"2024-09-19T06:49:10.376065+0000","last_became_peered":"2024-09-19T06:49:10.376065+0000","last_unstale":"2024-09-19T06:50:52.222640+0000","last_undegraded":"2024-09-19T06:50:52.222640+0000","last_fullsized":"2024-09-19T06:50:52.222640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:28:06.175670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":102,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:53.334707+0000","last_change":"2024-09-19T06:49:08.185183+0000","last_active":"2024-09-19T06:50:53.334707+0000","last_peered":"2024-09-19T06:50:53.334707+0000","last_clean":"2024-09-19T06:50:53.334707+0000","last_became_active":"2024-09-19T06:49:08.184738+0000","last_became_peered":"2024-09-19T06:49:08.184738+0000","last_unstale":"2024-09-19T06:50:53.334707+0000","last_undegraded":"2024-09-19T06:50:53.334707+0000","last_fullsized":"2024-09-19T06:50:53.334707+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:19:54.966966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.333375+0000","last_change":"2024-09-19T06:49:20.617986+0000","last_active":"2024-09-19T06:50:52.333375+0000","last_peered":"2024-09-19T06:50:52.333375+0000","last_clean":"2024-09-19T06:50:52.333375+0000","last_became_active":"2024-09-19T06:49:20.617325+0000","last_became_peered":"2024-09-19T06:49:20.617325+0000","last_unstale":"2024-09-19T06:50:52.333375+0000","last_undegraded":"2024-09-19T06:50:52.333375+0000","last_fullsized":"2024-09-19T06:50:52.333375+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:03.825005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332191+0000","last_change":"2024-09-19T06:49:10.385230+0000","last_active":"2024-09-19T06:50:52.332191+0000","last_peered":"2024-09-19T06:50:52.332191+0000","last_clean":"2024-09-19T06:50:52.332191+0000","last_became_active":"2024-09-19T06:49:10.384633+0000","last_became_peered":"2024-09-19T06:49:10.384633+0000","last_unstale":"2024-09-19T06:50:52.332191+0000","last_undegraded":"2024-09-19T06:50:52.332191+0000","last_fullsized":"2024-09-19T06:50:52.332191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:16:27.751338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'4","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222673+0000","last_change":"2024-09-19T06:49:07.908823+0000","last_active":"2024-09-19T06:50:52.222673+0000","last_peered":"2024-09-19T06:50:52.222673+0000","last_clean":"2024-09-19T06:50:52.222673+0000","last_became_active":"2024-09-19T06:49:07.908340+0000","last_became_peered":"2024-09-19T06:49:07.908340+0000","last_unstale":"2024-09-19T06:50:52.222673+0000","last_undegraded":"2024-09-19T06:50:52.222673+0000","last_fullsized":"2024-09-19T06:50:52.222673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:25.198073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.334852+0000","last_change":"2024-09-19T06:50:52.334222+0000","last_active":"2024-09-19T06:50:52.334852+0000","last_peered":"2024-09-19T06:50:52.334852+0000","last_clean":"2024-09-19T06:50:52.334852+0000","last_became_active":"2024-09-19T06:50:52.333646+0000","last_became_peered":"2024-09-19T06:50:52.333646+0000","last_unstale":"2024-09-19T06:50:52.334852+0000","last_undegraded":"2024-09-19T06:50:52.334852+0000","last_fullsized":"2024-09-19T06:50:52.334852+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:21.741223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218343+0000","last_change":"2024-09-19T06:49:10.374761+0000","last_active":"2024-09-19T06:50:52.218343+0000","last_peered":"2024-09-19T06:50:52.218343+0000","last_clean":"2024-09-19T06:50:52.218343+0000","last_became_active":"2024-09-19T06:49:10.374258+0000","last_became_peered":"2024-09-19T06:49:10.374258+0000","last_unstale":"2024-09-19T06:50:52.218343+0000","last_undegraded":"2024-09-19T06:50:52.218343+0000","last_fullsized":"2024-09-19T06:50:52.218343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:23:23.850658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'6","reported_seq":49,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217433+0000","last_change":"2024-09-19T06:49:07.912648+0000","last_active":"2024-09-19T06:50:52.217433+0000","last_peered":"2024-09-19T06:50:52.217433+0000","last_clean":"2024-09-19T06:50:52.217433+0000","last_became_active":"2024-09-19T06:49:07.912289+0000","last_became_peered":"2024-09-19T06:49:07.912289+0000","last_unstale":"2024-09-19T06:50:52.217433+0000","last_undegraded":"2024-09-19T06:50:52.217433+0000","last_fullsized":"2024-09-19T06:50:52.217433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:36.211030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222917+0000","last_change":"2024-09-19T06:47:53.158465+0000","last_active":"2024-09-19T06:50:52.222917+0000","last_peered":"2024-09-19T06:50:52.222917+0000","last_clean":"2024-09-19T06:50:52.222917+0000","last_became_active":"2024-09-19T06:47:53.158059+0000","last_became_peered":"2024-09-19T06:47:53.158059+0000","last_unstale":"2024-09-19T06:50:52.222917+0000","last_undegraded":"2024-09-19T06:50:52.222917+0000","last_fullsized":"2024-09-19T06:50:52.222917+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:44:13.613197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:44:13.613197+0000","last_clean_scrub_stamp":"2024-09-19T06:44:13.613197+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:16:41.310229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222892+0000","last_change":"2024-09-19T06:49:20.422112+0000","last_active":"2024-09-19T06:50:52.222892+0000","last_peered":"2024-09-19T06:50:52.222892+0000","last_clean":"2024-09-19T06:50:52.222892+0000","last_became_active":"2024-09-19T06:49:20.421720+0000","last_became_peered":"2024-09-19T06:49:20.421720+0000","last_unstale":"2024-09-19T06:50:52.222892+0000","last_undegraded":"2024-09-19T06:50:52.222892+0000","last_fullsized":"2024-09-19T06:50:52.222892+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:35:25.923946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223044+0000","last_change":"2024-09-19T06:49:10.379414+0000","last_active":"2024-09-19T06:50:52.223044+0000","last_peered":"2024-09-19T06:50:52.223044+0000","last_clean":"2024-09-19T06:50:52.223044+0000","last_became_active":"2024-09-19T06:49:10.379075+0000","last_became_peered":"2024-09-19T06:49:10.379075+0000","last_unstale":"2024-09-19T06:50:52.223044+0000","last_undegraded":"2024-09-19T06:50:52.223044+0000","last_fullsized":"2024-09-19T06:50:52.223044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:32:57.976929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'5","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:56.335865+0000","last_change":"2024-09-19T06:49:08.186883+0000","last_active":"2024-09-19T06:50:56.335865+0000","last_peered":"2024-09-19T06:50:56.335865+0000","last_clean":"2024-09-19T06:50:56.335865+0000","last_became_active":"2024-09-19T06:49:08.186472+0000","last_became_peered":"2024-09-19T06:49:08.186472+0000","last_unstale":"2024-09-19T06:50:56.335865+0000","last_undegraded":"2024-09-19T06:50:56.335865+0000","last_fullsized":"2024-09-19T06:50:56.335865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:58:12.146032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414698+0000","last_change":"2024-09-19T06:49:20.418542+0000","last_active":"2024-09-19T06:50:52.414698+0000","last_peered":"2024-09-19T06:50:52.414698+0000","last_clean":"2024-09-19T06:50:52.414698+0000","last_became_active":"2024-09-19T06:49:20.418044+0000","last_became_peered":"2024-09-19T06:49:20.418044+0000","last_unstale":"2024-09-19T06:50:52.414698+0000","last_undegraded":"2024-09-19T06:50:52.414698+0000","last_fullsized":"2024-09-19T06:50:52.414698+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:44:02.695919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223157+0000","last_change":"2024-09-19T06:49:10.373862+0000","last_active":"2024-09-19T06:50:52.223157+0000","last_peered":"2024-09-19T06:50:52.223157+0000","last_clean":"2024-09-19T06:50:52.223157+0000","last_became_active":"2024-09-19T06:49:10.373345+0000","last_became_peered":"2024-09-19T06:49:10.373345+0000","last_unstale":"2024-09-19T06:50:52.223157+0000","last_undegraded":"2024-09-19T06:50:52.223157+0000","last_fullsized":"2024-09-19T06:50:52.223157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:33.562984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'3","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223768+0000","last_change":"2024-09-19T06:49:07.909407+0000","last_active":"2024-09-19T06:50:52.223768+0000","last_peered":"2024-09-19T06:50:52.223768+0000","last_clean":"2024-09-19T06:50:52.223768+0000","last_became_active":"2024-09-19T06:49:07.909101+0000","last_became_peered":"2024-09-19T06:49:07.909101+0000","last_unstale":"2024-09-19T06:50:52.223768+0000","last_undegraded":"2024-09-19T06:50:52.223768+0000","last_fullsized":"2024-09-19T06:50:52.223768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:08:47.317018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.766985+0000","last_change":"2024-09-19T06:49:20.425109+0000","last_active":"2024-09-19T06:50:52.766985+0000","last_peered":"2024-09-19T06:50:52.766985+0000","last_clean":"2024-09-19T06:50:52.766985+0000","last_became_active":"2024-09-19T06:49:20.424701+0000","last_became_peered":"2024-09-19T06:49:20.424701+0000","last_unstale":"2024-09-19T06:50:52.766985+0000","last_undegraded":"2024-09-19T06:50:52.766985+0000","last_fullsized":"2024-09-19T06:50:52.766985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:13:10.878691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223092+0000","last_change":"2024-09-19T06:49:10.374089+0000","last_active":"2024-09-19T06:50:52.223092+0000","last_peered":"2024-09-19T06:50:52.223092+0000","last_clean":"2024-09-19T06:50:52.223092+0000","last_became_active":"2024-09-19T06:49:10.373729+0000","last_became_peered":"2024-09-19T06:49:10.373729+0000","last_unstale":"2024-09-19T06:50:52.223092+0000","last_undegraded":"2024-09-19T06:50:52.223092+0000","last_fullsized":"2024-09-19T06:50:52.223092+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:52.876874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"93'5","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.333165+0000","last_change":"2024-09-19T06:49:07.909616+0000","last_active":"2024-09-19T06:50:52.333165+0000","last_peered":"2024-09-19T06:50:52.333165+0000","last_clean":"2024-09-19T06:50:52.333165+0000","last_became_active":"2024-09-19T06:49:07.909172+0000","last_became_peered":"2024-09-19T06:49:07.909172+0000","last_unstale":"2024-09-19T06:50:52.333165+0000","last_undegraded":"2024-09-19T06:50:52.333165+0000","last_fullsized":"2024-09-19T06:50:52.333165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:57:27.685639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222460+0000","last_change":"2024-09-19T06:49:20.419188+0000","last_active":"2024-09-19T06:50:52.222460+0000","last_peered":"2024-09-19T06:50:52.222460+0000","last_clean":"2024-09-19T06:50:52.222460+0000","last_became_active":"2024-09-19T06:49:20.418641+0000","last_became_peered":"2024-09-19T06:49:20.418641+0000","last_unstale":"2024-09-19T06:50:52.222460+0000","last_undegraded":"2024-09-19T06:50:52.222460+0000","last_fullsized":"2024-09-19T06:50:52.222460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:06:05.278027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220813+0000","last_change":"2024-09-19T06:49:10.378491+0000","last_active":"2024-09-19T06:50:52.220813+0000","last_peered":"2024-09-19T06:50:52.220813+0000","last_clean":"2024-09-19T06:50:52.220813+0000","last_became_active":"2024-09-19T06:49:10.378141+0000","last_became_peered":"2024-09-19T06:49:10.378141+0000","last_unstale":"2024-09-19T06:50:52.220813+0000","last_undegraded":"2024-09-19T06:50:52.220813+0000","last_fullsized":"2024-09-19T06:50:52.220813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:57:39.195604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216307+0000","last_change":"2024-09-19T06:49:07.909836+0000","last_active":"2024-09-19T06:50:52.216307+0000","last_peered":"2024-09-19T06:50:52.216307+0000","last_clean":"2024-09-19T06:50:52.216307+0000","last_became_active":"2024-09-19T06:49:07.909527+0000","last_became_peered":"2024-09-19T06:49:07.909527+0000","last_unstale":"2024-09-19T06:50:52.216307+0000","last_undegraded":"2024-09-19T06:50:52.216307+0000","last_fullsized":"2024-09-19T06:50:52.216307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:51.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222641+0000","last_change":"2024-09-19T06:49:20.426184+0000","last_active":"2024-09-19T06:50:52.222641+0000","last_peered":"2024-09-19T06:50:52.222641+0000","last_clean":"2024-09-19T06:50:52.222641+0000","last_became_active":"2024-09-19T06:49:20.425844+0000","last_became_peered":"2024-09-19T06:49:20.425844+0000","last_unstale":"2024-09-19T06:50:52.222641+0000","last_undegraded":"2024-09-19T06:50:52.222641+0000","last_fullsized":"2024-09-19T06:50:52.222641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.120462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222862+0000","last_change":"2024-09-19T06:49:10.380589+0000","last_active":"2024-09-19T06:50:52.222862+0000","last_peered":"2024-09-19T06:50:52.222862+0000","last_clean":"2024-09-19T06:50:52.222862+0000","last_became_active":"2024-09-19T06:49:10.380263+0000","last_became_peered":"2024-09-19T06:49:10.380263+0000","last_unstale":"2024-09-19T06:50:52.222862+0000","last_undegraded":"2024-09-19T06:50:52.222862+0000","last_fullsized":"2024-09-19T06:50:52.222862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:30:48.426240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"93'17","reported_seq":74,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215601+0000","last_change":"2024-09-19T06:49:52.199971+0000","last_active":"2024-09-19T06:50:52.215601+0000","last_peered":"2024-09-19T06:50:52.215601+0000","last_clean":"2024-09-19T06:50:52.215601+0000","last_became_active":"2024-09-19T06:49:52.191887+0000","last_became_peered":"2024-09-19T06:49:52.191887+0000","last_unstale":"2024-09-19T06:50:52.215601+0000","last_undegraded":"2024-09-19T06:50:52.215601+0000","last_fullsized":"2024-09-19T06:50:52.215601+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:22:40.775893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.333002+0000","last_change":"2024-09-19T06:49:20.435213+0000","last_active":"2024-09-19T06:50:52.333002+0000","last_peered":"2024-09-19T06:50:52.333002+0000","last_clean":"2024-09-19T06:50:52.333002+0000","last_became_active":"2024-09-19T06:49:20.434776+0000","last_became_peered":"2024-09-19T06:49:20.434776+0000","last_unstale":"2024-09-19T06:50:52.333002+0000","last_undegraded":"2024-09-19T06:50:52.333002+0000","last_fullsized":"2024-09-19T06:50:52.333002+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:37.067947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222464+0000","last_change":"2024-09-19T06:49:10.375394+0000","last_active":"2024-09-19T06:50:52.222464+0000","last_peered":"2024-09-19T06:50:52.222464+0000","last_clean":"2024-09-19T06:50:52.222464+0000","last_became_active":"2024-09-19T06:49:10.375056+0000","last_became_peered":"2024-09-19T06:49:10.375056+0000","last_unstale":"2024-09-19T06:50:52.222464+0000","last_undegraded":"2024-09-19T06:50:52.222464+0000","last_fullsized":"2024-09-19T06:50:52.222464+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:18:40.158481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'6","reported_seq":108,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:56.338792+0000","last_change":"2024-09-19T06:49:07.910215+0000","last_active":"2024-09-19T06:50:56.338792+0000","last_peered":"2024-09-19T06:50:56.338792+0000","last_clean":"2024-09-19T06:50:56.338792+0000","last_became_active":"2024-09-19T06:49:07.909908+0000","last_became_peered":"2024-09-19T06:49:07.909908+0000","last_unstale":"2024-09-19T06:50:56.338792+0000","last_undegraded":"2024-09-19T06:50:56.338792+0000","last_fullsized":"2024-09-19T06:50:56.338792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:45:05.432960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":63,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330305+0000","last_change":"2024-09-19T06:49:20.434400+0000","last_active":"2024-09-19T06:50:52.330305+0000","last_peered":"2024-09-19T06:50:52.330305+0000","last_clean":"2024-09-19T06:50:52.330305+0000","last_became_active":"2024-09-19T06:49:20.433877+0000","last_became_peered":"2024-09-19T06:49:20.433877+0000","last_unstale":"2024-09-19T06:50:52.330305+0000","last_undegraded":"2024-09-19T06:50:52.330305+0000","last_fullsized":"2024-09-19T06:50:52.330305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:24:16.378509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222695+0000","last_change":"2024-09-19T06:49:10.378048+0000","last_active":"2024-09-19T06:50:52.222695+0000","last_peered":"2024-09-19T06:50:52.222695+0000","last_clean":"2024-09-19T06:50:52.222695+0000","last_became_active":"2024-09-19T06:49:10.377710+0000","last_became_peered":"2024-09-19T06:49:10.377710+0000","last_unstale":"2024-09-19T06:50:52.222695+0000","last_undegraded":"2024-09-19T06:50:52.222695+0000","last_fullsized":"2024-09-19T06:50:52.222695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:41:54.620218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"93'5","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.229301+0000","last_change":"2024-09-19T06:50:52.229301+0000","last_active":"2024-09-19T06:50:52.229301+0000","last_peered":"2024-09-19T06:50:52.229301+0000","last_clean":"2024-09-19T06:50:52.229301+0000","last_became_active":"2024-09-19T06:50:52.223798+0000","last_became_peered":"2024-09-19T06:50:52.223798+0000","last_unstale":"2024-09-19T06:50:52.229301+0000","last_undegraded":"2024-09-19T06:50:52.229301+0000","last_fullsized":"2024-09-19T06:50:52.229301+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:57:42.199295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.334454+0000","last_change":"2024-09-19T06:50:52.334454+0000","last_active":"2024-09-19T06:50:52.334454+0000","last_peered":"2024-09-19T06:50:52.334454+0000","last_clean":"2024-09-19T06:50:52.334454+0000","last_became_active":"2024-09-19T06:50:52.333886+0000","last_became_peered":"2024-09-19T06:50:52.333886+0000","last_unstale":"2024-09-19T06:50:52.334454+0000","last_undegraded":"2024-09-19T06:50:52.334454+0000","last_fullsized":"2024-09-19T06:50:52.334454+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:41:08.771168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332737+0000","last_change":"2024-09-19T06:49:10.382223+0000","last_active":"2024-09-19T06:50:52.332737+0000","last_peered":"2024-09-19T06:50:52.332737+0000","last_clean":"2024-09-19T06:50:52.332737+0000","last_became_active":"2024-09-19T06:49:10.381869+0000","last_became_peered":"2024-09-19T06:49:10.381869+0000","last_unstale":"2024-09-19T06:50:52.332737+0000","last_undegraded":"2024-09-19T06:50:52.332737+0000","last_fullsized":"2024-09-19T06:50:52.332737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:23:34.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"93'9","reported_seq":50,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330978+0000","last_change":"2024-09-19T06:49:07.907812+0000","last_active":"2024-09-19T06:50:52.330978+0000","last_peered":"2024-09-19T06:50:52.330978+0000","last_clean":"2024-09-19T06:50:52.330978+0000","last_became_active":"2024-09-19T06:49:07.907524+0000","last_became_peered":"2024-09-19T06:49:07.907524+0000","last_unstale":"2024-09-19T06:50:52.330978+0000","last_undegraded":"2024-09-19T06:50:52.330978+0000","last_fullsized":"2024-09-19T06:50:52.330978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:26:43.780682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223572+0000","last_change":"2024-09-19T06:49:20.436311+0000","last_active":"2024-09-19T06:50:52.223572+0000","last_peered":"2024-09-19T06:50:52.223572+0000","last_clean":"2024-09-19T06:50:52.223572+0000","last_became_active":"2024-09-19T06:49:20.435936+0000","last_became_peered":"2024-09-19T06:49:20.435936+0000","last_unstale":"2024-09-19T06:50:52.223572+0000","last_undegraded":"2024-09-19T06:50:52.223572+0000","last_fullsized":"2024-09-19T06:50:52.223572+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:54:04.688922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216102+0000","last_change":"2024-09-19T06:49:10.373764+0000","last_active":"2024-09-19T06:50:52.216102+0000","last_peered":"2024-09-19T06:50:52.216102+0000","last_clean":"2024-09-19T06:50:52.216102+0000","last_became_active":"2024-09-19T06:49:10.373324+0000","last_became_peered":"2024-09-19T06:49:10.373324+0000","last_unstale":"2024-09-19T06:50:52.216102+0000","last_undegraded":"2024-09-19T06:50:52.216102+0000","last_fullsized":"2024-09-19T06:50:52.216102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:25:03.898831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'34","reported_seq":193,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:54.331301+0000","last_change":"2024-09-19T06:49:07.907319+0000","last_active":"2024-09-19T06:50:54.331301+0000","last_peered":"2024-09-19T06:50:54.331301+0000","last_clean":"2024-09-19T06:50:54.331301+0000","last_became_active":"2024-09-19T06:49:07.906888+0000","last_became_peered":"2024-09-19T06:49:07.906888+0000","last_unstale":"2024-09-19T06:50:54.331301+0000","last_undegraded":"2024-09-19T06:50:54.331301+0000","last_fullsized":"2024-09-19T06:50:54.331301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:49.051957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62661,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":121,"num_write":36,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.329885+0000","last_change":"2024-09-19T06:49:20.428184+0000","last_active":"2024-09-19T06:50:52.329885+0000","last_peered":"2024-09-19T06:50:52.329885+0000","last_clean":"2024-09-19T06:50:52.329885+0000","last_became_active":"2024-09-19T06:49:20.427631+0000","last_became_peered":"2024-09-19T06:49:20.427631+0000","last_unstale":"2024-09-19T06:50:52.329885+0000","last_undegraded":"2024-09-19T06:50:52.329885+0000","last_fullsized":"2024-09-19T06:50:52.329885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:14:51.063413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414971+0000","last_change":"2024-09-19T06:49:10.373073+0000","last_active":"2024-09-19T06:50:52.414971+0000","last_peered":"2024-09-19T06:50:52.414971+0000","last_clean":"2024-09-19T06:50:52.414971+0000","last_became_active":"2024-09-19T06:49:10.371611+0000","last_became_peered":"2024-09-19T06:49:10.371611+0000","last_unstale":"2024-09-19T06:50:52.414971+0000","last_undegraded":"2024-09-19T06:50:52.414971+0000","last_fullsized":"2024-09-19T06:50:52.414971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:58.228946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"93'5","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223763+0000","last_change":"2024-09-19T06:49:08.186128+0000","last_active":"2024-09-19T06:50:52.223763+0000","last_peered":"2024-09-19T06:50:52.223763+0000","last_clean":"2024-09-19T06:50:52.223763+0000","last_became_active":"2024-09-19T06:49:08.185817+0000","last_became_peered":"2024-09-19T06:49:08.185817+0000","last_unstale":"2024-09-19T06:50:52.223763+0000","last_undegraded":"2024-09-19T06:50:52.223763+0000","last_fullsized":"2024-09-19T06:50:52.223763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:59:53.066405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217185+0000","last_change":"2024-09-19T06:49:20.431251+0000","last_active":"2024-09-19T06:50:52.217185+0000","last_peered":"2024-09-19T06:50:52.217185+0000","last_clean":"2024-09-19T06:50:52.217185+0000","last_became_active":"2024-09-19T06:49:20.430709+0000","last_became_peered":"2024-09-19T06:49:20.430709+0000","last_unstale":"2024-09-19T06:50:52.217185+0000","last_undegraded":"2024-09-19T06:50:52.217185+0000","last_fullsized":"2024-09-19T06:50:52.217185+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:55:06.742567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767345+0000","last_change":"2024-09-19T06:49:10.379045+0000","last_active":"2024-09-19T06:50:52.767345+0000","last_peered":"2024-09-19T06:50:52.767345+0000","last_clean":"2024-09-19T06:50:52.767345+0000","last_became_active":"2024-09-19T06:49:10.378481+0000","last_became_peered":"2024-09-19T06:49:10.378481+0000","last_unstale":"2024-09-19T06:50:52.767345+0000","last_undegraded":"2024-09-19T06:50:52.767345+0000","last_fullsized":"2024-09-19T06:50:52.767345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:29:55.770670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"93'4","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223019+0000","last_change":"2024-09-19T06:49:07.910246+0000","last_active":"2024-09-19T06:50:52.223019+0000","last_peered":"2024-09-19T06:50:52.223019+0000","last_clean":"2024-09-19T06:50:52.223019+0000","last_became_active":"2024-09-19T06:49:07.909855+0000","last_became_peered":"2024-09-19T06:49:07.909855+0000","last_unstale":"2024-09-19T06:50:52.223019+0000","last_undegraded":"2024-09-19T06:50:52.223019+0000","last_fullsized":"2024-09-19T06:50:52.223019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:33:24.011418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.329642+0000","last_change":"2024-09-19T06:49:20.421847+0000","last_active":"2024-09-19T06:50:52.329642+0000","last_peered":"2024-09-19T06:50:52.329642+0000","last_clean":"2024-09-19T06:50:52.329642+0000","last_became_active":"2024-09-19T06:49:20.421391+0000","last_became_peered":"2024-09-19T06:49:20.421391+0000","last_unstale":"2024-09-19T06:50:52.329642+0000","last_undegraded":"2024-09-19T06:50:52.329642+0000","last_fullsized":"2024-09-19T06:50:52.329642+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:54:24.207872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.415192+0000","last_change":"2024-09-19T06:49:10.376029+0000","last_active":"2024-09-19T06:50:52.415192+0000","last_peered":"2024-09-19T06:50:52.415192+0000","last_clean":"2024-09-19T06:50:52.415192+0000","last_became_active":"2024-09-19T06:49:10.374242+0000","last_became_peered":"2024-09-19T06:49:10.374242+0000","last_unstale":"2024-09-19T06:50:52.415192+0000","last_undegraded":"2024-09-19T06:50:52.415192+0000","last_fullsized":"2024-09-19T06:50:52.415192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:59:43.722173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:54.331298+0000","last_change":"2024-09-19T06:49:07.908090+0000","last_active":"2024-09-19T06:50:54.331298+0000","last_peered":"2024-09-19T06:50:54.331298+0000","last_clean":"2024-09-19T06:50:54.331298+0000","last_became_active":"2024-09-19T06:49:07.906933+0000","last_became_peered":"2024-09-19T06:49:07.906933+0000","last_unstale":"2024-09-19T06:50:54.331298+0000","last_undegraded":"2024-09-19T06:50:54.331298+0000","last_fullsized":"2024-09-19T06:50:54.331298+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:21.711371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218024+0000","last_change":"2024-09-19T06:49:20.433797+0000","last_active":"2024-09-19T06:50:52.218024+0000","last_peered":"2024-09-19T06:50:52.218024+0000","last_clean":"2024-09-19T06:50:52.218024+0000","last_became_active":"2024-09-19T06:49:20.433419+0000","last_became_peered":"2024-09-19T06:49:20.433419+0000","last_unstale":"2024-09-19T06:50:52.218024+0000","last_undegraded":"2024-09-19T06:50:52.218024+0000","last_fullsized":"2024-09-19T06:50:52.218024+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:18:39.691678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767189+0000","last_change":"2024-09-19T06:49:20.424995+0000","last_active":"2024-09-19T06:50:52.767189+0000","last_peered":"2024-09-19T06:50:52.767189+0000","last_clean":"2024-09-19T06:50:52.767189+0000","last_became_active":"2024-09-19T06:49:20.424589+0000","last_became_peered":"2024-09-19T06:49:20.424589+0000","last_unstale":"2024-09-19T06:50:52.767189+0000","last_undegraded":"2024-09-19T06:50:52.767189+0000","last_fullsized":"2024-09-19T06:50:52.767189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:17.968571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.329723+0000","last_change":"2024-09-19T06:49:10.382157+0000","last_active":"2024-09-19T06:50:52.329723+0000","last_peered":"2024-09-19T06:50:52.329723+0000","last_clean":"2024-09-19T06:50:52.329723+0000","last_became_active":"2024-09-19T06:49:10.381848+0000","last_became_peered":"2024-09-19T06:49:10.381848+0000","last_unstale":"2024-09-19T06:50:52.329723+0000","last_undegraded":"2024-09-19T06:50:52.329723+0000","last_fullsized":"2024-09-19T06:50:52.329723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:10.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"93'3","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222841+0000","last_change":"2024-09-19T06:49:07.906088+0000","last_active":"2024-09-19T06:50:52.222841+0000","last_peered":"2024-09-19T06:50:52.222841+0000","last_clean":"2024-09-19T06:50:52.222841+0000","last_became_active":"2024-09-19T06:49:07.905798+0000","last_became_peered":"2024-09-19T06:49:07.905798+0000","last_unstale":"2024-09-19T06:50:52.222841+0000","last_undegraded":"2024-09-19T06:50:52.222841+0000","last_fullsized":"2024-09-19T06:50:52.222841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:02.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220464+0000","last_change":"2024-09-19T06:49:20.432275+0000","last_active":"2024-09-19T06:50:52.220464+0000","last_peered":"2024-09-19T06:50:52.220464+0000","last_clean":"2024-09-19T06:50:52.220464+0000","last_became_active":"2024-09-19T06:49:20.431951+0000","last_became_peered":"2024-09-19T06:49:20.431951+0000","last_unstale":"2024-09-19T06:50:52.220464+0000","last_undegraded":"2024-09-19T06:50:52.220464+0000","last_fullsized":"2024-09-19T06:50:52.220464+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:38.649557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218570+0000","last_change":"2024-09-19T06:49:10.368061+0000","last_active":"2024-09-19T06:50:52.218570+0000","last_peered":"2024-09-19T06:50:52.218570+0000","last_clean":"2024-09-19T06:50:52.218570+0000","last_became_active":"2024-09-19T06:49:10.367732+0000","last_became_peered":"2024-09-19T06:49:10.367732+0000","last_unstale":"2024-09-19T06:50:52.218570+0000","last_undegraded":"2024-09-19T06:50:52.218570+0000","last_fullsized":"2024-09-19T06:50:52.218570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:37.623628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"93'3","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332201+0000","last_change":"2024-09-19T06:49:07.910722+0000","last_active":"2024-09-19T06:50:52.332201+0000","last_peered":"2024-09-19T06:50:52.332201+0000","last_clean":"2024-09-19T06:50:52.332201+0000","last_became_active":"2024-09-19T06:49:07.909960+0000","last_became_peered":"2024-09-19T06:49:07.909960+0000","last_unstale":"2024-09-19T06:50:52.332201+0000","last_undegraded":"2024-09-19T06:50:52.332201+0000","last_fullsized":"2024-09-19T06:50:52.332201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:40:18.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"93'7","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267267+0000","last_change":"2024-09-19T06:49:52.411869+0000","last_active":"2024-09-19T06:50:52.267267+0000","last_peered":"2024-09-19T06:50:52.267267+0000","last_clean":"2024-09-19T06:50:52.267267+0000","last_became_active":"2024-09-19T06:49:52.411491+0000","last_became_peered":"2024-09-19T06:49:52.411491+0000","last_unstale":"2024-09-19T06:50:52.267267+0000","last_undegraded":"2024-09-19T06:50:52.267267+0000","last_fullsized":"2024-09-19T06:50:52.267267+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:05:21.222348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223274+0000","last_change":"2024-09-19T06:49:10.388831+0000","last_active":"2024-09-19T06:50:52.223274+0000","last_peered":"2024-09-19T06:50:52.223274+0000","last_clean":"2024-09-19T06:50:52.223274+0000","last_became_active":"2024-09-19T06:49:10.388556+0000","last_became_peered":"2024-09-19T06:49:10.388556+0000","last_unstale":"2024-09-19T06:50:52.223274+0000","last_undegraded":"2024-09-19T06:50:52.223274+0000","last_fullsized":"2024-09-19T06:50:52.223274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:28:20.319429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330796+0000","last_change":"2024-09-19T06:49:20.424827+0000","last_active":"2024-09-19T06:50:52.330796+0000","last_peered":"2024-09-19T06:50:52.330796+0000","last_clean":"2024-09-19T06:50:52.330796+0000","last_became_active":"2024-09-19T06:49:20.424318+0000","last_became_peered":"2024-09-19T06:49:20.424318+0000","last_unstale":"2024-09-19T06:50:52.330796+0000","last_undegraded":"2024-09-19T06:50:52.330796+0000","last_fullsized":"2024-09-19T06:50:52.330796+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:37.814108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":108,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:53.330345+0000","last_change":"2024-09-19T06:49:52.188404+0000","last_active":"2024-09-19T06:50:53.330345+0000","last_peered":"2024-09-19T06:50:53.330345+0000","last_clean":"2024-09-19T06:50:53.330345+0000","last_became_active":"2024-09-19T06:49:52.187918+0000","last_became_peered":"2024-09-19T06:49:52.187918+0000","last_unstale":"2024-09-19T06:50:53.330345+0000","last_undegraded":"2024-09-19T06:50:53.330345+0000","last_fullsized":"2024-09-19T06:50:53.330345+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:53:38.209357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215711+0000","last_change":"2024-09-19T06:49:10.374333+0000","last_active":"2024-09-19T06:50:52.215711+0000","last_peered":"2024-09-19T06:50:52.215711+0000","last_clean":"2024-09-19T06:50:52.215711+0000","last_became_active":"2024-09-19T06:49:10.373944+0000","last_became_peered":"2024-09-19T06:49:10.373944+0000","last_unstale":"2024-09-19T06:50:52.215711+0000","last_undegraded":"2024-09-19T06:50:52.215711+0000","last_fullsized":"2024-09-19T06:50:52.215711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:01:43.609310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.331039+0000","last_change":"2024-09-19T06:49:20.417030+0000","last_active":"2024-09-19T06:50:52.331039+0000","last_peered":"2024-09-19T06:50:52.331039+0000","last_clean":"2024-09-19T06:50:52.331039+0000","last_became_active":"2024-09-19T06:49:20.416620+0000","last_became_peered":"2024-09-19T06:49:20.416620+0000","last_unstale":"2024-09-19T06:50:52.331039+0000","last_undegraded":"2024-09-19T06:50:52.331039+0000","last_fullsized":"2024-09-19T06:50:52.331039+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:47:18.177613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220233+0000","last_change":"2024-09-19T06:49:10.372503+0000","last_active":"2024-09-19T06:50:52.220233+0000","last_peered":"2024-09-19T06:50:52.220233+0000","last_clean":"2024-09-19T06:50:52.220233+0000","last_became_active":"2024-09-19T06:49:10.372201+0000","last_became_peered":"2024-09-19T06:49:10.372201+0000","last_unstale":"2024-09-19T06:50:52.220233+0000","last_undegraded":"2024-09-19T06:50:52.220233+0000","last_fullsized":"2024-09-19T06:50:52.220233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:31:44.270457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.331025+0000","last_change":"2024-09-19T06:49:10.374149+0000","last_active":"2024-09-19T06:50:52.331025+0000","last_peered":"2024-09-19T06:50:52.331025+0000","last_clean":"2024-09-19T06:50:52.331025+0000","last_became_active":"2024-09-19T06:49:10.372570+0000","last_became_peered":"2024-09-19T06:49:10.372570+0000","last_unstale":"2024-09-19T06:50:52.331025+0000","last_undegraded":"2024-09-19T06:50:52.331025+0000","last_fullsized":"2024-09-19T06:50:52.331025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:55:56.086106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330600+0000","last_change":"2024-09-19T06:49:10.373383+0000","last_active":"2024-09-19T06:50:52.330600+0000","last_peered":"2024-09-19T06:50:52.330600+0000","last_clean":"2024-09-19T06:50:52.330600+0000","last_became_active":"2024-09-19T06:49:10.372144+0000","last_became_peered":"2024-09-19T06:49:10.372144+0000","last_unstale":"2024-09-19T06:50:52.330600+0000","last_undegraded":"2024-09-19T06:50:52.330600+0000","last_fullsized":"2024-09-19T06:50:52.330600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:08:34.000815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767048+0000","last_change":"2024-09-19T06:49:10.379720+0000","last_active":"2024-09-19T06:50:52.767048+0000","last_peered":"2024-09-19T06:50:52.767048+0000","last_clean":"2024-09-19T06:50:52.767048+0000","last_became_active":"2024-09-19T06:49:10.379325+0000","last_became_peered":"2024-09-19T06:49:10.379325+0000","last_unstale":"2024-09-19T06:50:52.767048+0000","last_undegraded":"2024-09-19T06:50:52.767048+0000","last_fullsized":"2024-09-19T06:50:52.767048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:17:58.471617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332999+0000","last_change":"2024-09-19T06:49:52.194997+0000","last_active":"2024-09-19T06:50:52.332999+0000","last_peered":"2024-09-19T06:50:52.332999+0000","last_clean":"2024-09-19T06:50:52.332999+0000","last_became_active":"2024-09-19T06:49:52.194662+0000","last_became_peered":"2024-09-19T06:49:52.194662+0000","last_unstale":"2024-09-19T06:50:52.332999+0000","last_undegraded":"2024-09-19T06:50:52.332999+0000","last_fullsized":"2024-09-19T06:50:52.332999+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:18.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220724+0000","last_change":"2024-09-19T06:49:10.375868+0000","last_active":"2024-09-19T06:50:52.220724+0000","last_peered":"2024-09-19T06:50:52.220724+0000","last_clean":"2024-09-19T06:50:52.220724+0000","last_became_active":"2024-09-19T06:49:10.375552+0000","last_became_peered":"2024-09-19T06:49:10.375552+0000","last_unstale":"2024-09-19T06:50:52.220724+0000","last_undegraded":"2024-09-19T06:50:52.220724+0000","last_fullsized":"2024-09-19T06:50:52.220724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:55:22.292330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330966+0000","last_change":"2024-09-19T06:49:10.365959+0000","last_active":"2024-09-19T06:50:52.330966+0000","last_peered":"2024-09-19T06:50:52.330966+0000","last_clean":"2024-09-19T06:50:52.330966+0000","last_became_active":"2024-09-19T06:49:10.365018+0000","last_became_peered":"2024-09-19T06:49:10.365018+0000","last_unstale":"2024-09-19T06:50:52.330966+0000","last_undegraded":"2024-09-19T06:50:52.330966+0000","last_fullsized":"2024-09-19T06:50:52.330966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:45:50.417270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218366+0000","last_change":"2024-09-19T06:49:10.372988+0000","last_active":"2024-09-19T06:50:52.218366+0000","last_peered":"2024-09-19T06:50:52.218366+0000","last_clean":"2024-09-19T06:50:52.218366+0000","last_became_active":"2024-09-19T06:49:10.368995+0000","last_became_peered":"2024-09-19T06:49:10.368995+0000","last_unstale":"2024-09-19T06:50:52.218366+0000","last_undegraded":"2024-09-19T06:50:52.218366+0000","last_fullsized":"2024-09-19T06:50:52.218366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:29:04.939254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222301+0000","last_change":"2024-09-19T06:49:52.228705+0000","last_active":"2024-09-19T06:50:52.222301+0000","last_peered":"2024-09-19T06:50:52.222301+0000","last_clean":"2024-09-19T06:50:52.222301+0000","last_became_active":"2024-09-19T06:49:52.228371+0000","last_became_peered":"2024-09-19T06:49:52.228371+0000","last_unstale":"2024-09-19T06:50:52.222301+0000","last_undegraded":"2024-09-19T06:50:52.222301+0000","last_fullsized":"2024-09-19T06:50:52.222301+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:24:58.959892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218662+0000","last_change":"2024-09-19T06:49:10.372657+0000","last_active":"2024-09-19T06:50:52.218662+0000","last_peered":"2024-09-19T06:50:52.218662+0000","last_clean":"2024-09-19T06:50:52.218662+0000","last_became_active":"2024-09-19T06:49:10.371641+0000","last_became_peered":"2024-09-19T06:49:10.371641+0000","last_unstale":"2024-09-19T06:50:52.218662+0000","last_undegraded":"2024-09-19T06:50:52.218662+0000","last_fullsized":"2024-09-19T06:50:52.218662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:54:09.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414907+0000","last_change":"2024-09-19T06:49:10.370765+0000","last_active":"2024-09-19T06:50:52.414907+0000","last_peered":"2024-09-19T06:50:52.414907+0000","last_clean":"2024-09-19T06:50:52.414907+0000","last_became_active":"2024-09-19T06:49:10.369741+0000","last_became_peered":"2024-09-19T06:49:10.369741+0000","last_unstale":"2024-09-19T06:50:52.414907+0000","last_undegraded":"2024-09-19T06:50:52.414907+0000","last_fullsized":"2024-09-19T06:50:52.414907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:35:31.377034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332767+0000","last_change":"2024-09-19T06:49:10.386328+0000","last_active":"2024-09-19T06:50:52.332767+0000","last_peered":"2024-09-19T06:50:52.332767+0000","last_clean":"2024-09-19T06:50:52.332767+0000","last_became_active":"2024-09-19T06:49:10.385970+0000","last_became_peered":"2024-09-19T06:49:10.385970+0000","last_unstale":"2024-09-19T06:50:52.332767+0000","last_undegraded":"2024-09-19T06:50:52.332767+0000","last_fullsized":"2024-09-19T06:50:52.332767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:52:27.404825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220439+0000","last_change":"2024-09-19T06:49:10.372047+0000","last_active":"2024-09-19T06:50:52.220439+0000","last_peered":"2024-09-19T06:50:52.220439+0000","last_clean":"2024-09-19T06:50:52.220439+0000","last_became_active":"2024-09-19T06:49:10.371743+0000","last_became_peered":"2024-09-19T06:49:10.371743+0000","last_unstale":"2024-09-19T06:50:52.220439+0000","last_undegraded":"2024-09-19T06:50:52.220439+0000","last_fullsized":"2024-09-19T06:50:52.220439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:06.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.331192+0000","last_change":"2024-09-19T06:49:10.373221+0000","last_active":"2024-09-19T06:50:52.331192+0000","last_peered":"2024-09-19T06:50:52.331192+0000","last_clean":"2024-09-19T06:50:52.331192+0000","last_became_active":"2024-09-19T06:49:10.372859+0000","last_became_peered":"2024-09-19T06:49:10.372859+0000","last_unstale":"2024-09-19T06:50:52.331192+0000","last_undegraded":"2024-09-19T06:50:52.331192+0000","last_fullsized":"2024-09-19T06:50:52.331192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:53:19.333758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330296+0000","last_change":"2024-09-19T06:49:10.371359+0000","last_active":"2024-09-19T06:50:52.330296+0000","last_peered":"2024-09-19T06:50:52.330296+0000","last_clean":"2024-09-19T06:50:52.330296+0000","last_became_active":"2024-09-19T06:49:10.371057+0000","last_became_peered":"2024-09-19T06:49:10.371057+0000","last_unstale":"2024-09-19T06:50:52.330296+0000","last_undegraded":"2024-09-19T06:50:52.330296+0000","last_fullsized":"2024-09-19T06:50:52.330296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:18:40.387791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220873+0000","last_change":"2024-09-19T06:49:10.381371+0000","last_active":"2024-09-19T06:50:52.220873+0000","last_peered":"2024-09-19T06:50:52.220873+0000","last_clean":"2024-09-19T06:50:52.220873+0000","last_became_active":"2024-09-19T06:49:10.381017+0000","last_became_peered":"2024-09-19T06:49:10.381017+0000","last_unstale":"2024-09-19T06:50:52.220873+0000","last_undegraded":"2024-09-19T06:50:52.220873+0000","last_fullsized":"2024-09-19T06:50:52.220873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:17:54.998597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414586+0000","last_change":"2024-09-19T06:49:10.377596+0000","last_active":"2024-09-19T06:50:52.414586+0000","last_peered":"2024-09-19T06:50:52.414586+0000","last_clean":"2024-09-19T06:50:52.414586+0000","last_became_active":"2024-09-19T06:49:10.377271+0000","last_became_peered":"2024-09-19T06:49:10.377271+0000","last_unstale":"2024-09-19T06:50:52.414586+0000","last_undegraded":"2024-09-19T06:50:52.414586+0000","last_fullsized":"2024-09-19T06:50:52.414586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:32:00.797866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.766828+0000","last_change":"2024-09-19T06:49:52.194675+0000","last_active":"2024-09-19T06:50:52.766828+0000","last_peered":"2024-09-19T06:50:52.766828+0000","last_clean":"2024-09-19T06:50:52.766828+0000","last_became_active":"2024-09-19T06:49:52.194007+0000","last_became_peered":"2024-09-19T06:49:52.194007+0000","last_unstale":"2024-09-19T06:50:52.766828+0000","last_undegraded":"2024-09-19T06:50:52.766828+0000","last_fullsized":"2024-09-19T06:50:52.766828+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:45:33.109454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.766777+0000","last_change":"2024-09-19T06:49:10.387416+0000","last_active":"2024-09-19T06:50:52.766777+0000","last_peered":"2024-09-19T06:50:52.766777+0000","last_clean":"2024-09-19T06:50:52.766777+0000","last_became_active":"2024-09-19T06:49:10.384083+0000","last_became_peered":"2024-09-19T06:49:10.384083+0000","last_unstale":"2024-09-19T06:50:52.766777+0000","last_undegraded":"2024-09-19T06:50:52.766777+0000","last_fullsized":"2024-09-19T06:50:52.766777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:07:05.691663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332776+0000","last_change":"2024-09-19T06:49:10.375636+0000","last_active":"2024-09-19T06:50:52.332776+0000","last_peered":"2024-09-19T06:50:52.332776+0000","last_clean":"2024-09-19T06:50:52.332776+0000","last_became_active":"2024-09-19T06:49:10.375227+0000","last_became_peered":"2024-09-19T06:49:10.375227+0000","last_unstale":"2024-09-19T06:50:52.332776+0000","last_undegraded":"2024-09-19T06:50:52.332776+0000","last_fullsized":"2024-09-19T06:50:52.332776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:41:59.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217317+0000","last_change":"2024-09-19T06:49:10.383453+0000","last_active":"2024-09-19T06:50:52.217317+0000","last_peered":"2024-09-19T06:50:52.217317+0000","last_clean":"2024-09-19T06:50:52.217317+0000","last_became_active":"2024-09-19T06:49:10.383075+0000","last_became_peered":"2024-09-19T06:49:10.383075+0000","last_unstale":"2024-09-19T06:50:52.217317+0000","last_undegraded":"2024-09-19T06:50:52.217317+0000","last_fullsized":"2024-09-19T06:50:52.217317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:18:24.775578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222949+0000","last_change":"2024-09-19T06:49:10.378733+0000","last_active":"2024-09-19T06:50:52.222949+0000","last_peered":"2024-09-19T06:50:52.222949+0000","last_clean":"2024-09-19T06:50:52.222949+0000","last_became_active":"2024-09-19T06:49:10.378231+0000","last_became_peered":"2024-09-19T06:49:10.378231+0000","last_unstale":"2024-09-19T06:50:52.222949+0000","last_undegraded":"2024-09-19T06:50:52.222949+0000","last_fullsized":"2024-09-19T06:50:52.222949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:53:14.959853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267755+0000","last_change":"2024-09-19T06:49:10.379775+0000","last_active":"2024-09-19T06:50:52.267755+0000","last_peered":"2024-09-19T06:50:52.267755+0000","last_clean":"2024-09-19T06:50:52.267755+0000","last_became_active":"2024-09-19T06:49:10.379130+0000","last_became_peered":"2024-09-19T06:49:10.379130+0000","last_unstale":"2024-09-19T06:50:52.267755+0000","last_undegraded":"2024-09-19T06:50:52.267755+0000","last_fullsized":"2024-09-19T06:50:52.267755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:05.914354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.415263+0000","last_change":"2024-09-19T06:49:10.376239+0000","last_active":"2024-09-19T06:50:52.415263+0000","last_peered":"2024-09-19T06:50:52.415263+0000","last_clean":"2024-09-19T06:50:52.415263+0000","last_became_active":"2024-09-19T06:49:10.375913+0000","last_became_peered":"2024-09-19T06:49:10.375913+0000","last_unstale":"2024-09-19T06:50:52.415263+0000","last_undegraded":"2024-09-19T06:50:52.415263+0000","last_fullsized":"2024-09-19T06:50:52.415263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:14:26.373766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218467+0000","last_change":"2024-09-19T06:49:10.375479+0000","last_active":"2024-09-19T06:50:52.218467+0000","last_peered":"2024-09-19T06:50:52.218467+0000","last_clean":"2024-09-19T06:50:52.218467+0000","last_became_active":"2024-09-19T06:49:10.375201+0000","last_became_peered":"2024-09-19T06:49:10.375201+0000","last_unstale":"2024-09-19T06:50:52.218467+0000","last_undegraded":"2024-09-19T06:50:52.218467+0000","last_fullsized":"2024-09-19T06:50:52.218467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:44:56.421410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222690+0000","last_change":"2024-09-19T06:49:10.374495+0000","last_active":"2024-09-19T06:50:52.222690+0000","last_peered":"2024-09-19T06:50:52.222690+0000","last_clean":"2024-09-19T06:50:52.222690+0000","last_became_active":"2024-09-19T06:49:10.374110+0000","last_became_peered":"2024-09-19T06:49:10.374110+0000","last_unstale":"2024-09-19T06:50:52.222690+0000","last_undegraded":"2024-09-19T06:50:52.222690+0000","last_fullsized":"2024-09-19T06:50:52.222690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:51:27.641785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267168+0000","last_change":"2024-09-19T06:49:10.373548+0000","last_active":"2024-09-19T06:50:52.267168+0000","last_peered":"2024-09-19T06:50:52.267168+0000","last_clean":"2024-09-19T06:50:52.267168+0000","last_became_active":"2024-09-19T06:49:10.373251+0000","last_became_peered":"2024-09-19T06:49:10.373251+0000","last_unstale":"2024-09-19T06:50:52.267168+0000","last_undegraded":"2024-09-19T06:50:52.267168+0000","last_fullsized":"2024-09-19T06:50:52.267168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:59:35.353223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215665+0000","last_change":"2024-09-19T06:49:10.383000+0000","last_active":"2024-09-19T06:50:52.215665+0000","last_peered":"2024-09-19T06:50:52.215665+0000","last_clean":"2024-09-19T06:50:52.215665+0000","last_became_active":"2024-09-19T06:49:10.382653+0000","last_became_peered":"2024-09-19T06:49:10.382653+0000","last_unstale":"2024-09-19T06:50:52.215665+0000","last_undegraded":"2024-09-19T06:50:52.215665+0000","last_fullsized":"2024-09-19T06:50:52.215665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:31.317042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217451+0000","last_change":"2024-09-19T06:49:10.380239+0000","last_active":"2024-09-19T06:50:52.217451+0000","last_peered":"2024-09-19T06:50:52.217451+0000","last_clean":"2024-09-19T06:50:52.217451+0000","last_became_active":"2024-09-19T06:49:10.378491+0000","last_became_peered":"2024-09-19T06:49:10.378491+0000","last_unstale":"2024-09-19T06:50:52.217451+0000","last_undegraded":"2024-09-19T06:50:52.217451+0000","last_fullsized":"2024-09-19T06:50:52.217451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:56:14.016962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767120+0000","last_change":"2024-09-19T06:49:10.376989+0000","last_active":"2024-09-19T06:50:52.767120+0000","last_peered":"2024-09-19T06:50:52.767120+0000","last_clean":"2024-09-19T06:50:52.767120+0000","last_became_active":"2024-09-19T06:49:10.376701+0000","last_became_peered":"2024-09-19T06:49:10.376701+0000","last_unstale":"2024-09-19T06:50:52.767120+0000","last_undegraded":"2024-09-19T06:50:52.767120+0000","last_fullsized":"2024-09-19T06:50:52.767120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:13:34.153487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767231+0000","last_change":"2024-09-19T06:49:10.382585+0000","last_active":"2024-09-19T06:50:52.767231+0000","last_peered":"2024-09-19T06:50:52.767231+0000","last_clean":"2024-09-19T06:50:52.767231+0000","last_became_active":"2024-09-19T06:49:10.381573+0000","last_became_peered":"2024-09-19T06:49:10.381573+0000","last_unstale":"2024-09-19T06:50:52.767231+0000","last_undegraded":"2024-09-19T06:50:52.767231+0000","last_fullsized":"2024-09-19T06:50:52.767231+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:29:08.324309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220839+0000","last_change":"2024-09-19T06:49:10.381681+0000","last_active":"2024-09-19T06:50:52.220839+0000","last_peered":"2024-09-19T06:50:52.220839+0000","last_clean":"2024-09-19T06:50:52.220839+0000","last_became_active":"2024-09-19T06:49:10.381344+0000","last_became_peered":"2024-09-19T06:49:10.381344+0000","last_unstale":"2024-09-19T06:50:52.220839+0000","last_undegraded":"2024-09-19T06:50:52.220839+0000","last_fullsized":"2024-09-19T06:50:52.220839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:39:01.539679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69476,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":603,"num_read_kb":375,"num_write":248,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":208428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215885,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449123,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1064960,"data_stored":541446,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":71,"seq":304942678055,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1204224,"data_stored":1191114,"data_compressed":65674,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841582,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1081344,"data_stored":545270,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1597,"internal_metadata":31914435},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103219,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1064960,"data_stored":541440,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332152,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1126400,"data_stored":1138051,"data_compressed":65674,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1052672,"data_stored":536678,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1130496,"data_stored":1133150,"data_compressed":65674,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":541879,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1064960,"data_stored":536386,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1126400,"data_stored":599855,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1122304,"data_stored":600369,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1077248,"data_stored":538134,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34605,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T06:51:00.757 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-19T06:51:00.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:00 smithi118 ceph-mon[35527]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s; 11 B/s, 0 objects/s recovering 2024-09-19T06:51:01.121 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:01.566 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:01 smithi114 ceph-mon[30820]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:51:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 -- 172.21.15.114:0/3808588576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494109120 msgr2=0x7f9494110ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 --2- 172.21.15.114:0/3808588576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494109120 0x7f9494110ca0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f948c009ca0 tx=0x7f948c02a8c0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 -- 172.21.15.114:0/3808588576 shutdown_connections 2024-09-19T06:51:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 --2- 172.21.15.114:0/3808588576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494109120 0x7f9494110ca0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 --2- 172.21.15.114:0/3808588576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94941049a0 0x7f94941089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 --2- 172.21.15.114:0/3808588576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9494103ff0 0x7f94941043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 -- 172.21.15.114:0/3808588576 >> 172.21.15.114:0/3808588576 conn(0x7f94940ffa90 msgr2=0x7f9494101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 -- 172.21.15.114:0/3808588576 shutdown_connections 2024-09-19T06:51:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.674+0000 7f949b400640 1 -- 172.21.15.114:0/3808588576 wait complete. 2024-09-19T06:51:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.675+0000 7f949b400640 1 Processor -- start 2024-09-19T06:51:01.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.675+0000 7f949b400640 1 -- start start 2024-09-19T06:51:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f949b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494103ff0 0x7f949419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f949b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94941049a0 0x7f949419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f949b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9494109120 0x7f949419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f949b400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9494114150 con 0x7f9494103ff0 2024-09-19T06:51:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f949b400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9494113fd0 con 0x7f9494109120 2024-09-19T06:51:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f949b400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f94941142d0 con 0x7f94941049a0 2024-09-19T06:51:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9498a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494103ff0 0x7f949419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:01.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9498a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494103ff0 0x7f949419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50286/0 (socket says 172.21.15.114:50286) 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9498a00640 1 -- 172.21.15.114:0/20484702 learned_addr learned my addr 172.21.15.114:0/20484702 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9493e00640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94941049a0 0x7f949419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9499400640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9494109120 0x7f949419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9498a00640 1 -- 172.21.15.114:0/20484702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94941049a0 msgr2=0x7f949419ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9498a00640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94941049a0 0x7f949419ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9498a00640 1 -- 172.21.15.114:0/20484702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9494109120 msgr2=0x7f949419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9498a00640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9494109120 0x7f949419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.676+0000 7f9498a00640 1 -- 172.21.15.114:0/20484702 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949419f5e0 con 0x7f9494103ff0 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.677+0000 7f9498a00640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494103ff0 0x7f949419a630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f948800e9f0 tx=0x7f948800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.677+0000 7f9491600640 1 -- 172.21.15.114:0/20484702 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948800cd80 con 0x7f9494103ff0 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.677+0000 7f9491600640 1 -- 172.21.15.114:0/20484702 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9488005500 con 0x7f9494103ff0 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.677+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949419f8d0 con 0x7f9494103ff0 2024-09-19T06:51:01.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.677+0000 7f9491600640 1 -- 172.21.15.114:0/20484702 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488010690 con 0x7f9494103ff0 2024-09-19T06:51:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.677+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94941a71b0 con 0x7f9494103ff0 2024-09-19T06:51:01.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:01 smithi116 ceph-mon[35411]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:51:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.678+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94941043d0 con 0x7f9494103ff0 2024-09-19T06:51:01.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.680+0000 7f9491600640 1 -- 172.21.15.114:0/20484702 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f94880046c0 con 0x7f9494103ff0 2024-09-19T06:51:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.682+0000 7f9491600640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f947c0a3350 0x7f947c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.682+0000 7f9491600640 1 -- 172.21.15.114:0/20484702 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f94880d0270 con 0x7f9494103ff0 2024-09-19T06:51:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.682+0000 7f9493e00640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f947c0a3350 0x7f947c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:01.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.683+0000 7f9493e00640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f947c0a3350 0x7f947c0a5810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f94800095f0 tx=0x7f9480005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:01.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.686+0000 7f9491600640 1 -- 172.21.15.114:0/20484702 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f948809c9e0 con 0x7f9494103ff0 2024-09-19T06:51:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:01 smithi118 ceph-mon[35527]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:51:01.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.856+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9494100cc0 con 0x7f947c0a3350 2024-09-19T06:51:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.865+0000 7f9491600640 1 -- 172.21.15.114:0/20484702 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340646 (secure 0 0 0) 0x7f9494100cc0 con 0x7f947c0a3350 2024-09-19T06:51:01.869 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:01.871 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-09-19T06:51:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f947c0a3350 msgr2=0x7f947c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f947c0a3350 0x7f947c0a5810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f94800095f0 tx=0x7f9480005f90 comp rx=0 tx=0).stop 2024-09-19T06:51:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494103ff0 msgr2=0x7f949419a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494103ff0 0x7f949419a630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f948800e9f0 tx=0x7f948800edf0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 shutdown_connections 2024-09-19T06:51:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f947c0a3350 0x7f947c0a5810 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9494109120 0x7f949419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94941049a0 0x7f949419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 --2- 172.21.15.114:0/20484702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9494103ff0 0x7f949419a630 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 >> 172.21.15.114:0/20484702 conn(0x7f94940ffa90 msgr2=0x7f94941005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 shutdown_connections 2024-09-19T06:51:01.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:01.870+0000 7f949b400640 1 -- 172.21.15.114:0/20484702 wait complete. 2024-09-19T06:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:02 smithi114 ceph-mon[30820]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-09-19T06:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:02 smithi116 ceph-mon[35411]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-09-19T06:51:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:02 smithi118 ceph-mon[35527]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-09-19T06:51:02.818 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-19T06:51:01.112568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660078,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":768,"num_read_kb":508,"num_write":431,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180826,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":352,"ondisk_log_size":352,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":345564,"kb_used_data":12884,"kb_used_omap":18,"kb_used_meta":332397,"kb_avail":1124596260,"statfs":{"total":1151940427776,"available":1151586570240,"internally_reserved":0,"allocated":13193216,"data_stored":8443772,"data_compressed":461784,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19139,"internal_metadata":340374845},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":47,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.904974"},"pg_stats":[{"pgid":"2.1f","version":"93'13","reported_seq":115,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:58.338557+0000","last_change":"2024-09-19T06:49:07.909205+0000","last_active":"2024-09-19T06:50:58.338557+0000","last_peered":"2024-09-19T06:50:58.338557+0000","last_clean":"2024-09-19T06:50:58.338557+0000","last_became_active":"2024-09-19T06:49:07.907471+0000","last_became_peered":"2024-09-19T06:49:07.907471+0000","last_unstale":"2024-09-19T06:50:58.338557+0000","last_undegraded":"2024-09-19T06:50:58.338557+0000","last_fullsized":"2024-09-19T06:50:58.338557+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:55:00.655927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330051+0000","last_change":"2024-09-19T06:49:52.196586+0000","last_active":"2024-09-19T06:50:52.330051+0000","last_peered":"2024-09-19T06:50:52.330051+0000","last_clean":"2024-09-19T06:50:52.330051+0000","last_became_active":"2024-09-19T06:49:52.195680+0000","last_became_peered":"2024-09-19T06:49:52.195680+0000","last_unstale":"2024-09-19T06:50:52.330051+0000","last_undegraded":"2024-09-19T06:50:52.330051+0000","last_fullsized":"2024-09-19T06:50:52.330051+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:46:53.026767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330518+0000","last_change":"2024-09-19T06:49:20.424335+0000","last_active":"2024-09-19T06:50:52.330518+0000","last_peered":"2024-09-19T06:50:52.330518+0000","last_clean":"2024-09-19T06:50:52.330518+0000","last_became_active":"2024-09-19T06:49:20.423890+0000","last_became_peered":"2024-09-19T06:49:20.423890+0000","last_unstale":"2024-09-19T06:50:52.330518+0000","last_undegraded":"2024-09-19T06:50:52.330518+0000","last_fullsized":"2024-09-19T06:50:52.330518+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:30:49.042220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"93'2","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218579+0000","last_change":"2024-09-19T06:49:08.185286+0000","last_active":"2024-09-19T06:50:52.218579+0000","last_peered":"2024-09-19T06:50:52.218579+0000","last_clean":"2024-09-19T06:50:52.218579+0000","last_became_active":"2024-09-19T06:49:08.184971+0000","last_became_peered":"2024-09-19T06:49:08.184971+0000","last_unstale":"2024-09-19T06:50:52.218579+0000","last_undegraded":"2024-09-19T06:50:52.218579+0000","last_fullsized":"2024-09-19T06:50:52.218579+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:56:26.532594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767343+0000","last_change":"2024-09-19T06:49:10.385824+0000","last_active":"2024-09-19T06:50:52.767343+0000","last_peered":"2024-09-19T06:50:52.767343+0000","last_clean":"2024-09-19T06:50:52.767343+0000","last_became_active":"2024-09-19T06:49:10.381448+0000","last_became_peered":"2024-09-19T06:49:10.381448+0000","last_unstale":"2024-09-19T06:50:52.767343+0000","last_undegraded":"2024-09-19T06:50:52.767343+0000","last_fullsized":"2024-09-19T06:50:52.767343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:43:39.192054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.334354+0000","last_change":"2024-09-19T06:50:52.334354+0000","last_active":"2024-09-19T06:50:52.334354+0000","last_peered":"2024-09-19T06:50:52.334354+0000","last_clean":"2024-09-19T06:50:52.334354+0000","last_became_active":"2024-09-19T06:50:52.333866+0000","last_became_peered":"2024-09-19T06:50:52.333866+0000","last_unstale":"2024-09-19T06:50:52.334354+0000","last_undegraded":"2024-09-19T06:50:52.334354+0000","last_fullsized":"2024-09-19T06:50:52.334354+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:27:08.379318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,10,3],"acting":[4,6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"93'12","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216299+0000","last_change":"2024-09-19T06:49:07.907830+0000","last_active":"2024-09-19T06:50:52.216299+0000","last_peered":"2024-09-19T06:50:52.216299+0000","last_clean":"2024-09-19T06:50:52.216299+0000","last_became_active":"2024-09-19T06:49:07.907510+0000","last_became_peered":"2024-09-19T06:49:07.907510+0000","last_unstale":"2024-09-19T06:50:52.216299+0000","last_undegraded":"2024-09-19T06:50:52.216299+0000","last_fullsized":"2024-09-19T06:50:52.216299+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:52:46.192671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414918+0000","last_change":"2024-09-19T06:49:10.371663+0000","last_active":"2024-09-19T06:50:52.414918+0000","last_peered":"2024-09-19T06:50:52.414918+0000","last_clean":"2024-09-19T06:50:52.414918+0000","last_became_active":"2024-09-19T06:49:10.371325+0000","last_became_peered":"2024-09-19T06:49:10.371325+0000","last_unstale":"2024-09-19T06:50:52.414918+0000","last_undegraded":"2024-09-19T06:50:52.414918+0000","last_fullsized":"2024-09-19T06:50:52.414918+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:48:58.406933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217200+0000","last_change":"2024-09-19T06:49:20.429446+0000","last_active":"2024-09-19T06:50:52.217200+0000","last_peered":"2024-09-19T06:50:52.217200+0000","last_clean":"2024-09-19T06:50:52.217200+0000","last_became_active":"2024-09-19T06:49:20.428996+0000","last_became_peered":"2024-09-19T06:49:20.428996+0000","last_unstale":"2024-09-19T06:50:52.217200+0000","last_undegraded":"2024-09-19T06:50:52.217200+0000","last_fullsized":"2024-09-19T06:50:52.217200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:29:39.542347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"93'5","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216067+0000","last_change":"2024-09-19T06:49:07.906940+0000","last_active":"2024-09-19T06:50:52.216067+0000","last_peered":"2024-09-19T06:50:52.216067+0000","last_clean":"2024-09-19T06:50:52.216067+0000","last_became_active":"2024-09-19T06:49:07.906635+0000","last_became_peered":"2024-09-19T06:49:07.906635+0000","last_unstale":"2024-09-19T06:50:52.216067+0000","last_undegraded":"2024-09-19T06:50:52.216067+0000","last_fullsized":"2024-09-19T06:50:52.216067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:22:00.794049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223602+0000","last_change":"2024-09-19T06:49:52.411553+0000","last_active":"2024-09-19T06:50:52.223602+0000","last_peered":"2024-09-19T06:50:52.223602+0000","last_clean":"2024-09-19T06:50:52.223602+0000","last_became_active":"2024-09-19T06:49:52.411251+0000","last_became_peered":"2024-09-19T06:49:52.411251+0000","last_unstale":"2024-09-19T06:50:52.223602+0000","last_undegraded":"2024-09-19T06:50:52.223602+0000","last_fullsized":"2024-09-19T06:50:52.223602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:40:17.774698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216749+0000","last_change":"2024-09-19T06:49:20.427188+0000","last_active":"2024-09-19T06:50:52.216749+0000","last_peered":"2024-09-19T06:50:52.216749+0000","last_clean":"2024-09-19T06:50:52.216749+0000","last_became_active":"2024-09-19T06:49:20.426716+0000","last_became_peered":"2024-09-19T06:49:20.426716+0000","last_unstale":"2024-09-19T06:50:52.216749+0000","last_undegraded":"2024-09-19T06:50:52.216749+0000","last_fullsized":"2024-09-19T06:50:52.216749+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:25:09.086438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222719+0000","last_change":"2024-09-19T06:49:10.371433+0000","last_active":"2024-09-19T06:50:52.222719+0000","last_peered":"2024-09-19T06:50:52.222719+0000","last_clean":"2024-09-19T06:50:52.222719+0000","last_became_active":"2024-09-19T06:49:10.370947+0000","last_became_peered":"2024-09-19T06:49:10.370947+0000","last_unstale":"2024-09-19T06:50:52.222719+0000","last_undegraded":"2024-09-19T06:50:52.222719+0000","last_fullsized":"2024-09-19T06:50:52.222719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:56:40.954670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"93'4","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215809+0000","last_change":"2024-09-19T06:49:07.907972+0000","last_active":"2024-09-19T06:50:52.215809+0000","last_peered":"2024-09-19T06:50:52.215809+0000","last_clean":"2024-09-19T06:50:52.215809+0000","last_became_active":"2024-09-19T06:49:07.907631+0000","last_became_peered":"2024-09-19T06:49:07.907631+0000","last_unstale":"2024-09-19T06:50:52.215809+0000","last_undegraded":"2024-09-19T06:50:52.215809+0000","last_fullsized":"2024-09-19T06:50:52.215809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:12:16.298696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217079+0000","last_change":"2024-09-19T06:49:20.617446+0000","last_active":"2024-09-19T06:50:52.217079+0000","last_peered":"2024-09-19T06:50:52.217079+0000","last_clean":"2024-09-19T06:50:52.217079+0000","last_became_active":"2024-09-19T06:49:20.616950+0000","last_became_peered":"2024-09-19T06:49:20.616950+0000","last_unstale":"2024-09-19T06:50:52.217079+0000","last_undegraded":"2024-09-19T06:50:52.217079+0000","last_fullsized":"2024-09-19T06:50:52.217079+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:37:00.640364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.329931+0000","last_change":"2024-09-19T06:49:52.195937+0000","last_active":"2024-09-19T06:50:52.329931+0000","last_peered":"2024-09-19T06:50:52.329931+0000","last_clean":"2024-09-19T06:50:52.329931+0000","last_became_active":"2024-09-19T06:49:52.195615+0000","last_became_peered":"2024-09-19T06:49:52.195615+0000","last_unstale":"2024-09-19T06:50:52.329931+0000","last_undegraded":"2024-09-19T06:50:52.329931+0000","last_fullsized":"2024-09-19T06:50:52.329931+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:28:33.296849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.a","version":"93'3","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330730+0000","last_change":"2024-09-19T06:49:07.906081+0000","last_active":"2024-09-19T06:50:52.330730+0000","last_peered":"2024-09-19T06:50:52.330730+0000","last_clean":"2024-09-19T06:50:52.330730+0000","last_became_active":"2024-09-19T06:49:07.905559+0000","last_became_peered":"2024-09-19T06:49:07.905559+0000","last_unstale":"2024-09-19T06:50:52.330730+0000","last_undegraded":"2024-09-19T06:50:52.330730+0000","last_fullsized":"2024-09-19T06:50:52.330730+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:30:56.922084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"93'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217028+0000","last_change":"2024-09-19T06:49:20.431255+0000","last_active":"2024-09-19T06:50:52.217028+0000","last_peered":"2024-09-19T06:50:52.217028+0000","last_clean":"2024-09-19T06:50:52.217028+0000","last_became_active":"2024-09-19T06:49:20.430727+0000","last_became_peered":"2024-09-19T06:49:20.430727+0000","last_unstale":"2024-09-19T06:50:52.217028+0000","last_undegraded":"2024-09-19T06:50:52.217028+0000","last_fullsized":"2024-09-19T06:50:52.217028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:23:01.761121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220297+0000","last_change":"2024-09-19T06:49:10.382739+0000","last_active":"2024-09-19T06:50:52.220297+0000","last_peered":"2024-09-19T06:50:52.220297+0000","last_clean":"2024-09-19T06:50:52.220297+0000","last_became_active":"2024-09-19T06:49:10.382216+0000","last_became_peered":"2024-09-19T06:49:10.382216+0000","last_unstale":"2024-09-19T06:50:52.220297+0000","last_undegraded":"2024-09-19T06:50:52.220297+0000","last_fullsized":"2024-09-19T06:50:52.220297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:31:37.908345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"93'7","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218241+0000","last_change":"2024-09-19T06:49:07.906876+0000","last_active":"2024-09-19T06:50:52.218241+0000","last_peered":"2024-09-19T06:50:52.218241+0000","last_clean":"2024-09-19T06:50:52.218241+0000","last_became_active":"2024-09-19T06:49:07.906567+0000","last_became_peered":"2024-09-19T06:49:07.906567+0000","last_unstale":"2024-09-19T06:50:52.218241+0000","last_undegraded":"2024-09-19T06:50:52.218241+0000","last_fullsized":"2024-09-19T06:50:52.218241+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:21:39.900536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332912+0000","last_change":"2024-09-19T06:50:52.332343+0000","last_active":"2024-09-19T06:50:52.332912+0000","last_peered":"2024-09-19T06:50:52.332912+0000","last_clean":"2024-09-19T06:50:52.332912+0000","last_became_active":"2024-09-19T06:50:52.331869+0000","last_became_peered":"2024-09-19T06:50:52.331869+0000","last_unstale":"2024-09-19T06:50:52.332912+0000","last_undegraded":"2024-09-19T06:50:52.332912+0000","last_fullsized":"2024-09-19T06:50:52.332912+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:56:59.286185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215623+0000","last_change":"2024-09-19T06:49:10.369037+0000","last_active":"2024-09-19T06:50:52.215623+0000","last_peered":"2024-09-19T06:50:52.215623+0000","last_clean":"2024-09-19T06:50:52.215623+0000","last_became_active":"2024-09-19T06:49:10.368653+0000","last_became_peered":"2024-09-19T06:49:10.368653+0000","last_unstale":"2024-09-19T06:50:52.215623+0000","last_undegraded":"2024-09-19T06:50:52.215623+0000","last_fullsized":"2024-09-19T06:50:52.215623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:46:02.268843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"93'2","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414742+0000","last_change":"2024-09-19T06:49:08.185517+0000","last_active":"2024-09-19T06:50:52.414742+0000","last_peered":"2024-09-19T06:50:52.414742+0000","last_clean":"2024-09-19T06:50:52.414742+0000","last_became_active":"2024-09-19T06:49:08.185138+0000","last_became_peered":"2024-09-19T06:49:08.185138+0000","last_unstale":"2024-09-19T06:50:52.414742+0000","last_undegraded":"2024-09-19T06:50:52.414742+0000","last_fullsized":"2024-09-19T06:50:52.414742+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:57:59.082821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217098+0000","last_change":"2024-09-19T06:49:20.617423+0000","last_active":"2024-09-19T06:50:52.217098+0000","last_peered":"2024-09-19T06:50:52.217098+0000","last_clean":"2024-09-19T06:50:52.217098+0000","last_became_active":"2024-09-19T06:49:20.616834+0000","last_became_peered":"2024-09-19T06:49:20.616834+0000","last_unstale":"2024-09-19T06:50:52.217098+0000","last_undegraded":"2024-09-19T06:50:52.217098+0000","last_fullsized":"2024-09-19T06:50:52.217098+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:53:47.384169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767310+0000","last_change":"2024-09-19T06:49:10.388955+0000","last_active":"2024-09-19T06:50:52.767310+0000","last_peered":"2024-09-19T06:50:52.767310+0000","last_clean":"2024-09-19T06:50:52.767310+0000","last_became_active":"2024-09-19T06:49:10.387947+0000","last_became_peered":"2024-09-19T06:49:10.387947+0000","last_unstale":"2024-09-19T06:50:52.767310+0000","last_undegraded":"2024-09-19T06:50:52.767310+0000","last_fullsized":"2024-09-19T06:50:52.767310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:27:44.556097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"93'7","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217734+0000","last_change":"2024-09-19T06:49:07.912897+0000","last_active":"2024-09-19T06:50:52.217734+0000","last_peered":"2024-09-19T06:50:52.217734+0000","last_clean":"2024-09-19T06:50:52.217734+0000","last_became_active":"2024-09-19T06:49:07.912520+0000","last_became_peered":"2024-09-19T06:49:07.912520+0000","last_unstale":"2024-09-19T06:50:52.217734+0000","last_undegraded":"2024-09-19T06:50:52.217734+0000","last_fullsized":"2024-09-19T06:50:52.217734+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:23:56.582808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330848+0000","last_change":"2024-09-19T06:49:20.420851+0000","last_active":"2024-09-19T06:50:52.330848+0000","last_peered":"2024-09-19T06:50:52.330848+0000","last_clean":"2024-09-19T06:50:52.330848+0000","last_became_active":"2024-09-19T06:49:20.420307+0000","last_became_peered":"2024-09-19T06:49:20.420307+0000","last_unstale":"2024-09-19T06:50:52.330848+0000","last_undegraded":"2024-09-19T06:50:52.330848+0000","last_fullsized":"2024-09-19T06:50:52.330848+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:20.633070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218467+0000","last_change":"2024-09-19T06:49:10.372935+0000","last_active":"2024-09-19T06:50:52.218467+0000","last_peered":"2024-09-19T06:50:52.218467+0000","last_clean":"2024-09-19T06:50:52.218467+0000","last_became_active":"2024-09-19T06:49:10.372641+0000","last_became_peered":"2024-09-19T06:49:10.372641+0000","last_unstale":"2024-09-19T06:50:52.218467+0000","last_undegraded":"2024-09-19T06:50:52.218467+0000","last_fullsized":"2024-09-19T06:50:52.218467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:13:07.917094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'16","reported_seq":56,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330389+0000","last_change":"2024-09-19T06:49:07.908184+0000","last_active":"2024-09-19T06:50:52.330389+0000","last_peered":"2024-09-19T06:50:52.330389+0000","last_clean":"2024-09-19T06:50:52.330389+0000","last_became_active":"2024-09-19T06:49:07.907722+0000","last_became_peered":"2024-09-19T06:49:07.907722+0000","last_unstale":"2024-09-19T06:50:52.330389+0000","last_undegraded":"2024-09-19T06:50:52.330389+0000","last_fullsized":"2024-09-19T06:50:52.330389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:24:25.957501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218316+0000","last_change":"2024-09-19T06:49:20.616966+0000","last_active":"2024-09-19T06:50:52.218316+0000","last_peered":"2024-09-19T06:50:52.218316+0000","last_clean":"2024-09-19T06:50:52.218316+0000","last_became_active":"2024-09-19T06:49:20.616402+0000","last_became_peered":"2024-09-19T06:49:20.616402+0000","last_unstale":"2024-09-19T06:50:52.218316+0000","last_undegraded":"2024-09-19T06:50:52.218316+0000","last_fullsized":"2024-09-19T06:50:52.218316+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:15:12.572214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.333101+0000","last_change":"2024-09-19T06:49:10.373048+0000","last_active":"2024-09-19T06:50:52.333101+0000","last_peered":"2024-09-19T06:50:52.333101+0000","last_clean":"2024-09-19T06:50:52.333101+0000","last_became_active":"2024-09-19T06:49:10.372257+0000","last_became_peered":"2024-09-19T06:49:10.372257+0000","last_unstale":"2024-09-19T06:50:52.333101+0000","last_undegraded":"2024-09-19T06:50:52.333101+0000","last_fullsized":"2024-09-19T06:50:52.333101+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:30.707948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"93'4","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220484+0000","last_change":"2024-09-19T06:49:52.195314+0000","last_active":"2024-09-19T06:50:52.220484+0000","last_peered":"2024-09-19T06:50:52.220484+0000","last_clean":"2024-09-19T06:50:52.220484+0000","last_became_active":"2024-09-19T06:49:52.194849+0000","last_became_peered":"2024-09-19T06:49:52.194849+0000","last_unstale":"2024-09-19T06:50:52.220484+0000","last_undegraded":"2024-09-19T06:50:52.220484+0000","last_fullsized":"2024-09-19T06:50:52.220484+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:01:20.902382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220312+0000","last_change":"2024-09-19T06:49:20.422658+0000","last_active":"2024-09-19T06:50:52.220312+0000","last_peered":"2024-09-19T06:50:52.220312+0000","last_clean":"2024-09-19T06:50:52.220312+0000","last_became_active":"2024-09-19T06:49:20.422190+0000","last_became_peered":"2024-09-19T06:49:20.422190+0000","last_unstale":"2024-09-19T06:50:52.220312+0000","last_undegraded":"2024-09-19T06:50:52.220312+0000","last_fullsized":"2024-09-19T06:50:52.220312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:13:14.338388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267449+0000","last_change":"2024-09-19T06:49:10.376579+0000","last_active":"2024-09-19T06:50:52.267449+0000","last_peered":"2024-09-19T06:50:52.267449+0000","last_clean":"2024-09-19T06:50:52.267449+0000","last_became_active":"2024-09-19T06:49:10.375920+0000","last_became_peered":"2024-09-19T06:49:10.375920+0000","last_unstale":"2024-09-19T06:50:52.267449+0000","last_undegraded":"2024-09-19T06:50:52.267449+0000","last_fullsized":"2024-09-19T06:50:52.267449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:47:35.976117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"93'1","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330330+0000","last_change":"2024-09-19T06:49:08.187353+0000","last_active":"2024-09-19T06:50:52.330330+0000","last_peered":"2024-09-19T06:50:52.330330+0000","last_clean":"2024-09-19T06:50:52.330330+0000","last_became_active":"2024-09-19T06:49:08.186860+0000","last_became_peered":"2024-09-19T06:49:08.186860+0000","last_unstale":"2024-09-19T06:50:52.330330+0000","last_undegraded":"2024-09-19T06:50:52.330330+0000","last_fullsized":"2024-09-19T06:50:52.330330+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:45:06.236143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215603+0000","last_change":"2024-09-19T06:49:20.617028+0000","last_active":"2024-09-19T06:50:52.215603+0000","last_peered":"2024-09-19T06:50:52.215603+0000","last_clean":"2024-09-19T06:50:52.215603+0000","last_became_active":"2024-09-19T06:49:20.616443+0000","last_became_peered":"2024-09-19T06:49:20.616443+0000","last_unstale":"2024-09-19T06:50:52.215603+0000","last_undegraded":"2024-09-19T06:50:52.215603+0000","last_fullsized":"2024-09-19T06:50:52.215603+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:46:30.394051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267323+0000","last_change":"2024-09-19T06:49:10.386030+0000","last_active":"2024-09-19T06:50:52.267323+0000","last_peered":"2024-09-19T06:50:52.267323+0000","last_clean":"2024-09-19T06:50:52.267323+0000","last_became_active":"2024-09-19T06:49:10.385698+0000","last_became_peered":"2024-09-19T06:49:10.385698+0000","last_unstale":"2024-09-19T06:50:52.267323+0000","last_undegraded":"2024-09-19T06:50:52.267323+0000","last_fullsized":"2024-09-19T06:50:52.267323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:56:12.688507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"93'5","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:57.337913+0000","last_change":"2024-09-19T06:49:07.910172+0000","last_active":"2024-09-19T06:50:57.337913+0000","last_peered":"2024-09-19T06:50:57.337913+0000","last_clean":"2024-09-19T06:50:57.337913+0000","last_became_active":"2024-09-19T06:49:07.909798+0000","last_became_peered":"2024-09-19T06:49:07.909798+0000","last_unstale":"2024-09-19T06:50:57.337913+0000","last_undegraded":"2024-09-19T06:50:57.337913+0000","last_fullsized":"2024-09-19T06:50:57.337913+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:08:09.592988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332850+0000","last_change":"2024-09-19T06:49:20.424233+0000","last_active":"2024-09-19T06:50:52.332850+0000","last_peered":"2024-09-19T06:50:52.332850+0000","last_clean":"2024-09-19T06:50:52.332850+0000","last_became_active":"2024-09-19T06:49:20.423797+0000","last_became_peered":"2024-09-19T06:49:20.423797+0000","last_unstale":"2024-09-19T06:50:52.332850+0000","last_undegraded":"2024-09-19T06:50:52.332850+0000","last_fullsized":"2024-09-19T06:50:52.332850+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:58:54.983250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222640+0000","last_change":"2024-09-19T06:49:10.376408+0000","last_active":"2024-09-19T06:50:52.222640+0000","last_peered":"2024-09-19T06:50:52.222640+0000","last_clean":"2024-09-19T06:50:52.222640+0000","last_became_active":"2024-09-19T06:49:10.376065+0000","last_became_peered":"2024-09-19T06:49:10.376065+0000","last_unstale":"2024-09-19T06:50:52.222640+0000","last_undegraded":"2024-09-19T06:50:52.222640+0000","last_fullsized":"2024-09-19T06:50:52.222640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:28:06.175670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":107,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:58.340757+0000","last_change":"2024-09-19T06:49:08.185183+0000","last_active":"2024-09-19T06:50:58.340757+0000","last_peered":"2024-09-19T06:50:58.340757+0000","last_clean":"2024-09-19T06:50:58.340757+0000","last_became_active":"2024-09-19T06:49:08.184738+0000","last_became_peered":"2024-09-19T06:49:08.184738+0000","last_unstale":"2024-09-19T06:50:58.340757+0000","last_undegraded":"2024-09-19T06:50:58.340757+0000","last_fullsized":"2024-09-19T06:50:58.340757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:19:54.966966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":62,"num_read_kb":62,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.333375+0000","last_change":"2024-09-19T06:49:20.617986+0000","last_active":"2024-09-19T06:50:52.333375+0000","last_peered":"2024-09-19T06:50:52.333375+0000","last_clean":"2024-09-19T06:50:52.333375+0000","last_became_active":"2024-09-19T06:49:20.617325+0000","last_became_peered":"2024-09-19T06:49:20.617325+0000","last_unstale":"2024-09-19T06:50:52.333375+0000","last_undegraded":"2024-09-19T06:50:52.333375+0000","last_fullsized":"2024-09-19T06:50:52.333375+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:03.825005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332191+0000","last_change":"2024-09-19T06:49:10.385230+0000","last_active":"2024-09-19T06:50:52.332191+0000","last_peered":"2024-09-19T06:50:52.332191+0000","last_clean":"2024-09-19T06:50:52.332191+0000","last_became_active":"2024-09-19T06:49:10.384633+0000","last_became_peered":"2024-09-19T06:49:10.384633+0000","last_unstale":"2024-09-19T06:50:52.332191+0000","last_undegraded":"2024-09-19T06:50:52.332191+0000","last_fullsized":"2024-09-19T06:50:52.332191+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:16:27.751338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"93'4","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222673+0000","last_change":"2024-09-19T06:49:07.908823+0000","last_active":"2024-09-19T06:50:52.222673+0000","last_peered":"2024-09-19T06:50:52.222673+0000","last_clean":"2024-09-19T06:50:52.222673+0000","last_became_active":"2024-09-19T06:49:07.908340+0000","last_became_peered":"2024-09-19T06:49:07.908340+0000","last_unstale":"2024-09-19T06:50:52.222673+0000","last_undegraded":"2024-09-19T06:50:52.222673+0000","last_fullsized":"2024-09-19T06:50:52.222673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:25.198073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.334852+0000","last_change":"2024-09-19T06:50:52.334222+0000","last_active":"2024-09-19T06:50:52.334852+0000","last_peered":"2024-09-19T06:50:52.334852+0000","last_clean":"2024-09-19T06:50:52.334852+0000","last_became_active":"2024-09-19T06:50:52.333646+0000","last_became_peered":"2024-09-19T06:50:52.333646+0000","last_unstale":"2024-09-19T06:50:52.334852+0000","last_undegraded":"2024-09-19T06:50:52.334852+0000","last_fullsized":"2024-09-19T06:50:52.334852+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:21.741223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218343+0000","last_change":"2024-09-19T06:49:10.374761+0000","last_active":"2024-09-19T06:50:52.218343+0000","last_peered":"2024-09-19T06:50:52.218343+0000","last_clean":"2024-09-19T06:50:52.218343+0000","last_became_active":"2024-09-19T06:49:10.374258+0000","last_became_peered":"2024-09-19T06:49:10.374258+0000","last_unstale":"2024-09-19T06:50:52.218343+0000","last_undegraded":"2024-09-19T06:50:52.218343+0000","last_fullsized":"2024-09-19T06:50:52.218343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:23:23.850658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"93'6","reported_seq":49,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217433+0000","last_change":"2024-09-19T06:49:07.912648+0000","last_active":"2024-09-19T06:50:52.217433+0000","last_peered":"2024-09-19T06:50:52.217433+0000","last_clean":"2024-09-19T06:50:52.217433+0000","last_became_active":"2024-09-19T06:49:07.912289+0000","last_became_peered":"2024-09-19T06:49:07.912289+0000","last_unstale":"2024-09-19T06:50:52.217433+0000","last_undegraded":"2024-09-19T06:50:52.217433+0000","last_fullsized":"2024-09-19T06:50:52.217433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:23:36.211030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222917+0000","last_change":"2024-09-19T06:47:53.158465+0000","last_active":"2024-09-19T06:50:52.222917+0000","last_peered":"2024-09-19T06:50:52.222917+0000","last_clean":"2024-09-19T06:50:52.222917+0000","last_became_active":"2024-09-19T06:47:53.158059+0000","last_became_peered":"2024-09-19T06:47:53.158059+0000","last_unstale":"2024-09-19T06:50:52.222917+0000","last_undegraded":"2024-09-19T06:50:52.222917+0000","last_fullsized":"2024-09-19T06:50:52.222917+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:44:13.613197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:44:13.613197+0000","last_clean_scrub_stamp":"2024-09-19T06:44:13.613197+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:16:41.310229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"93'1","reported_seq":26,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222892+0000","last_change":"2024-09-19T06:49:20.422112+0000","last_active":"2024-09-19T06:50:52.222892+0000","last_peered":"2024-09-19T06:50:52.222892+0000","last_clean":"2024-09-19T06:50:52.222892+0000","last_became_active":"2024-09-19T06:49:20.421720+0000","last_became_peered":"2024-09-19T06:49:20.421720+0000","last_unstale":"2024-09-19T06:50:52.222892+0000","last_undegraded":"2024-09-19T06:50:52.222892+0000","last_fullsized":"2024-09-19T06:50:52.222892+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:35:25.923946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223044+0000","last_change":"2024-09-19T06:49:10.379414+0000","last_active":"2024-09-19T06:50:52.223044+0000","last_peered":"2024-09-19T06:50:52.223044+0000","last_clean":"2024-09-19T06:50:52.223044+0000","last_became_active":"2024-09-19T06:49:10.379075+0000","last_became_peered":"2024-09-19T06:49:10.379075+0000","last_unstale":"2024-09-19T06:50:52.223044+0000","last_undegraded":"2024-09-19T06:50:52.223044+0000","last_fullsized":"2024-09-19T06:50:52.223044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:32:57.976929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"93'5","reported_seq":105,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:56.335865+0000","last_change":"2024-09-19T06:49:08.186883+0000","last_active":"2024-09-19T06:50:56.335865+0000","last_peered":"2024-09-19T06:50:56.335865+0000","last_clean":"2024-09-19T06:50:56.335865+0000","last_became_active":"2024-09-19T06:49:08.186472+0000","last_became_peered":"2024-09-19T06:49:08.186472+0000","last_unstale":"2024-09-19T06:50:56.335865+0000","last_undegraded":"2024-09-19T06:50:56.335865+0000","last_fullsized":"2024-09-19T06:50:56.335865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:58:12.146032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414698+0000","last_change":"2024-09-19T06:49:20.418542+0000","last_active":"2024-09-19T06:50:52.414698+0000","last_peered":"2024-09-19T06:50:52.414698+0000","last_clean":"2024-09-19T06:50:52.414698+0000","last_became_active":"2024-09-19T06:49:20.418044+0000","last_became_peered":"2024-09-19T06:49:20.418044+0000","last_unstale":"2024-09-19T06:50:52.414698+0000","last_undegraded":"2024-09-19T06:50:52.414698+0000","last_fullsized":"2024-09-19T06:50:52.414698+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:44:02.695919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223157+0000","last_change":"2024-09-19T06:49:10.373862+0000","last_active":"2024-09-19T06:50:52.223157+0000","last_peered":"2024-09-19T06:50:52.223157+0000","last_clean":"2024-09-19T06:50:52.223157+0000","last_became_active":"2024-09-19T06:49:10.373345+0000","last_became_peered":"2024-09-19T06:49:10.373345+0000","last_unstale":"2024-09-19T06:50:52.223157+0000","last_undegraded":"2024-09-19T06:50:52.223157+0000","last_fullsized":"2024-09-19T06:50:52.223157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:43:33.562984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"93'3","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223768+0000","last_change":"2024-09-19T06:49:07.909407+0000","last_active":"2024-09-19T06:50:52.223768+0000","last_peered":"2024-09-19T06:50:52.223768+0000","last_clean":"2024-09-19T06:50:52.223768+0000","last_became_active":"2024-09-19T06:49:07.909101+0000","last_became_peered":"2024-09-19T06:49:07.909101+0000","last_unstale":"2024-09-19T06:50:52.223768+0000","last_undegraded":"2024-09-19T06:50:52.223768+0000","last_fullsized":"2024-09-19T06:50:52.223768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:08:47.317018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.766985+0000","last_change":"2024-09-19T06:49:20.425109+0000","last_active":"2024-09-19T06:50:52.766985+0000","last_peered":"2024-09-19T06:50:52.766985+0000","last_clean":"2024-09-19T06:50:52.766985+0000","last_became_active":"2024-09-19T06:49:20.424701+0000","last_became_peered":"2024-09-19T06:49:20.424701+0000","last_unstale":"2024-09-19T06:50:52.766985+0000","last_undegraded":"2024-09-19T06:50:52.766985+0000","last_fullsized":"2024-09-19T06:50:52.766985+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:13:10.878691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223092+0000","last_change":"2024-09-19T06:49:10.374089+0000","last_active":"2024-09-19T06:50:52.223092+0000","last_peered":"2024-09-19T06:50:52.223092+0000","last_clean":"2024-09-19T06:50:52.223092+0000","last_became_active":"2024-09-19T06:49:10.373729+0000","last_became_peered":"2024-09-19T06:49:10.373729+0000","last_unstale":"2024-09-19T06:50:52.223092+0000","last_undegraded":"2024-09-19T06:50:52.223092+0000","last_fullsized":"2024-09-19T06:50:52.223092+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:07:52.876874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"93'5","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.333165+0000","last_change":"2024-09-19T06:49:07.909616+0000","last_active":"2024-09-19T06:50:52.333165+0000","last_peered":"2024-09-19T06:50:52.333165+0000","last_clean":"2024-09-19T06:50:52.333165+0000","last_became_active":"2024-09-19T06:49:07.909172+0000","last_became_peered":"2024-09-19T06:49:07.909172+0000","last_unstale":"2024-09-19T06:50:52.333165+0000","last_undegraded":"2024-09-19T06:50:52.333165+0000","last_fullsized":"2024-09-19T06:50:52.333165+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:57:27.685639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222460+0000","last_change":"2024-09-19T06:49:20.419188+0000","last_active":"2024-09-19T06:50:52.222460+0000","last_peered":"2024-09-19T06:50:52.222460+0000","last_clean":"2024-09-19T06:50:52.222460+0000","last_became_active":"2024-09-19T06:49:20.418641+0000","last_became_peered":"2024-09-19T06:49:20.418641+0000","last_unstale":"2024-09-19T06:50:52.222460+0000","last_undegraded":"2024-09-19T06:50:52.222460+0000","last_fullsized":"2024-09-19T06:50:52.222460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:06:05.278027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220813+0000","last_change":"2024-09-19T06:49:10.378491+0000","last_active":"2024-09-19T06:50:52.220813+0000","last_peered":"2024-09-19T06:50:52.220813+0000","last_clean":"2024-09-19T06:50:52.220813+0000","last_became_active":"2024-09-19T06:49:10.378141+0000","last_became_peered":"2024-09-19T06:49:10.378141+0000","last_unstale":"2024-09-19T06:50:52.220813+0000","last_undegraded":"2024-09-19T06:50:52.220813+0000","last_fullsized":"2024-09-19T06:50:52.220813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:57:39.195604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"93'2","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216307+0000","last_change":"2024-09-19T06:49:07.909836+0000","last_active":"2024-09-19T06:50:52.216307+0000","last_peered":"2024-09-19T06:50:52.216307+0000","last_clean":"2024-09-19T06:50:52.216307+0000","last_became_active":"2024-09-19T06:49:07.909527+0000","last_became_peered":"2024-09-19T06:49:07.909527+0000","last_unstale":"2024-09-19T06:50:52.216307+0000","last_undegraded":"2024-09-19T06:50:52.216307+0000","last_fullsized":"2024-09-19T06:50:52.216307+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:06:51.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222641+0000","last_change":"2024-09-19T06:49:20.426184+0000","last_active":"2024-09-19T06:50:52.222641+0000","last_peered":"2024-09-19T06:50:52.222641+0000","last_clean":"2024-09-19T06:50:52.222641+0000","last_became_active":"2024-09-19T06:49:20.425844+0000","last_became_peered":"2024-09-19T06:49:20.425844+0000","last_unstale":"2024-09-19T06:50:52.222641+0000","last_undegraded":"2024-09-19T06:50:52.222641+0000","last_fullsized":"2024-09-19T06:50:52.222641+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:52:29.120462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222862+0000","last_change":"2024-09-19T06:49:10.380589+0000","last_active":"2024-09-19T06:50:52.222862+0000","last_peered":"2024-09-19T06:50:52.222862+0000","last_clean":"2024-09-19T06:50:52.222862+0000","last_became_active":"2024-09-19T06:49:10.380263+0000","last_became_peered":"2024-09-19T06:49:10.380263+0000","last_unstale":"2024-09-19T06:50:52.222862+0000","last_undegraded":"2024-09-19T06:50:52.222862+0000","last_fullsized":"2024-09-19T06:50:52.222862+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:30:48.426240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"93'17","reported_seq":74,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215601+0000","last_change":"2024-09-19T06:49:52.199971+0000","last_active":"2024-09-19T06:50:52.215601+0000","last_peered":"2024-09-19T06:50:52.215601+0000","last_clean":"2024-09-19T06:50:52.215601+0000","last_became_active":"2024-09-19T06:49:52.191887+0000","last_became_peered":"2024-09-19T06:49:52.191887+0000","last_unstale":"2024-09-19T06:50:52.215601+0000","last_undegraded":"2024-09-19T06:50:52.215601+0000","last_fullsized":"2024-09-19T06:50:52.215601+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:22:40.775893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.333002+0000","last_change":"2024-09-19T06:49:20.435213+0000","last_active":"2024-09-19T06:50:52.333002+0000","last_peered":"2024-09-19T06:50:52.333002+0000","last_clean":"2024-09-19T06:50:52.333002+0000","last_became_active":"2024-09-19T06:49:20.434776+0000","last_became_peered":"2024-09-19T06:49:20.434776+0000","last_unstale":"2024-09-19T06:50:52.333002+0000","last_undegraded":"2024-09-19T06:50:52.333002+0000","last_fullsized":"2024-09-19T06:50:52.333002+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:30:37.067947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222464+0000","last_change":"2024-09-19T06:49:10.375394+0000","last_active":"2024-09-19T06:50:52.222464+0000","last_peered":"2024-09-19T06:50:52.222464+0000","last_clean":"2024-09-19T06:50:52.222464+0000","last_became_active":"2024-09-19T06:49:10.375056+0000","last_became_peered":"2024-09-19T06:49:10.375056+0000","last_unstale":"2024-09-19T06:50:52.222464+0000","last_undegraded":"2024-09-19T06:50:52.222464+0000","last_fullsized":"2024-09-19T06:50:52.222464+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:18:40.158481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"93'6","reported_seq":108,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:56.338792+0000","last_change":"2024-09-19T06:49:07.910215+0000","last_active":"2024-09-19T06:50:56.338792+0000","last_peered":"2024-09-19T06:50:56.338792+0000","last_clean":"2024-09-19T06:50:56.338792+0000","last_became_active":"2024-09-19T06:49:07.909908+0000","last_became_peered":"2024-09-19T06:49:07.909908+0000","last_unstale":"2024-09-19T06:50:56.338792+0000","last_undegraded":"2024-09-19T06:50:56.338792+0000","last_fullsized":"2024-09-19T06:50:56.338792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:45:05.432960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":63,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330305+0000","last_change":"2024-09-19T06:49:20.434400+0000","last_active":"2024-09-19T06:50:52.330305+0000","last_peered":"2024-09-19T06:50:52.330305+0000","last_clean":"2024-09-19T06:50:52.330305+0000","last_became_active":"2024-09-19T06:49:20.433877+0000","last_became_peered":"2024-09-19T06:49:20.433877+0000","last_unstale":"2024-09-19T06:50:52.330305+0000","last_undegraded":"2024-09-19T06:50:52.330305+0000","last_fullsized":"2024-09-19T06:50:52.330305+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:24:16.378509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222695+0000","last_change":"2024-09-19T06:49:10.378048+0000","last_active":"2024-09-19T06:50:52.222695+0000","last_peered":"2024-09-19T06:50:52.222695+0000","last_clean":"2024-09-19T06:50:52.222695+0000","last_became_active":"2024-09-19T06:49:10.377710+0000","last_became_peered":"2024-09-19T06:49:10.377710+0000","last_unstale":"2024-09-19T06:50:52.222695+0000","last_undegraded":"2024-09-19T06:50:52.222695+0000","last_fullsized":"2024-09-19T06:50:52.222695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:41:54.620218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"93'5","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.229301+0000","last_change":"2024-09-19T06:50:52.229301+0000","last_active":"2024-09-19T06:50:52.229301+0000","last_peered":"2024-09-19T06:50:52.229301+0000","last_clean":"2024-09-19T06:50:52.229301+0000","last_became_active":"2024-09-19T06:50:52.223798+0000","last_became_peered":"2024-09-19T06:50:52.223798+0000","last_unstale":"2024-09-19T06:50:52.229301+0000","last_undegraded":"2024-09-19T06:50:52.229301+0000","last_fullsized":"2024-09-19T06:50:52.229301+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:57:42.199295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.334454+0000","last_change":"2024-09-19T06:50:52.334454+0000","last_active":"2024-09-19T06:50:52.334454+0000","last_peered":"2024-09-19T06:50:52.334454+0000","last_clean":"2024-09-19T06:50:52.334454+0000","last_became_active":"2024-09-19T06:50:52.333886+0000","last_became_peered":"2024-09-19T06:50:52.333886+0000","last_unstale":"2024-09-19T06:50:52.334454+0000","last_undegraded":"2024-09-19T06:50:52.334454+0000","last_fullsized":"2024-09-19T06:50:52.334454+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:41:08.771168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332737+0000","last_change":"2024-09-19T06:49:10.382223+0000","last_active":"2024-09-19T06:50:52.332737+0000","last_peered":"2024-09-19T06:50:52.332737+0000","last_clean":"2024-09-19T06:50:52.332737+0000","last_became_active":"2024-09-19T06:49:10.381869+0000","last_became_peered":"2024-09-19T06:49:10.381869+0000","last_unstale":"2024-09-19T06:50:52.332737+0000","last_undegraded":"2024-09-19T06:50:52.332737+0000","last_fullsized":"2024-09-19T06:50:52.332737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:23:34.975802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"93'9","reported_seq":50,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330978+0000","last_change":"2024-09-19T06:49:07.907812+0000","last_active":"2024-09-19T06:50:52.330978+0000","last_peered":"2024-09-19T06:50:52.330978+0000","last_clean":"2024-09-19T06:50:52.330978+0000","last_became_active":"2024-09-19T06:49:07.907524+0000","last_became_peered":"2024-09-19T06:49:07.907524+0000","last_unstale":"2024-09-19T06:50:52.330978+0000","last_undegraded":"2024-09-19T06:50:52.330978+0000","last_fullsized":"2024-09-19T06:50:52.330978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:26:43.780682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223572+0000","last_change":"2024-09-19T06:49:20.436311+0000","last_active":"2024-09-19T06:50:52.223572+0000","last_peered":"2024-09-19T06:50:52.223572+0000","last_clean":"2024-09-19T06:50:52.223572+0000","last_became_active":"2024-09-19T06:49:20.435936+0000","last_became_peered":"2024-09-19T06:49:20.435936+0000","last_unstale":"2024-09-19T06:50:52.223572+0000","last_undegraded":"2024-09-19T06:50:52.223572+0000","last_fullsized":"2024-09-19T06:50:52.223572+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:54:04.688922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.216102+0000","last_change":"2024-09-19T06:49:10.373764+0000","last_active":"2024-09-19T06:50:52.216102+0000","last_peered":"2024-09-19T06:50:52.216102+0000","last_clean":"2024-09-19T06:50:52.216102+0000","last_became_active":"2024-09-19T06:49:10.373324+0000","last_became_peered":"2024-09-19T06:49:10.373324+0000","last_unstale":"2024-09-19T06:50:52.216102+0000","last_undegraded":"2024-09-19T06:50:52.216102+0000","last_fullsized":"2024-09-19T06:50:52.216102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:25:03.898831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"93'34","reported_seq":203,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:59.338170+0000","last_change":"2024-09-19T06:49:07.907319+0000","last_active":"2024-09-19T06:50:59.338170+0000","last_peered":"2024-09-19T06:50:59.338170+0000","last_clean":"2024-09-19T06:50:59.338170+0000","last_became_active":"2024-09-19T06:49:07.906888+0000","last_became_peered":"2024-09-19T06:49:07.906888+0000","last_unstale":"2024-09-19T06:50:59.338170+0000","last_undegraded":"2024-09-19T06:50:59.338170+0000","last_fullsized":"2024-09-19T06:50:59.338170+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:39:49.051957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62661,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":130,"num_read_kb":126,"num_write":36,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.329885+0000","last_change":"2024-09-19T06:49:20.428184+0000","last_active":"2024-09-19T06:50:52.329885+0000","last_peered":"2024-09-19T06:50:52.329885+0000","last_clean":"2024-09-19T06:50:52.329885+0000","last_became_active":"2024-09-19T06:49:20.427631+0000","last_became_peered":"2024-09-19T06:49:20.427631+0000","last_unstale":"2024-09-19T06:50:52.329885+0000","last_undegraded":"2024-09-19T06:50:52.329885+0000","last_fullsized":"2024-09-19T06:50:52.329885+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:14:51.063413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414971+0000","last_change":"2024-09-19T06:49:10.373073+0000","last_active":"2024-09-19T06:50:52.414971+0000","last_peered":"2024-09-19T06:50:52.414971+0000","last_clean":"2024-09-19T06:50:52.414971+0000","last_became_active":"2024-09-19T06:49:10.371611+0000","last_became_peered":"2024-09-19T06:49:10.371611+0000","last_unstale":"2024-09-19T06:50:52.414971+0000","last_undegraded":"2024-09-19T06:50:52.414971+0000","last_fullsized":"2024-09-19T06:50:52.414971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:58.228946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"93'5","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223763+0000","last_change":"2024-09-19T06:49:08.186128+0000","last_active":"2024-09-19T06:50:52.223763+0000","last_peered":"2024-09-19T06:50:52.223763+0000","last_clean":"2024-09-19T06:50:52.223763+0000","last_became_active":"2024-09-19T06:49:08.185817+0000","last_became_peered":"2024-09-19T06:49:08.185817+0000","last_unstale":"2024-09-19T06:50:52.223763+0000","last_undegraded":"2024-09-19T06:50:52.223763+0000","last_fullsized":"2024-09-19T06:50:52.223763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:59:53.066405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217185+0000","last_change":"2024-09-19T06:49:20.431251+0000","last_active":"2024-09-19T06:50:52.217185+0000","last_peered":"2024-09-19T06:50:52.217185+0000","last_clean":"2024-09-19T06:50:52.217185+0000","last_became_active":"2024-09-19T06:49:20.430709+0000","last_became_peered":"2024-09-19T06:49:20.430709+0000","last_unstale":"2024-09-19T06:50:52.217185+0000","last_undegraded":"2024-09-19T06:50:52.217185+0000","last_fullsized":"2024-09-19T06:50:52.217185+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:55:06.742567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767345+0000","last_change":"2024-09-19T06:49:10.379045+0000","last_active":"2024-09-19T06:50:52.767345+0000","last_peered":"2024-09-19T06:50:52.767345+0000","last_clean":"2024-09-19T06:50:52.767345+0000","last_became_active":"2024-09-19T06:49:10.378481+0000","last_became_peered":"2024-09-19T06:49:10.378481+0000","last_unstale":"2024-09-19T06:50:52.767345+0000","last_undegraded":"2024-09-19T06:50:52.767345+0000","last_fullsized":"2024-09-19T06:50:52.767345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:29:55.770670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"93'4","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223019+0000","last_change":"2024-09-19T06:49:07.910246+0000","last_active":"2024-09-19T06:50:52.223019+0000","last_peered":"2024-09-19T06:50:52.223019+0000","last_clean":"2024-09-19T06:50:52.223019+0000","last_became_active":"2024-09-19T06:49:07.909855+0000","last_became_peered":"2024-09-19T06:49:07.909855+0000","last_unstale":"2024-09-19T06:50:52.223019+0000","last_undegraded":"2024-09-19T06:50:52.223019+0000","last_fullsized":"2024-09-19T06:50:52.223019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:33:24.011418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.329642+0000","last_change":"2024-09-19T06:49:20.421847+0000","last_active":"2024-09-19T06:50:52.329642+0000","last_peered":"2024-09-19T06:50:52.329642+0000","last_clean":"2024-09-19T06:50:52.329642+0000","last_became_active":"2024-09-19T06:49:20.421391+0000","last_became_peered":"2024-09-19T06:49:20.421391+0000","last_unstale":"2024-09-19T06:50:52.329642+0000","last_undegraded":"2024-09-19T06:50:52.329642+0000","last_fullsized":"2024-09-19T06:50:52.329642+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:54:24.207872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.415192+0000","last_change":"2024-09-19T06:49:10.376029+0000","last_active":"2024-09-19T06:50:52.415192+0000","last_peered":"2024-09-19T06:50:52.415192+0000","last_clean":"2024-09-19T06:50:52.415192+0000","last_became_active":"2024-09-19T06:49:10.374242+0000","last_became_peered":"2024-09-19T06:49:10.374242+0000","last_unstale":"2024-09-19T06:50:52.415192+0000","last_undegraded":"2024-09-19T06:50:52.415192+0000","last_fullsized":"2024-09-19T06:50:52.415192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:59:43.722173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":165,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:59.338029+0000","last_change":"2024-09-19T06:49:07.908090+0000","last_active":"2024-09-19T06:50:59.338029+0000","last_peered":"2024-09-19T06:50:59.338029+0000","last_clean":"2024-09-19T06:50:59.338029+0000","last_became_active":"2024-09-19T06:49:07.906933+0000","last_became_peered":"2024-09-19T06:49:07.906933+0000","last_unstale":"2024-09-19T06:50:59.338029+0000","last_undegraded":"2024-09-19T06:50:59.338029+0000","last_fullsized":"2024-09-19T06:50:59.338029+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:59:21.711371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218024+0000","last_change":"2024-09-19T06:49:20.433797+0000","last_active":"2024-09-19T06:50:52.218024+0000","last_peered":"2024-09-19T06:50:52.218024+0000","last_clean":"2024-09-19T06:50:52.218024+0000","last_became_active":"2024-09-19T06:49:20.433419+0000","last_became_peered":"2024-09-19T06:49:20.433419+0000","last_unstale":"2024-09-19T06:50:52.218024+0000","last_undegraded":"2024-09-19T06:50:52.218024+0000","last_fullsized":"2024-09-19T06:50:52.218024+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:18:39.691678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767189+0000","last_change":"2024-09-19T06:49:20.424995+0000","last_active":"2024-09-19T06:50:52.767189+0000","last_peered":"2024-09-19T06:50:52.767189+0000","last_clean":"2024-09-19T06:50:52.767189+0000","last_became_active":"2024-09-19T06:49:20.424589+0000","last_became_peered":"2024-09-19T06:49:20.424589+0000","last_unstale":"2024-09-19T06:50:52.767189+0000","last_undegraded":"2024-09-19T06:50:52.767189+0000","last_fullsized":"2024-09-19T06:50:52.767189+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:17.968571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.329723+0000","last_change":"2024-09-19T06:49:10.382157+0000","last_active":"2024-09-19T06:50:52.329723+0000","last_peered":"2024-09-19T06:50:52.329723+0000","last_clean":"2024-09-19T06:50:52.329723+0000","last_became_active":"2024-09-19T06:49:10.381848+0000","last_became_peered":"2024-09-19T06:49:10.381848+0000","last_unstale":"2024-09-19T06:50:52.329723+0000","last_undegraded":"2024-09-19T06:50:52.329723+0000","last_fullsized":"2024-09-19T06:50:52.329723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:28:10.738364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"93'3","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222841+0000","last_change":"2024-09-19T06:49:07.906088+0000","last_active":"2024-09-19T06:50:52.222841+0000","last_peered":"2024-09-19T06:50:52.222841+0000","last_clean":"2024-09-19T06:50:52.222841+0000","last_became_active":"2024-09-19T06:49:07.905798+0000","last_became_peered":"2024-09-19T06:49:07.905798+0000","last_unstale":"2024-09-19T06:50:52.222841+0000","last_undegraded":"2024-09-19T06:50:52.222841+0000","last_fullsized":"2024-09-19T06:50:52.222841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:37:02.185689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220464+0000","last_change":"2024-09-19T06:49:20.432275+0000","last_active":"2024-09-19T06:50:52.220464+0000","last_peered":"2024-09-19T06:50:52.220464+0000","last_clean":"2024-09-19T06:50:52.220464+0000","last_became_active":"2024-09-19T06:49:20.431951+0000","last_became_peered":"2024-09-19T06:49:20.431951+0000","last_unstale":"2024-09-19T06:50:52.220464+0000","last_undegraded":"2024-09-19T06:50:52.220464+0000","last_fullsized":"2024-09-19T06:50:52.220464+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:21:38.649557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218570+0000","last_change":"2024-09-19T06:49:10.368061+0000","last_active":"2024-09-19T06:50:52.218570+0000","last_peered":"2024-09-19T06:50:52.218570+0000","last_clean":"2024-09-19T06:50:52.218570+0000","last_became_active":"2024-09-19T06:49:10.367732+0000","last_became_peered":"2024-09-19T06:49:10.367732+0000","last_unstale":"2024-09-19T06:50:52.218570+0000","last_undegraded":"2024-09-19T06:50:52.218570+0000","last_fullsized":"2024-09-19T06:50:52.218570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:41:37.623628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"93'3","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332201+0000","last_change":"2024-09-19T06:49:07.910722+0000","last_active":"2024-09-19T06:50:52.332201+0000","last_peered":"2024-09-19T06:50:52.332201+0000","last_clean":"2024-09-19T06:50:52.332201+0000","last_became_active":"2024-09-19T06:49:07.909960+0000","last_became_peered":"2024-09-19T06:49:07.909960+0000","last_unstale":"2024-09-19T06:50:52.332201+0000","last_undegraded":"2024-09-19T06:50:52.332201+0000","last_fullsized":"2024-09-19T06:50:52.332201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:40:18.366567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"93'7","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267267+0000","last_change":"2024-09-19T06:49:52.411869+0000","last_active":"2024-09-19T06:50:52.267267+0000","last_peered":"2024-09-19T06:50:52.267267+0000","last_clean":"2024-09-19T06:50:52.267267+0000","last_became_active":"2024-09-19T06:49:52.411491+0000","last_became_peered":"2024-09-19T06:49:52.411491+0000","last_unstale":"2024-09-19T06:50:52.267267+0000","last_undegraded":"2024-09-19T06:50:52.267267+0000","last_fullsized":"2024-09-19T06:50:52.267267+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:05:21.222348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.223274+0000","last_change":"2024-09-19T06:49:10.388831+0000","last_active":"2024-09-19T06:50:52.223274+0000","last_peered":"2024-09-19T06:50:52.223274+0000","last_clean":"2024-09-19T06:50:52.223274+0000","last_became_active":"2024-09-19T06:49:10.388556+0000","last_became_peered":"2024-09-19T06:49:10.388556+0000","last_unstale":"2024-09-19T06:50:52.223274+0000","last_undegraded":"2024-09-19T06:50:52.223274+0000","last_fullsized":"2024-09-19T06:50:52.223274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:28:20.319429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330796+0000","last_change":"2024-09-19T06:49:20.424827+0000","last_active":"2024-09-19T06:50:52.330796+0000","last_peered":"2024-09-19T06:50:52.330796+0000","last_clean":"2024-09-19T06:50:52.330796+0000","last_became_active":"2024-09-19T06:49:20.424318+0000","last_became_peered":"2024-09-19T06:49:20.424318+0000","last_unstale":"2024-09-19T06:50:52.330796+0000","last_undegraded":"2024-09-19T06:50:52.330796+0000","last_fullsized":"2024-09-19T06:50:52.330796+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:36:37.814108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'5","reported_seq":113,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:58.337452+0000","last_change":"2024-09-19T06:49:52.188404+0000","last_active":"2024-09-19T06:50:58.337452+0000","last_peered":"2024-09-19T06:50:58.337452+0000","last_clean":"2024-09-19T06:50:58.337452+0000","last_became_active":"2024-09-19T06:49:52.187918+0000","last_became_peered":"2024-09-19T06:49:52.187918+0000","last_unstale":"2024-09-19T06:50:58.337452+0000","last_undegraded":"2024-09-19T06:50:58.337452+0000","last_fullsized":"2024-09-19T06:50:58.337452+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:06.878742+0000","last_clean_scrub_stamp":"2024-09-19T06:49:06.878742+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T06:53:38.209357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215711+0000","last_change":"2024-09-19T06:49:10.374333+0000","last_active":"2024-09-19T06:50:52.215711+0000","last_peered":"2024-09-19T06:50:52.215711+0000","last_clean":"2024-09-19T06:50:52.215711+0000","last_became_active":"2024-09-19T06:49:10.373944+0000","last_became_peered":"2024-09-19T06:49:10.373944+0000","last_unstale":"2024-09-19T06:50:52.215711+0000","last_undegraded":"2024-09-19T06:50:52.215711+0000","last_fullsized":"2024-09-19T06:50:52.215711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:01:43.609310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.331039+0000","last_change":"2024-09-19T06:49:20.417030+0000","last_active":"2024-09-19T06:50:52.331039+0000","last_peered":"2024-09-19T06:50:52.331039+0000","last_clean":"2024-09-19T06:50:52.331039+0000","last_became_active":"2024-09-19T06:49:20.416620+0000","last_became_peered":"2024-09-19T06:49:20.416620+0000","last_unstale":"2024-09-19T06:50:52.331039+0000","last_undegraded":"2024-09-19T06:50:52.331039+0000","last_fullsized":"2024-09-19T06:50:52.331039+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:19.399125+0000","last_clean_scrub_stamp":"2024-09-19T06:49:19.399125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:47:18.177613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220233+0000","last_change":"2024-09-19T06:49:10.372503+0000","last_active":"2024-09-19T06:50:52.220233+0000","last_peered":"2024-09-19T06:50:52.220233+0000","last_clean":"2024-09-19T06:50:52.220233+0000","last_became_active":"2024-09-19T06:49:10.372201+0000","last_became_peered":"2024-09-19T06:49:10.372201+0000","last_unstale":"2024-09-19T06:50:52.220233+0000","last_undegraded":"2024-09-19T06:50:52.220233+0000","last_fullsized":"2024-09-19T06:50:52.220233+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:31:44.270457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.331025+0000","last_change":"2024-09-19T06:49:10.374149+0000","last_active":"2024-09-19T06:50:52.331025+0000","last_peered":"2024-09-19T06:50:52.331025+0000","last_clean":"2024-09-19T06:50:52.331025+0000","last_became_active":"2024-09-19T06:49:10.372570+0000","last_became_peered":"2024-09-19T06:49:10.372570+0000","last_unstale":"2024-09-19T06:50:52.331025+0000","last_undegraded":"2024-09-19T06:50:52.331025+0000","last_fullsized":"2024-09-19T06:50:52.331025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:55:56.086106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330600+0000","last_change":"2024-09-19T06:49:10.373383+0000","last_active":"2024-09-19T06:50:52.330600+0000","last_peered":"2024-09-19T06:50:52.330600+0000","last_clean":"2024-09-19T06:50:52.330600+0000","last_became_active":"2024-09-19T06:49:10.372144+0000","last_became_peered":"2024-09-19T06:49:10.372144+0000","last_unstale":"2024-09-19T06:50:52.330600+0000","last_undegraded":"2024-09-19T06:50:52.330600+0000","last_fullsized":"2024-09-19T06:50:52.330600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:08:34.000815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767048+0000","last_change":"2024-09-19T06:49:10.379720+0000","last_active":"2024-09-19T06:50:52.767048+0000","last_peered":"2024-09-19T06:50:52.767048+0000","last_clean":"2024-09-19T06:50:52.767048+0000","last_became_active":"2024-09-19T06:49:10.379325+0000","last_became_peered":"2024-09-19T06:49:10.379325+0000","last_unstale":"2024-09-19T06:50:52.767048+0000","last_undegraded":"2024-09-19T06:50:52.767048+0000","last_fullsized":"2024-09-19T06:50:52.767048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:17:58.471617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332999+0000","last_change":"2024-09-19T06:49:52.194997+0000","last_active":"2024-09-19T06:50:52.332999+0000","last_peered":"2024-09-19T06:50:52.332999+0000","last_clean":"2024-09-19T06:50:52.332999+0000","last_became_active":"2024-09-19T06:49:52.194662+0000","last_became_peered":"2024-09-19T06:49:52.194662+0000","last_unstale":"2024-09-19T06:50:52.332999+0000","last_undegraded":"2024-09-19T06:50:52.332999+0000","last_fullsized":"2024-09-19T06:50:52.332999+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:18.915611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220724+0000","last_change":"2024-09-19T06:49:10.375868+0000","last_active":"2024-09-19T06:50:52.220724+0000","last_peered":"2024-09-19T06:50:52.220724+0000","last_clean":"2024-09-19T06:50:52.220724+0000","last_became_active":"2024-09-19T06:49:10.375552+0000","last_became_peered":"2024-09-19T06:49:10.375552+0000","last_unstale":"2024-09-19T06:50:52.220724+0000","last_undegraded":"2024-09-19T06:50:52.220724+0000","last_fullsized":"2024-09-19T06:50:52.220724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:55:22.292330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330966+0000","last_change":"2024-09-19T06:49:10.365959+0000","last_active":"2024-09-19T06:50:52.330966+0000","last_peered":"2024-09-19T06:50:52.330966+0000","last_clean":"2024-09-19T06:50:52.330966+0000","last_became_active":"2024-09-19T06:49:10.365018+0000","last_became_peered":"2024-09-19T06:49:10.365018+0000","last_unstale":"2024-09-19T06:50:52.330966+0000","last_undegraded":"2024-09-19T06:50:52.330966+0000","last_fullsized":"2024-09-19T06:50:52.330966+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:45:50.417270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218366+0000","last_change":"2024-09-19T06:49:10.372988+0000","last_active":"2024-09-19T06:50:52.218366+0000","last_peered":"2024-09-19T06:50:52.218366+0000","last_clean":"2024-09-19T06:50:52.218366+0000","last_became_active":"2024-09-19T06:49:10.368995+0000","last_became_peered":"2024-09-19T06:49:10.368995+0000","last_unstale":"2024-09-19T06:50:52.218366+0000","last_undegraded":"2024-09-19T06:50:52.218366+0000","last_fullsized":"2024-09-19T06:50:52.218366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:29:04.939254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222301+0000","last_change":"2024-09-19T06:49:52.228705+0000","last_active":"2024-09-19T06:50:52.222301+0000","last_peered":"2024-09-19T06:50:52.222301+0000","last_clean":"2024-09-19T06:50:52.222301+0000","last_became_active":"2024-09-19T06:49:52.228371+0000","last_became_peered":"2024-09-19T06:49:52.228371+0000","last_unstale":"2024-09-19T06:50:52.222301+0000","last_undegraded":"2024-09-19T06:50:52.222301+0000","last_fullsized":"2024-09-19T06:50:52.222301+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:24:58.959892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218662+0000","last_change":"2024-09-19T06:49:10.372657+0000","last_active":"2024-09-19T06:50:52.218662+0000","last_peered":"2024-09-19T06:50:52.218662+0000","last_clean":"2024-09-19T06:50:52.218662+0000","last_became_active":"2024-09-19T06:49:10.371641+0000","last_became_peered":"2024-09-19T06:49:10.371641+0000","last_unstale":"2024-09-19T06:50:52.218662+0000","last_undegraded":"2024-09-19T06:50:52.218662+0000","last_fullsized":"2024-09-19T06:50:52.218662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:54:09.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414907+0000","last_change":"2024-09-19T06:49:10.370765+0000","last_active":"2024-09-19T06:50:52.414907+0000","last_peered":"2024-09-19T06:50:52.414907+0000","last_clean":"2024-09-19T06:50:52.414907+0000","last_became_active":"2024-09-19T06:49:10.369741+0000","last_became_peered":"2024-09-19T06:49:10.369741+0000","last_unstale":"2024-09-19T06:50:52.414907+0000","last_undegraded":"2024-09-19T06:50:52.414907+0000","last_fullsized":"2024-09-19T06:50:52.414907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:35:31.377034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332767+0000","last_change":"2024-09-19T06:49:10.386328+0000","last_active":"2024-09-19T06:50:52.332767+0000","last_peered":"2024-09-19T06:50:52.332767+0000","last_clean":"2024-09-19T06:50:52.332767+0000","last_became_active":"2024-09-19T06:49:10.385970+0000","last_became_peered":"2024-09-19T06:49:10.385970+0000","last_unstale":"2024-09-19T06:50:52.332767+0000","last_undegraded":"2024-09-19T06:50:52.332767+0000","last_fullsized":"2024-09-19T06:50:52.332767+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T07:52:27.404825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220439+0000","last_change":"2024-09-19T06:49:10.372047+0000","last_active":"2024-09-19T06:50:52.220439+0000","last_peered":"2024-09-19T06:50:52.220439+0000","last_clean":"2024-09-19T06:50:52.220439+0000","last_became_active":"2024-09-19T06:49:10.371743+0000","last_became_peered":"2024-09-19T06:49:10.371743+0000","last_unstale":"2024-09-19T06:50:52.220439+0000","last_undegraded":"2024-09-19T06:50:52.220439+0000","last_fullsized":"2024-09-19T06:50:52.220439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T13:27:06.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.331192+0000","last_change":"2024-09-19T06:49:10.373221+0000","last_active":"2024-09-19T06:50:52.331192+0000","last_peered":"2024-09-19T06:50:52.331192+0000","last_clean":"2024-09-19T06:50:52.331192+0000","last_became_active":"2024-09-19T06:49:10.372859+0000","last_became_peered":"2024-09-19T06:49:10.372859+0000","last_unstale":"2024-09-19T06:50:52.331192+0000","last_undegraded":"2024-09-19T06:50:52.331192+0000","last_fullsized":"2024-09-19T06:50:52.331192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:53:19.333758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.330296+0000","last_change":"2024-09-19T06:49:10.371359+0000","last_active":"2024-09-19T06:50:52.330296+0000","last_peered":"2024-09-19T06:50:52.330296+0000","last_clean":"2024-09-19T06:50:52.330296+0000","last_became_active":"2024-09-19T06:49:10.371057+0000","last_became_peered":"2024-09-19T06:49:10.371057+0000","last_unstale":"2024-09-19T06:50:52.330296+0000","last_undegraded":"2024-09-19T06:50:52.330296+0000","last_fullsized":"2024-09-19T06:50:52.330296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T16:18:40.387791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220873+0000","last_change":"2024-09-19T06:49:10.381371+0000","last_active":"2024-09-19T06:50:52.220873+0000","last_peered":"2024-09-19T06:50:52.220873+0000","last_clean":"2024-09-19T06:50:52.220873+0000","last_became_active":"2024-09-19T06:49:10.381017+0000","last_became_peered":"2024-09-19T06:49:10.381017+0000","last_unstale":"2024-09-19T06:50:52.220873+0000","last_undegraded":"2024-09-19T06:50:52.220873+0000","last_fullsized":"2024-09-19T06:50:52.220873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:17:54.998597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.414586+0000","last_change":"2024-09-19T06:49:10.377596+0000","last_active":"2024-09-19T06:50:52.414586+0000","last_peered":"2024-09-19T06:50:52.414586+0000","last_clean":"2024-09-19T06:50:52.414586+0000","last_became_active":"2024-09-19T06:49:10.377271+0000","last_became_peered":"2024-09-19T06:49:10.377271+0000","last_unstale":"2024-09-19T06:50:52.414586+0000","last_undegraded":"2024-09-19T06:50:52.414586+0000","last_fullsized":"2024-09-19T06:50:52.414586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:32:00.797866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.766828+0000","last_change":"2024-09-19T06:49:52.194675+0000","last_active":"2024-09-19T06:50:52.766828+0000","last_peered":"2024-09-19T06:50:52.766828+0000","last_clean":"2024-09-19T06:50:52.766828+0000","last_became_active":"2024-09-19T06:49:52.194007+0000","last_became_peered":"2024-09-19T06:49:52.194007+0000","last_unstale":"2024-09-19T06:50:52.766828+0000","last_undegraded":"2024-09-19T06:50:52.766828+0000","last_fullsized":"2024-09-19T06:50:52.766828+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:45:33.109454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.766777+0000","last_change":"2024-09-19T06:49:10.387416+0000","last_active":"2024-09-19T06:50:52.766777+0000","last_peered":"2024-09-19T06:50:52.766777+0000","last_clean":"2024-09-19T06:50:52.766777+0000","last_became_active":"2024-09-19T06:49:10.384083+0000","last_became_peered":"2024-09-19T06:49:10.384083+0000","last_unstale":"2024-09-19T06:50:52.766777+0000","last_undegraded":"2024-09-19T06:50:52.766777+0000","last_fullsized":"2024-09-19T06:50:52.766777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:07:05.691663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.332776+0000","last_change":"2024-09-19T06:49:10.375636+0000","last_active":"2024-09-19T06:50:52.332776+0000","last_peered":"2024-09-19T06:50:52.332776+0000","last_clean":"2024-09-19T06:50:52.332776+0000","last_became_active":"2024-09-19T06:49:10.375227+0000","last_became_peered":"2024-09-19T06:49:10.375227+0000","last_unstale":"2024-09-19T06:50:52.332776+0000","last_undegraded":"2024-09-19T06:50:52.332776+0000","last_fullsized":"2024-09-19T06:50:52.332776+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T11:41:59.589374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217317+0000","last_change":"2024-09-19T06:49:10.383453+0000","last_active":"2024-09-19T06:50:52.217317+0000","last_peered":"2024-09-19T06:50:52.217317+0000","last_clean":"2024-09-19T06:50:52.217317+0000","last_became_active":"2024-09-19T06:49:10.383075+0000","last_became_peered":"2024-09-19T06:49:10.383075+0000","last_unstale":"2024-09-19T06:50:52.217317+0000","last_undegraded":"2024-09-19T06:50:52.217317+0000","last_fullsized":"2024-09-19T06:50:52.217317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:18:24.775578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222949+0000","last_change":"2024-09-19T06:49:10.378733+0000","last_active":"2024-09-19T06:50:52.222949+0000","last_peered":"2024-09-19T06:50:52.222949+0000","last_clean":"2024-09-19T06:50:52.222949+0000","last_became_active":"2024-09-19T06:49:10.378231+0000","last_became_peered":"2024-09-19T06:49:10.378231+0000","last_unstale":"2024-09-19T06:50:52.222949+0000","last_undegraded":"2024-09-19T06:50:52.222949+0000","last_fullsized":"2024-09-19T06:50:52.222949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:53:14.959853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267755+0000","last_change":"2024-09-19T06:49:10.379775+0000","last_active":"2024-09-19T06:50:52.267755+0000","last_peered":"2024-09-19T06:50:52.267755+0000","last_clean":"2024-09-19T06:50:52.267755+0000","last_became_active":"2024-09-19T06:49:10.379130+0000","last_became_peered":"2024-09-19T06:49:10.379130+0000","last_unstale":"2024-09-19T06:50:52.267755+0000","last_undegraded":"2024-09-19T06:50:52.267755+0000","last_fullsized":"2024-09-19T06:50:52.267755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:54:05.914354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.415263+0000","last_change":"2024-09-19T06:49:10.376239+0000","last_active":"2024-09-19T06:50:52.415263+0000","last_peered":"2024-09-19T06:50:52.415263+0000","last_clean":"2024-09-19T06:50:52.415263+0000","last_became_active":"2024-09-19T06:49:10.375913+0000","last_became_peered":"2024-09-19T06:49:10.375913+0000","last_unstale":"2024-09-19T06:50:52.415263+0000","last_undegraded":"2024-09-19T06:50:52.415263+0000","last_fullsized":"2024-09-19T06:50:52.415263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T12:14:26.373766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.218467+0000","last_change":"2024-09-19T06:49:10.375479+0000","last_active":"2024-09-19T06:50:52.218467+0000","last_peered":"2024-09-19T06:50:52.218467+0000","last_clean":"2024-09-19T06:50:52.218467+0000","last_became_active":"2024-09-19T06:49:10.375201+0000","last_became_peered":"2024-09-19T06:49:10.375201+0000","last_unstale":"2024-09-19T06:50:52.218467+0000","last_undegraded":"2024-09-19T06:50:52.218467+0000","last_fullsized":"2024-09-19T06:50:52.218467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T09:44:56.421410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.222690+0000","last_change":"2024-09-19T06:49:10.374495+0000","last_active":"2024-09-19T06:50:52.222690+0000","last_peered":"2024-09-19T06:50:52.222690+0000","last_clean":"2024-09-19T06:50:52.222690+0000","last_became_active":"2024-09-19T06:49:10.374110+0000","last_became_peered":"2024-09-19T06:49:10.374110+0000","last_unstale":"2024-09-19T06:50:52.222690+0000","last_undegraded":"2024-09-19T06:50:52.222690+0000","last_fullsized":"2024-09-19T06:50:52.222690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T17:51:27.641785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.267168+0000","last_change":"2024-09-19T06:49:10.373548+0000","last_active":"2024-09-19T06:50:52.267168+0000","last_peered":"2024-09-19T06:50:52.267168+0000","last_clean":"2024-09-19T06:50:52.267168+0000","last_became_active":"2024-09-19T06:49:10.373251+0000","last_became_peered":"2024-09-19T06:49:10.373251+0000","last_unstale":"2024-09-19T06:50:52.267168+0000","last_undegraded":"2024-09-19T06:50:52.267168+0000","last_fullsized":"2024-09-19T06:50:52.267168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T10:59:35.353223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.215665+0000","last_change":"2024-09-19T06:49:10.383000+0000","last_active":"2024-09-19T06:50:52.215665+0000","last_peered":"2024-09-19T06:50:52.215665+0000","last_clean":"2024-09-19T06:50:52.215665+0000","last_became_active":"2024-09-19T06:49:10.382653+0000","last_became_peered":"2024-09-19T06:49:10.382653+0000","last_unstale":"2024-09-19T06:50:52.215665+0000","last_undegraded":"2024-09-19T06:50:52.215665+0000","last_fullsized":"2024-09-19T06:50:52.215665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T18:42:31.317042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.217451+0000","last_change":"2024-09-19T06:49:10.380239+0000","last_active":"2024-09-19T06:50:52.217451+0000","last_peered":"2024-09-19T06:50:52.217451+0000","last_clean":"2024-09-19T06:50:52.217451+0000","last_became_active":"2024-09-19T06:49:10.378491+0000","last_became_peered":"2024-09-19T06:49:10.378491+0000","last_unstale":"2024-09-19T06:50:52.217451+0000","last_undegraded":"2024-09-19T06:50:52.217451+0000","last_fullsized":"2024-09-19T06:50:52.217451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T08:56:14.016962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767120+0000","last_change":"2024-09-19T06:49:10.376989+0000","last_active":"2024-09-19T06:50:52.767120+0000","last_peered":"2024-09-19T06:50:52.767120+0000","last_clean":"2024-09-19T06:50:52.767120+0000","last_became_active":"2024-09-19T06:49:10.376701+0000","last_became_peered":"2024-09-19T06:49:10.376701+0000","last_unstale":"2024-09-19T06:50:52.767120+0000","last_undegraded":"2024-09-19T06:50:52.767120+0000","last_fullsized":"2024-09-19T06:50:52.767120+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T14:13:34.153487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.767231+0000","last_change":"2024-09-19T06:49:10.382585+0000","last_active":"2024-09-19T06:50:52.767231+0000","last_peered":"2024-09-19T06:50:52.767231+0000","last_clean":"2024-09-19T06:50:52.767231+0000","last_became_active":"2024-09-19T06:49:10.381573+0000","last_became_peered":"2024-09-19T06:49:10.381573+0000","last_unstale":"2024-09-19T06:50:52.767231+0000","last_undegraded":"2024-09-19T06:50:52.767231+0000","last_fullsized":"2024-09-19T06:50:52.767231+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:29:08.324309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-19T06:50:52.220839+0000","last_change":"2024-09-19T06:49:10.381681+0000","last_active":"2024-09-19T06:50:52.220839+0000","last_peered":"2024-09-19T06:50:52.220839+0000","last_clean":"2024-09-19T06:50:52.220839+0000","last_became_active":"2024-09-19T06:49:10.381344+0000","last_became_peered":"2024-09-19T06:49:10.381344+0000","last_unstale":"2024-09-19T06:50:52.220839+0000","last_undegraded":"2024-09-19T06:50:52.220839+0000","last_fullsized":"2024-09-19T06:50:52.220839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-19T06:49:09.345219+0000","last_clean_scrub_stamp":"2024-09-19T06:49:09.345219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T15:39:01.539679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69476,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":642,"num_read_kb":399,"num_write":248,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":208428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":3542208,"data_compressed":215885,"data_compressed_allocated":270336,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449123,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1064960,"data_stored":541446,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":71,"seq":304942678056,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1204224,"data_stored":1191114,"data_compressed":65674,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841583,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1081344,"data_stored":545270,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1597,"internal_metadata":31914435},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":58,"seq":249108103220,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1064960,"data_stored":541440,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332153,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1126400,"data_stored":1138051,"data_compressed":65674,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1052672,"data_stored":536678,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1130496,"data_stored":1133150,"data_compressed":65674,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":541879,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1064960,"data_stored":536386,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1126400,"data_stored":599855,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1122304,"data_stored":600369,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1077248,"data_stored":538134,"data_compressed":29418,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34605,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36256,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-19T06:51:02.823 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-19T06:51:02.823 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-19T06:51:02.823 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-19T06:51:02.823 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-19T06:51:03.226 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:03.553 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:03 smithi114 ceph-mon[30820]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:51:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:03 smithi116 ceph-mon[35411]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:51:03.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:03 smithi118 ceph-mon[35527]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-19T06:51:03.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 -- 172.21.15.114:0/1784518435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c104010 msgr2=0x7f0f7c1043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:03.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 --2- 172.21.15.114:0/1784518435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c104010 0x7f0f7c1043f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009900 tx=0x7f0f6c02a400 comp rx=0 tx=0).stop 2024-09-19T06:51:03.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 -- 172.21.15.114:0/1784518435 shutdown_connections 2024-09-19T06:51:03.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 --2- 172.21.15.114:0/1784518435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f7c109140 0x7f0f7c110cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:03.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 --2- 172.21.15.114:0/1784518435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c1049c0 0x7f0f7c108a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:03.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 --2- 172.21.15.114:0/1784518435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c104010 0x7f0f7c1043f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:03.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 -- 172.21.15.114:0/1784518435 >> 172.21.15.114:0/1784518435 conn(0x7f0f7c0ffa50 msgr2=0x7f0f7c101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:03.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 -- 172.21.15.114:0/1784518435 shutdown_connections 2024-09-19T06:51:03.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.774+0000 7f0f80e00640 1 -- 172.21.15.114:0/1784518435 wait complete. 2024-09-19T06:51:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.775+0000 7f0f80e00640 1 Processor -- start 2024-09-19T06:51:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.775+0000 7f0f80e00640 1 -- start start 2024-09-19T06:51:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.775+0000 7f0f80e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c104010 0x7f0f7c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.775+0000 7f0f80e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c1049c0 0x7f0f7c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f80e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f7c109140 0x7f0f7c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:03.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f80e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c114170 con 0x7f0f7c1049c0 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f80e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c113ff0 con 0x7f0f7c104010 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f80e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1142f0 con 0x7f0f7c109140 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c104010 0x7f0f7c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c104010 0x7f0f7c19a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:49194/0 (socket says 172.21.15.114:49194) 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c1049c0 0x7f0f7c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79e00640 1 -- 172.21.15.114:0/3285522892 learned_addr learned my addr 172.21.15.114:0/3285522892 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f7a800640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f7c109140 0x7f0f7c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79400640 1 -- 172.21.15.114:0/3285522892 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f7c109140 msgr2=0x7f0f7c19ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79400640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f7c109140 0x7f0f7c19ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79400640 1 -- 172.21.15.114:0/3285522892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c104010 msgr2=0x7f0f7c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79400640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c104010 0x7f0f7c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79400640 1 -- 172.21.15.114:0/3285522892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c19f600 con 0x7f0f7c1049c0 2024-09-19T06:51:03.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.776+0000 7f0f79e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c104010 0x7f0f7c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:03.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.777+0000 7f0f79400640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c1049c0 0x7f0f7c19ab90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800e9f0 tx=0x7f0f6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:03.779 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.777+0000 7f0f72a00640 1 -- 172.21.15.114:0/3285522892 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6800ccc0 con 0x7f0f7c1049c0 2024-09-19T06:51:03.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.777+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7c19f8f0 con 0x7f0f7c1049c0 2024-09-19T06:51:03.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.777+0000 7f0f72a00640 1 -- 172.21.15.114:0/3285522892 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f68010910 con 0x7f0f7c1049c0 2024-09-19T06:51:03.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.777+0000 7f0f72a00640 1 -- 172.21.15.114:0/3285522892 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f68018960 con 0x7f0f7c1049c0 2024-09-19T06:51:03.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.777+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f7c1a71d0 con 0x7f0f7c1049c0 2024-09-19T06:51:03.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.778+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7c1043f0 con 0x7f0f7c1049c0 2024-09-19T06:51:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.781+0000 7f0f72a00640 1 -- 172.21.15.114:0/3285522892 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f0f68010ac0 con 0x7f0f7c1049c0 2024-09-19T06:51:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.782+0000 7f0f72a00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0f600a32d0 0x7f0f600a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.782+0000 7f0f72a00640 1 -- 172.21.15.114:0/3285522892 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f0f680d0230 con 0x7f0f7c1049c0 2024-09-19T06:51:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.782+0000 7f0f79e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0f600a32d0 0x7f0f600a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.783+0000 7f0f79e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0f600a32d0 0x7f0f600a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0096c0 tx=0x7f0f6c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:03.786+0000 7f0f72a00640 1 -- 172.21.15.114:0/3285522892 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f0f6809c350 con 0x7f0f7c1049c0 2024-09-19T06:51:04.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.003+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0f7c02d7c0 con 0x7f0f7c1049c0 2024-09-19T06:51:04.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.003+0000 7f0f72a00640 1 -- 172.21.15.114:0/3285522892 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0f680a0650 con 0x7f0f7c1049c0 2024-09-19T06:51:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:04.006 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-19T06:51:04.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0f600a32d0 msgr2=0x7f0f600a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0f600a32d0 0x7f0f600a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0096c0 tx=0x7f0f6c005a20 comp rx=0 tx=0).stop 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c1049c0 msgr2=0x7f0f7c19ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c1049c0 0x7f0f7c19ab90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800e9f0 tx=0x7f0f6800eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 shutdown_connections 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f0f600a32d0 0x7f0f600a5790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0f7c109140 0x7f0f7c19ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f7c1049c0 0x7f0f7c19ab90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 --2- 172.21.15.114:0/3285522892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f7c104010 0x7f0f7c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 >> 172.21.15.114:0/3285522892 conn(0x7f0f7c0ffa50 msgr2=0x7f0f7c100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 shutdown_connections 2024-09-19T06:51:04.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:04.006+0000 7f0f80e00640 1 -- 172.21.15.114:0/3285522892 wait complete. 2024-09-19T06:51:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:04 smithi114 ceph-mon[30820]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 8 B/s, 0 objects/s recovering 2024-09-19T06:51:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:04 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3285522892' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-19T06:51:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:04 smithi116 ceph-mon[35411]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 8 B/s, 0 objects/s recovering 2024-09-19T06:51:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:04 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3285522892' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-19T06:51:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:04 smithi118 ceph-mon[35527]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 8 B/s, 0 objects/s recovering 2024-09-19T06:51:04.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:04 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3285522892' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-19T06:51:04.938 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-19T06:51:04.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:51:05.306 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:05.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 -- 172.21.15.114:0/3155380123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4528103e30 msgr2=0x7f4528104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:05.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 --2- 172.21.15.114:0/3155380123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4528103e30 0x7f4528104210 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4518009900 tx=0x7f451802a410 comp rx=0 tx=0).stop 2024-09-19T06:51:05.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 -- 172.21.15.114:0/3155380123 shutdown_connections 2024-09-19T06:51:05.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 --2- 172.21.15.114:0/3155380123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4528108f60 0x7f4528110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:05.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 --2- 172.21.15.114:0/3155380123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45281047e0 0x7f4528108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:05.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 --2- 172.21.15.114:0/3155380123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4528103e30 0x7f4528104210 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:05.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 -- 172.21.15.114:0/3155380123 >> 172.21.15.114:0/3155380123 conn(0x7f45280ff8d0 msgr2=0x7f4528101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:05.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 -- 172.21.15.114:0/3155380123 shutdown_connections 2024-09-19T06:51:05.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.831+0000 7f452da00640 1 -- 172.21.15.114:0/3155380123 wait complete. 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.832+0000 7f452da00640 1 Processor -- start 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.832+0000 7f452da00640 1 -- start start 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.832+0000 7f452da00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4528103e30 0x7f452819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.832+0000 7f452da00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45281047e0 0x7f452819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.832+0000 7f452da00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4528108f60 0x7f452819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.832+0000 7f452da00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4528114010 con 0x7f45281047e0 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.832+0000 7f452da00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4528113e90 con 0x7f4528103e30 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.832+0000 7f452da00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4528114190 con 0x7f4528108f60 2024-09-19T06:51:05.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f452d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4528108f60 0x7f452819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:05.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f4527400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45281047e0 0x7f452819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:05.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f452d000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4528108f60 0x7f452819ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:55052/0 (socket says 172.21.15.114:55052) 2024-09-19T06:51:05.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f452d000640 1 -- 172.21.15.114:0/901845878 learned_addr learned my addr 172.21.15.114:0/901845878 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:05.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f4527e00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4528103e30 0x7f452819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:05.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f4527400640 1 -- 172.21.15.114:0/901845878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4528108f60 msgr2=0x7f452819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f4527400640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4528108f60 0x7f452819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f4527400640 1 -- 172.21.15.114:0/901845878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4528103e30 msgr2=0x7f452819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f4527400640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4528103e30 0x7f452819a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.833+0000 7f4527400640 1 -- 172.21.15.114:0/901845878 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452819f450 con 0x7f45281047e0 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.834+0000 7f4527400640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45281047e0 0x7f452819a9e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f451400e9f0 tx=0x7f451400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.834+0000 7f4524c00640 1 -- 172.21.15.114:0/901845878 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451400ccc0 con 0x7f45281047e0 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.834+0000 7f4524c00640 1 -- 172.21.15.114:0/901845878 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4514005020 con 0x7f45281047e0 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.834+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452819f740 con 0x7f45281047e0 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.834+0000 7f4524c00640 1 -- 172.21.15.114:0/901845878 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45140049e0 con 0x7f45281047e0 2024-09-19T06:51:05.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.834+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45281a7020 con 0x7f45281047e0 2024-09-19T06:51:05.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.835+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4528104210 con 0x7f45281047e0 2024-09-19T06:51:05.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.837+0000 7f4524c00640 1 -- 172.21.15.114:0/901845878 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f4514005560 con 0x7f45281047e0 2024-09-19T06:51:05.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.839+0000 7f4524c00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f450c0a3350 0x7f450c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:05.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.839+0000 7f4524c00640 1 -- 172.21.15.114:0/901845878 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f45140cfe00 con 0x7f45281047e0 2024-09-19T06:51:05.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.839+0000 7f4527e00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f450c0a3350 0x7f450c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:05.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.840+0000 7f4527e00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f450c0a3350 0x7f450c0a5810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4518000c00 tx=0x7f4518005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:05.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:05.843+0000 7f4524c00640 1 -- 172.21.15.114:0/901845878 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f45140051d0 con 0x7f45281047e0 2024-09-19T06:51:06.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.054+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f452802d7c0 con 0x7f45281047e0 2024-09-19T06:51:06.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.055+0000 7f4524c00640 1 -- 172.21.15.114:0/901845878 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11999 (secure 0 0 0) 0x7f451409c570 con 0x7f45281047e0 2024-09-19T06:51:06.057 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:06.057 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"btime":"2024-09-19T06:50:00:244614+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T06:49:11.450651+0000","modified":"2024-09-19T06:49:59.255237+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24538,"mds_1":24524,"mds_2":24520,"mds_3":24508,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14520":{"gid":14520,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":21,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":20,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24538,"qdb_cluster":[24482,24508,24520,24524,24538]},"id":1}]} 2024-09-19T06:51:06.057 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-09-19T06:51:06.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f450c0a3350 msgr2=0x7f450c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:06.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f450c0a3350 0x7f450c0a5810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4518000c00 tx=0x7f4518005b50 comp rx=0 tx=0).stop 2024-09-19T06:51:06.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45281047e0 msgr2=0x7f452819a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:06.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45281047e0 0x7f452819a9e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f451400e9f0 tx=0x7f451400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:06.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 shutdown_connections 2024-09-19T06:51:06.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f450c0a3350 0x7f450c0a5810 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:06.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4528108f60 0x7f452819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:06.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45281047e0 0x7f452819a9e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:06.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 --2- 172.21.15.114:0/901845878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4528103e30 0x7f452819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:06.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 >> 172.21.15.114:0/901845878 conn(0x7f45280ff8d0 msgr2=0x7f4528100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:06.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 shutdown_connections 2024-09-19T06:51:06.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:06.058+0000 7f452da00640 1 -- 172.21.15.114:0/901845878 wait complete. 2024-09-19T06:51:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:06 smithi114 ceph-mon[30820]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 7 B/s, 0 objects/s recovering 2024-09-19T06:51:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:06 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/901845878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:06 smithi116 ceph-mon[35411]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 7 B/s, 0 objects/s recovering 2024-09-19T06:51:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:06 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/901845878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:06 smithi118 ceph-mon[35527]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 7 B/s, 0 objects/s recovering 2024-09-19T06:51:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:06 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/901845878' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:06.855 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:51:07.222 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:07.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.794+0000 7fdfdd000640 1 -- 172.21.15.114:0/3939443680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8108f60 msgr2=0x7fdfd8110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.794+0000 7fdfdd000640 1 --2- 172.21.15.114:0/3939443680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8108f60 0x7fdfd8110ae0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009be0 tx=0x7fdfd002a6c0 comp rx=0 tx=0).stop 2024-09-19T06:51:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.794+0000 7fdfdd000640 1 -- 172.21.15.114:0/3939443680 shutdown_connections 2024-09-19T06:51:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.794+0000 7fdfdd000640 1 --2- 172.21.15.114:0/3939443680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8108f60 0x7fdfd8110ae0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.794+0000 7fdfdd000640 1 --2- 172.21.15.114:0/3939443680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdfd81047e0 0x7fdfd8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.794+0000 7fdfdd000640 1 --2- 172.21.15.114:0/3939443680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd8103e30 0x7fdfd8104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.794+0000 7fdfdd000640 1 -- 172.21.15.114:0/3939443680 >> 172.21.15.114:0/3939443680 conn(0x7fdfd80ff8d0 msgr2=0x7fdfd8101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.795+0000 7fdfdd000640 1 -- 172.21.15.114:0/3939443680 shutdown_connections 2024-09-19T06:51:07.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.795+0000 7fdfdd000640 1 -- 172.21.15.114:0/3939443680 wait complete. 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.795+0000 7fdfdd000640 1 Processor -- start 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfdd000640 1 -- start start 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfdd000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8103e30 0x7fdfd819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfdd000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdfd81047e0 0x7fdfd819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfdd000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd8108f60 0x7fdfd819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfdd000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd8113fc0 con 0x7fdfd8103e30 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfdd000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd8113e40 con 0x7fdfd8108f60 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfdd000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd8114140 con 0x7fdfd81047e0 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfd7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8103e30 0x7fdfd819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:07.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfd7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8103e30 0x7fdfd819a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38506/0 (socket says 172.21.15.114:38506) 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfd7400640 1 -- 172.21.15.114:0/12674928 learned_addr learned my addr 172.21.15.114:0/12674928 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.796+0000 7fdfd6a00640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdfd81047e0 0x7fdfd819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfd7e00640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd8108f60 0x7fdfd819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfd7400640 1 -- 172.21.15.114:0/12674928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdfd81047e0 msgr2=0x7fdfd819a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfd7400640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdfd81047e0 0x7fdfd819a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfd7400640 1 -- 172.21.15.114:0/12674928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd8108f60 msgr2=0x7fdfd819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfd7400640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd8108f60 0x7fdfd819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfd7400640 1 -- 172.21.15.114:0/12674928 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd819f3d0 con 0x7fdfd8103e30 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfd7e00640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd8108f60 0x7fdfd819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfd7400640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8103e30 0x7fdfd819a420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800e9c0 tx=0x7fdfc800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfcfe00640 1 -- 172.21.15.114:0/12674928 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc800ccc0 con 0x7fdfd8103e30 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfcfe00640 1 -- 172.21.15.114:0/12674928 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc8005500 con 0x7fdfd8103e30 2024-09-19T06:51:07.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd819f6c0 con 0x7fdfd8103e30 2024-09-19T06:51:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.797+0000 7fdfcfe00640 1 -- 172.21.15.114:0/12674928 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8010430 con 0x7fdfd8103e30 2024-09-19T06:51:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.798+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd81a6fa0 con 0x7fdfd8103e30 2024-09-19T06:51:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.799+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd8104210 con 0x7fdfd8103e30 2024-09-19T06:51:07.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.800+0000 7fdfcfe00640 1 -- 172.21.15.114:0/12674928 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fdfc80046c0 con 0x7fdfd8103e30 2024-09-19T06:51:07.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.801+0000 7fdfcfe00640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdfbc0a3350 0x7fdfbc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.802+0000 7fdfcfe00640 1 -- 172.21.15.114:0/12674928 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fdfc80d0200 con 0x7fdfd8103e30 2024-09-19T06:51:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.802+0000 7fdfd6a00640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdfbc0a3350 0x7fdfbc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:07.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.802+0000 7fdfd6a00640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdfbc0a3350 0x7fdfbc0a5810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdfd810a930 tx=0x7fdfc4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:07.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:07.805+0000 7fdfcfe00640 1 -- 172.21.15.114:0/12674928 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdfc809c970 con 0x7fdfd8103e30 2024-09-19T06:51:08.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.015+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfd819b780 con 0x7fdfd8103e30 2024-09-19T06:51:08.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.016+0000 7fdfcfe00640 1 -- 172.21.15.114:0/12674928 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11999 (secure 0 0 0) 0x7fdfd819b780 con 0x7fdfd8103e30 2024-09-19T06:51:08.018 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:08.018 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"btime":"2024-09-19T06:50:00:244614+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T06:49:11.450651+0000","modified":"2024-09-19T06:49:59.255237+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24538,"mds_1":24524,"mds_2":24520,"mds_3":24508,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14520":{"gid":14520,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":21,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":20,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24538,"qdb_cluster":[24482,24508,24520,24524,24538]},"id":1}]} 2024-09-19T06:51:08.019 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-09-19T06:51:08.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdfbc0a3350 msgr2=0x7fdfbc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:08.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdfbc0a3350 0x7fdfbc0a5810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdfd810a930 tx=0x7fdfc4005f90 comp rx=0 tx=0).stop 2024-09-19T06:51:08.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8103e30 msgr2=0x7fdfd819a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:08.020 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8103e30 0x7fdfd819a420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800e9c0 tx=0x7fdfc800ee80 comp rx=0 tx=0).stop 2024-09-19T06:51:08.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 shutdown_connections 2024-09-19T06:51:08.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdfbc0a3350 0x7fdfbc0a5810 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:08.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd8108f60 0x7fdfd819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdfd81047e0 0x7fdfd819a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 --2- 172.21.15.114:0/12674928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdfd8103e30 0x7fdfd819a420 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 >> 172.21.15.114:0/12674928 conn(0x7fdfd80ff8d0 msgr2=0x7fdfd8100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 shutdown_connections 2024-09-19T06:51:08.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:08.019+0000 7fdfdd000640 1 -- 172.21.15.114:0/12674928 wait complete. 2024-09-19T06:51:08.558 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:08 smithi118 ceph-mon[35527]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 7 B/s, 0 objects/s recovering 2024-09-19T06:51:08.558 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:08 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/12674928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:08 smithi114 ceph-mon[30820]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 7 B/s, 0 objects/s recovering 2024-09-19T06:51:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:08 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/12674928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:08 smithi116 ceph-mon[35411]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 7 B/s, 0 objects/s recovering 2024-09-19T06:51:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:08 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/12674928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:08.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:51:09.142 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.756+0000 7fdc33e00640 1 -- 172.21.15.114:0/2645913897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c074620 msgr2=0x7fdc2c074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.756+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2645913897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c074620 0x7fdc2c074a00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009900 tx=0x7fdc2002a480 comp rx=0 tx=0).stop 2024-09-19T06:51:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.756+0000 7fdc33e00640 1 -- 172.21.15.114:0/2645913897 shutdown_connections 2024-09-19T06:51:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.756+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2645913897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc2c10e3c0 0x7fdc2c110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.756+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2645913897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c074f40 0x7fdc2c10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.756+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2645913897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c074620 0x7fdc2c074a00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.756+0000 7fdc33e00640 1 -- 172.21.15.114:0/2645913897 >> 172.21.15.114:0/2645913897 conn(0x7fdc2c0fd8c0 msgr2=0x7fdc2c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:09.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.756+0000 7fdc33e00640 1 -- 172.21.15.114:0/2645913897 shutdown_connections 2024-09-19T06:51:09.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.757+0000 7fdc33e00640 1 -- 172.21.15.114:0/2645913897 wait complete. 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.757+0000 7fdc33e00640 1 Processor -- start 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc33e00640 1 -- start start 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc33e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c074620 0x7fdc2c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc33e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc2c074f40 0x7fdc2c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc33e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c10e3c0 0x7fdc2c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc33e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c1141c0 con 0x7fdc2c074f40 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc33e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c114040 con 0x7fdc2c10e3c0 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc33e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdc2c114340 con 0x7fdc2c074620 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc30a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc2c074f40 0x7fdc2c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc30a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc2c074f40 0x7fdc2c19abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38530/0 (socket says 172.21.15.114:38530) 2024-09-19T06:51:09.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc31400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c074620 0x7fdc2c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc30a00640 1 -- 172.21.15.114:0/2286053156 learned_addr learned my addr 172.21.15.114:0/2286053156 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.758+0000 7fdc31e00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c10e3c0 0x7fdc2c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc30a00640 1 -- 172.21.15.114:0/2286053156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c074620 msgr2=0x7fdc2c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc30a00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c074620 0x7fdc2c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc30a00640 1 -- 172.21.15.114:0/2286053156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c10e3c0 msgr2=0x7fdc2c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc30a00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c10e3c0 0x7fdc2c19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc30a00640 1 -- 172.21.15.114:0/2286053156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2c19f650 con 0x7fdc2c074f40 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc31e00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c10e3c0 0x7fdc2c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc31400640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c074620 0x7fdc2c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc30a00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc2c074f40 0x7fdc2c19abe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e9f0 tx=0x7fdc1800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc2a000640 1 -- 172.21.15.114:0/2286053156 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800ccc0 con 0x7fdc2c074f40 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc2a000640 1 -- 172.21.15.114:0/2286053156 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc1800ce60 con 0x7fdc2c074f40 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2c19f940 con 0x7fdc2c074f40 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.759+0000 7fdc2a000640 1 -- 172.21.15.114:0/2286053156 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc18010690 con 0x7fdc2c074f40 2024-09-19T06:51:09.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.760+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2c1a7220 con 0x7fdc2c074f40 2024-09-19T06:51:09.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.761+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc2c074a00 con 0x7fdc2c074f40 2024-09-19T06:51:09.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.763+0000 7fdc2a000640 1 -- 172.21.15.114:0/2286053156 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fdc180046c0 con 0x7fdc2c074f40 2024-09-19T06:51:09.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.764+0000 7fdc2a000640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdc140a3350 0x7fdc140a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:09.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.765+0000 7fdc2a000640 1 -- 172.21.15.114:0/2286053156 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fdc180d0250 con 0x7fdc2c074f40 2024-09-19T06:51:09.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.765+0000 7fdc31400640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdc140a3350 0x7fdc140a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:09.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.765+0000 7fdc31400640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdc140a3350 0x7fdc140a5810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdc20000c00 tx=0x7fdc20031510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:09.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.769+0000 7fdc2a000640 1 -- 172.21.15.114:0/2286053156 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdc1809c9c0 con 0x7fdc2c074f40 2024-09-19T06:51:09.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.946+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc2c02d7c0 con 0x7fdc2c074f40 2024-09-19T06:51:09.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.948+0000 7fdc2a000640 1 -- 172.21.15.114:0/2286053156 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25089 (secure 0 0 0) 0x7fdc1809c2c0 con 0x7fdc2c074f40 2024-09-19T06:51:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:09.950 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":95,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:50:52.207228+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:51:09.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdc140a3350 msgr2=0x7fdc140a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:09.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdc140a3350 0x7fdc140a5810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdc20000c00 tx=0x7fdc20031510 comp rx=0 tx=0).stop 2024-09-19T06:51:09.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc2c074f40 msgr2=0x7fdc2c19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:09.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc2c074f40 0x7fdc2c19abe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800e9f0 tx=0x7fdc1800edf0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 shutdown_connections 2024-09-19T06:51:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdc140a3350 0x7fdc140a5810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc2c10e3c0 0x7fdc2c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc2c074f40 0x7fdc2c19abe0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 --2- 172.21.15.114:0/2286053156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c074620 0x7fdc2c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 >> 172.21.15.114:0/2286053156 conn(0x7fdc2c0fd8c0 msgr2=0x7fdc2c0fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.950+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 shutdown_connections 2024-09-19T06:51:09.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:09.951+0000 7fdc33e00640 1 -- 172.21.15.114:0/2286053156 wait complete. 2024-09-19T06:51:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:10 smithi114 ceph-mon[30820]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-09-19T06:51:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:10 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/2286053156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:51:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:10 smithi116 ceph-mon[35411]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-09-19T06:51:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:10 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/2286053156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:51:10.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:51:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:10 smithi118 ceph-mon[35527]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-09-19T06:51:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:10 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/2286053156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:51:11.100 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:11.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:51:11.603 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:11 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:51:11.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:51:11.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 -- 172.21.15.114:0/596910013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540102790 msgr2=0x7f75401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:11.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 --2- 172.21.15.114:0/596910013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540102790 0x7f75401067e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7528009900 tx=0x7f752802a3f0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 -- 172.21.15.114:0/596910013 shutdown_connections 2024-09-19T06:51:11.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 --2- 172.21.15.114:0/596910013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7540106f10 0x7f754010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 --2- 172.21.15.114:0/596910013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540102790 0x7f75401067e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 --2- 172.21.15.114:0/596910013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7540101de0 0x7f75401021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 -- 172.21.15.114:0/596910013 >> 172.21.15.114:0/596910013 conn(0x7f75400fd8c0 msgr2=0x7f75400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:11.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 -- 172.21.15.114:0/596910013 shutdown_connections 2024-09-19T06:51:11.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.664+0000 7f7546a00640 1 -- 172.21.15.114:0/596910013 wait complete. 2024-09-19T06:51:11.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.665+0000 7f7546a00640 1 Processor -- start 2024-09-19T06:51:11.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.665+0000 7f7546a00640 1 -- start start 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.665+0000 7f7546a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540101de0 0x7f754019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.665+0000 7f7546a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7540102790 0x7f754019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f7546a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7540106f10 0x7f754019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f7546a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7540111fc0 con 0x7f7540102790 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f7546a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7540111e40 con 0x7f7540106f10 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f7546a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f7540112140 con 0x7f7540101de0 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f753fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540101de0 0x7f754019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f7537400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7540102790 0x7f754019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f7544a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7540106f10 0x7f754019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f753fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540101de0 0x7f754019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:53768/0 (socket says 172.21.15.114:53768) 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f753fe00640 1 -- 172.21.15.114:0/345045946 learned_addr learned my addr 172.21.15.114:0/345045946 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:11.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f753fe00640 1 -- 172.21.15.114:0/345045946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7540106f10 msgr2=0x7f754019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f753fe00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7540106f10 0x7f754019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f753fe00640 1 -- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7540102790 msgr2=0x7f754019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f753fe00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7540102790 0x7f754019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.666+0000 7f753fe00640 1 -- 172.21.15.114:0/345045946 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754019f610 con 0x7f7540101de0 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.667+0000 7f753fe00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540101de0 0x7f754019a660 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f753000e9f0 tx=0x7f753000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.667+0000 7f753d600640 1 -- 172.21.15.114:0/345045946 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753000cd80 con 0x7f7540101de0 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.667+0000 7f753d600640 1 -- 172.21.15.114:0/345045946 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7530010910 con 0x7f7540101de0 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.667+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754019f900 con 0x7f7540101de0 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.667+0000 7f753d600640 1 -- 172.21.15.114:0/345045946 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7530018990 con 0x7f7540101de0 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.667+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75401a71e0 con 0x7f7540101de0 2024-09-19T06:51:11.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.668+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75401021c0 con 0x7f7540101de0 2024-09-19T06:51:11.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.670+0000 7f753d600640 1 -- 172.21.15.114:0/345045946 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f7530010ac0 con 0x7f7540101de0 2024-09-19T06:51:11.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.671+0000 7f753d600640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f75240a32d0 0x7f75240a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:11.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.671+0000 7f753d600640 1 -- 172.21.15.114:0/345045946 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f75300d0200 con 0x7f7540101de0 2024-09-19T06:51:11.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.671+0000 7f7537400640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f75240a32d0 0x7f75240a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:11.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.672+0000 7f7537400640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f75240a32d0 0x7f75240a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7528000c00 tx=0x7f7528005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:11.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.675+0000 7f753d600640 1 -- 172.21.15.114:0/345045946 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f753009c320 con 0x7f7540101de0 2024-09-19T06:51:11.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.926+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75401a7410 con 0x7f7540101de0 2024-09-19T06:51:11.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.926+0000 7f753d600640 1 -- 172.21.15.114:0/345045946 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11999 (secure 0 0 0) 0x7f75300a0620 con 0x7f7540101de0 2024-09-19T06:51:11.928 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:11.928 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"btime":"2024-09-19T06:50:00:244614+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T06:49:11.450651+0000","modified":"2024-09-19T06:49:59.255237+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24538,"mds_1":24524,"mds_2":24520,"mds_3":24508,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14520":{"gid":14520,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":21,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":20,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24538,"qdb_cluster":[24482,24508,24520,24524,24538]},"id":1}]} 2024-09-19T06:51:11.928 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-09-19T06:51:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f75240a32d0 msgr2=0x7f75240a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f75240a32d0 0x7f75240a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7528000c00 tx=0x7f7528005b50 comp rx=0 tx=0).stop 2024-09-19T06:51:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540101de0 msgr2=0x7f754019a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:11.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540101de0 0x7f754019a660 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f753000e9f0 tx=0x7f753000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 shutdown_connections 2024-09-19T06:51:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f75240a32d0 0x7f75240a5790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7540106f10 0x7f754019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7540102790 0x7f754019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 --2- 172.21.15.114:0/345045946 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7540101de0 0x7f754019a660 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 >> 172.21.15.114:0/345045946 conn(0x7f75400fd8c0 msgr2=0x7f75400feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 shutdown_connections 2024-09-19T06:51:11.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:11.929+0000 7f7546a00640 1 -- 172.21.15.114:0/345045946 wait complete. 2024-09-19T06:51:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:12 smithi114 ceph-mon[30820]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:12 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/345045946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:12 smithi116 ceph-mon[35411]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:12 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/345045946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:12 smithi118 ceph-mon[35527]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:12 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/345045946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:13.342 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-19T06:49:11.450651+0000', 'modified': '2024-09-19T06:49:59.255237+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24538, 'mds_1': 24524, 'mds_2': 24520, 'mds_3': 24508, 'mds_4': 24482}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14520': {'gid': 14520, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6841/2459666351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6840', 'nonce': 2459666351}, {'type': 'v1', 'addr': '172.21.15.114:6841', 'nonce': 2459666351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24419': {'gid': 24419, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6833/1884054485', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6832', 'nonce': 1884054485}, {'type': 'v1', 'addr': '172.21.15.116:6833', 'nonce': 1884054485}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/2797655862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 2797655862}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 2797655862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/2147208941', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 2147208941}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 2147208941}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'l', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.118:6833/3230062684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6832', 'nonce': 3230062684}, {'type': 'v1', 'addr': '172.21.15.118:6833', 'nonce': 3230062684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.116:6837/2673389539', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 2673389539}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 2673389539}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.118:6835/449495659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6834', 'nonce': 449495659}, {'type': 'v1', 'addr': '172.21.15.118:6835', 'nonce': 449495659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.118:6837/4109230511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6836', 'nonce': 4109230511}, {'type': 'v1', 'addr': '172.21.15.118:6837', 'nonce': 4109230511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.116:6839/2386737584', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 2386737584}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 2386737584}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.118:6839/345383252', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6838', 'nonce': 345383252}, {'type': 'v1', 'addr': '172.21.15.118:6839', 'nonce': 345383252}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24538, 'qdb_cluster': [24482, 24508, 24520, 24524, 24538]} 2024-09-19T06:51:13.342 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-19T06:51:13.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-19T06:51:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:13 smithi114 ceph-mon[30820]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:13 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:13 smithi118 ceph-mon[35527]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:13.767 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:13 smithi116 ceph-mon[35411]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:14.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.320+0000 7fc8b7200640 1 -- 172.21.15.114:0/2920519996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 msgr2=0x7fc8b8114c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.320+0000 7fc8b7200640 1 --2- 172.21.15.114:0/2920519996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 0x7fc8b8114c60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009ca0 tx=0x7fc8b002a8c0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.321+0000 7fc8b7200640 1 -- 172.21.15.114:0/2920519996 shutdown_connections 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.321+0000 7fc8b7200640 1 --2- 172.21.15.114:0/2920519996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 0x7fc8b8114c60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.321+0000 7fc8b7200640 1 --2- 172.21.15.114:0/2920519996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8b8077040 0x7fc8b8112330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.321+0000 7fc8b7200640 1 --2- 172.21.15.114:0/2920519996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8076720 0x7fc8b8076b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.321+0000 7fc8b7200640 1 -- 172.21.15.114:0/2920519996 >> 172.21.15.114:0/2920519996 conn(0x7fc8b80ff8d0 msgr2=0x7fc8b8101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.321+0000 7fc8b7200640 1 -- 172.21.15.114:0/2920519996 shutdown_connections 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.321+0000 7fc8b7200640 1 -- 172.21.15.114:0/2920519996 wait complete. 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b7200640 1 Processor -- start 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b7200640 1 -- start start 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b7200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8b8076720 0x7fc8b81a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b7200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8077040 0x7fc8b81a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 0x7fc8b819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b8118400 con 0x7fc8b8112870 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b7200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b8118280 con 0x7fc8b8077040 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b7200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b8118580 con 0x7fc8b8076720 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.322+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8b8076720 0x7fc8b81a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8b8076720 0x7fc8b81a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:53784/0 (socket says 172.21.15.114:53784) 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b6800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 0x7fc8b819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b5e00640 1 -- 172.21.15.114:0/569898993 learned_addr learned my addr 172.21.15.114:0/569898993 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b5400640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8077040 0x7fc8b81a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b6800640 1 -- 172.21.15.114:0/569898993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8b8076720 msgr2=0x7fc8b81a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b6800640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8b8076720 0x7fc8b81a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b6800640 1 -- 172.21.15.114:0/569898993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8077040 msgr2=0x7fc8b81a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b6800640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8077040 0x7fc8b81a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b6800640 1 -- 172.21.15.114:0/569898993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b819ef80 con 0x7fc8b8112870 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8b8076720 0x7fc8b81a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.323+0000 7fc8b6800640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 0x7fc8b819e710 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc8b002aeb0 tx=0x7fc8b0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.324+0000 7fc8aea00640 1 -- 172.21.15.114:0/569898993 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b0031650 con 0x7fc8b8112870 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.324+0000 7fc8aea00640 1 -- 172.21.15.114:0/569898993 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8b00317f0 con 0x7fc8b8112870 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.324+0000 7fc8aea00640 1 -- 172.21.15.114:0/569898993 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b002fa30 con 0x7fc8b8112870 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.324+0000 7fc8b7200640 1 -- 172.21.15.114:0/569898993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b819f210 con 0x7fc8b8112870 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.324+0000 7fc8b7200640 1 -- 172.21.15.114:0/569898993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b81ab3f0 con 0x7fc8b8112870 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.327+0000 7fc8aea00640 1 -- 172.21.15.114:0/569898993 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fc8b002fbd0 con 0x7fc8b8112870 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.327+0000 7fc8bfc24740 1 Processor -- start 2024-09-19T06:51:14.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.327+0000 7fc8bfc24740 1 -- start start 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd97a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97ae00 0x5644bd985180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd987ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5644bd8e00b0 con 0x5644bd97ae00 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5644bd887ab0 con 0x5644bd9856c0 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5644bd8dc070 con 0x5644bd97a4e0 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97ae00 0x5644bd985180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97ae00 0x5644bd985180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38566/0 (socket says 172.21.15.114:38566) 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5e00640 1 -- 172.21.15.114:0/664539481 learned_addr learned my addr 172.21.15.114:0/664539481 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5400640 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd97a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5400640 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd987ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5e00640 1 -- 172.21.15.114:0/664539481 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 msgr2=0x5644bd97a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd97a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5e00640 1 -- 172.21.15.114:0/664539481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 msgr2=0x5644bd987ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd987ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.328+0000 7fc8b5e00640 1 -- 172.21.15.114:0/664539481 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644bd8944a0 con 0x5644bd97ae00 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.329+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97ae00 0x5644bd985180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8011650 tx=0x7fc8a8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba29e0413ddf2395 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.329+0000 7fc8aea00640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc89c0a3350 0x7fc89c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.329+0000 7fc8aea00640 1 -- 172.21.15.114:0/569898993 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fc8b00f0ff0 con 0x7fc8b8112870 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.329+0000 7fc8a3400640 1 -- 172.21.15.114:0/664539481 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a801e070 con 0x5644bd97ae00 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.329+0000 7fc8b5400640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc89c0a3350 0x7fc89c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.329+0000 7fc8a3400640 1 -- 172.21.15.114:0/664539481 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8a800fba0 con 0x5644bd97ae00 2024-09-19T06:51:14.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.329+0000 7fc8a3400640 1 -- 172.21.15.114:0/664539481 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8019cb0 con 0x5644bd97ae00 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 -- 172.21.15.114:0/664539481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97ae00 msgr2=0x5644bd985180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8b5400640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc89c0a3350 0x7fc89c0a5810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5644bd977df0 tx=0x7fc8a400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97ae00 0x5644bd985180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8011650 tx=0x7fc8a8009f90 comp rx=0 tx=0).stop 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 -- 172.21.15.114:0/664539481 shutdown_connections 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd987ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97ae00 0x5644bd985180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/664539481 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd97a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 -- 172.21.15.114:0/664539481 >> 172.21.15.114:0/664539481 conn(0x5644bd976a70 msgr2=0x5644bd976e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:14.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 -- 172.21.15.114:0/664539481 shutdown_connections 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.330+0000 7fc8bfc24740 1 -- 172.21.15.114:0/664539481 wait complete. 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 1 -- start start 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 1 Processor -- start 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97a4e0 0x5644bd982560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd982aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd982fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5644bd988330 con 0x5644bd97a4e0 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5644bd868920 con 0x5644bd9856c0 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5644bd9884b0 con 0x5644bd97ae00 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8bfc24740 10 client.0 _ll_register_callbacks cb 0x5644bd88c8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8b5400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd982aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97a4e0 0x5644bd982560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8b5400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd982aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:53812/0 (socket says 172.21.15.114:53812) 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.331+0000 7fc8b5400640 1 -- 172.21.15.114:0/2644494661 learned_addr learned my addr 172.21.15.114:0/2644494661 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8b6800640 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd982fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8b6800640 1 -- 172.21.15.114:0/2644494661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 msgr2=0x5644bd982aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8b6800640 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd982aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8b6800640 1 -- 172.21.15.114:0/2644494661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97a4e0 msgr2=0x5644bd982560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8b6800640 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97a4e0 0x5644bd982560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8b6800640 1 -- 172.21.15.114:0/2644494661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644bd6512b0 con 0x5644bd9856c0 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97a4e0 0x5644bd982560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8b6800640 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd982fe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00ee680 tx=0x7fc8b00bb6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28dfa4d84200dba9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:14.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc897e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc897e00640 1 -- 172.21.15.114:0/2644494661 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8b00c1540 con 0x5644bd9856c0 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc897e00640 1 -- 172.21.15.114:0/2644494661 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fc8b00dd0c0 con 0x5644bd9856c0 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.332+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2644494661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8a800fba0 con 0x5644bd9856c0 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc897e00640 1 -- 172.21.15.114:0/2644494661 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc8b00c5e00 con 0x5644bd9856c0 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc8bfc24740 10 client.24965 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2644494661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5644bd988330 con 0x5644bd9856c0 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc897e00640 1 -- 172.21.15.114:0/2644494661 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8a800fba0 con 0x5644bd9856c0 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc897e00640 1 -- 172.21.15.114:0/2644494661 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc8b00c5970 con 0x5644bd9856c0 2024-09-19T06:51:14.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc8bfc24740 10 client.24965 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc8bfc24740 10 client.24965 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc8bfc24740 20 client.24965 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x5644bd996690 0x5644bd997b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc8bfc24740 4 client.24965 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.333+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2644494661 --> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644bd8ed620 con 0x5644bd996690 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.334+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x5644bd996690 0x5644bd997b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.334+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x5644bd996690 0x5644bd997b00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=63fbb6953532b4df server_cookie=f1a8eaa14ae84134 in_seq=0 out_seq=0 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.334+0000 7fc897e00640 10 client.24965 ms_handle_connect on v2:172.21.15.116:6836/2673389539 2024-09-19T06:51:14.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc897e00640 1 -- 172.21.15.114:0/2644494661 <== mds.4 v2:172.21.15.116:6836/2673389539 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5644bd8ed620 con 0x5644bd996690 2024-09-19T06:51:14.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc897e00640 10 client.24965 handle_command_reply: tid=0 2024-09-19T06:51:14.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 client.24965 shutdown 2024-09-19T06:51:14.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 msgr2=0x5644bd982fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd982fe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00ee680 tx=0x7fc8b00bb6a0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2644494661 shutdown_connections 2024-09-19T06:51:14.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x5644bd996690 0x5644bd997b00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd9856c0 0x5644bd982fe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd982aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc897e00640 0 client.24965 ms_handle_reset on v2:172.21.15.116:6836/2673389539 2024-09-19T06:51:14.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2644494661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd97a4e0 0x5644bd982560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2644494661 >> 172.21.15.114:0/2644494661 conn(0x5644bd976a70 msgr2=0x5644bd97c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:14.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2644494661 shutdown_connections 2024-09-19T06:51:14.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.335+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2644494661 wait complete. 2024-09-19T06:51:14.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.336+0000 7fc8bfc24740 20 client.24965 trim_cache size 0 max 16384 2024-09-19T06:51:14.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8bfc24740 1 Processor -- start 2024-09-19T06:51:14.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8bfc24740 1 -- start start 2024-09-19T06:51:14.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd997b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97ae00 0x5644bd995110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 0x5644bd995650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5644bd995e40 con 0x5644bd9856c0 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5644bd996510 con 0x5644bd97ae00 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5644bd996be0 con 0x5644bd97a4e0 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd997b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.355+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd997b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:53820/0 (socket says 172.21.15.114:53820) 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b5e00640 1 -- 172.21.15.114:0/3891019074 learned_addr learned my addr 172.21.15.114:0/3891019074 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b6800640 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 0x5644bd995650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b5400640 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97ae00 0x5644bd995110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b5e00640 1 -- 172.21.15.114:0/3891019074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97ae00 msgr2=0x5644bd995110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97ae00 0x5644bd995110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.359 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b5e00640 1 -- 172.21.15.114:0/3891019074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 msgr2=0x5644bd995650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 0x5644bd995650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b5e00640 1 -- 172.21.15.114:0/3891019074 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644bd9768a0 con 0x5644bd97a4e0 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.356+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd997b00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8005fd0 tx=0x7fc8a800fd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a59513f9a65329a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.357+0000 7fc8a2a00640 1 -- 172.21.15.114:0/3891019074 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a801e040 con 0x5644bd97a4e0 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.357+0000 7fc8a2a00640 1 -- 172.21.15.114:0/3891019074 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8a8029030 con 0x5644bd97a4e0 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.357+0000 7fc8a2a00640 1 -- 172.21.15.114:0/3891019074 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8019af0 con 0x5644bd97a4e0 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 -- 172.21.15.114:0/3891019074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 msgr2=0x5644bd997b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd997b00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8005fd0 tx=0x7fc8a800fd30 comp rx=0 tx=0).stop 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 -- 172.21.15.114:0/3891019074 shutdown_connections 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 0x5644bd995650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97ae00 0x5644bd995110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/3891019074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97a4e0 0x5644bd997b00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 -- 172.21.15.114:0/3891019074 >> 172.21.15.114:0/3891019074 conn(0x5644bd976a70 msgr2=0x5644bd9717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 -- 172.21.15.114:0/3891019074 shutdown_connections 2024-09-19T06:51:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 -- 172.21.15.114:0/3891019074 wait complete. 2024-09-19T06:51:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 -- start start 2024-09-19T06:51:14.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.358+0000 7fc8bfc24740 1 Processor -- start 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97a4e0 0x5644bd9a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd9997d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8bfc24740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 0x5644bd999d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5644bd997980 con 0x5644bd9856c0 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5644bd9972b0 con 0x5644bd97a4e0 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8bfc24740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5644bd998050 con 0x5644bd97ae00 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8bfc24740 10 client.0 _ll_register_callbacks cb 0x7fc8b00bcda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97a4e0 0x5644bd9a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd9997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97a4e0 0x5644bd9a04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:35954/0 (socket says 172.21.15.114:35954) 2024-09-19T06:51:14.363 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5e00640 1 -- 172.21.15.114:0/2216089863 learned_addr learned my addr 172.21.15.114:0/2216089863 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd9997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:53828/0 (socket says 172.21.15.114:53828) 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b6800640 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 0x5644bd999d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5e00640 1 -- 172.21.15.114:0/2216089863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 msgr2=0x5644bd9997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd9997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5e00640 1 -- 172.21.15.114:0/2216089863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 msgr2=0x5644bd999d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.359+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 0x5644bd999d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.360+0000 7fc8b5e00640 1 -- 172.21.15.114:0/2216089863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644bd99a4f0 con 0x5644bd97a4e0 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.360+0000 7fc8b5e00640 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97a4e0 0x5644bd9a04a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8009250 tx=0x7fc8a800f4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5aff765f07a7339 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.360+0000 7fc897e00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.360+0000 7fc897e00640 1 -- 172.21.15.114:0/2216089863 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a801e040 con 0x5644bd97a4e0 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.360+0000 7fc897e00640 1 -- 172.21.15.114:0/2216089863 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fc8a8009e40 con 0x5644bd97a4e0 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.360+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2216089863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8a8029030 con 0x5644bd97a4e0 2024-09-19T06:51:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.361+0000 7fc897e00640 1 -- 172.21.15.114:0/2216089863 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc8a8005080 con 0x5644bd97a4e0 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.361+0000 7fc8bfc24740 10 client.24971 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.361+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2216089863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5644bda00da0 con 0x5644bd97a4e0 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc897e00640 1 -- 172.21.15.114:0/2216089863 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8a8029030 con 0x5644bd97a4e0 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc897e00640 1 -- 172.21.15.114:0/2216089863 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc8a80035f0 con 0x5644bd97a4e0 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc8bfc24740 10 client.24971 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc8bfc24740 10 client.24971 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc8bfc24740 20 client.24971 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x5644bda052a0 0x5644bda03fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc8bfc24740 4 client.24971 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2216089863 --> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] -- command(tid 0: {"prefix": "status"}) -- 0x5644bd8e7b30 con 0x5644bda052a0 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.362+0000 7fc8b5400640 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x5644bda052a0 0x5644bda03fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc897e00640 10 client.24971 ms_handle_connect on v2:172.21.15.116:6836/2673389539 2024-09-19T06:51:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8b5400640 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x5644bda052a0 0x5644bda03fb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=943cac18e59cedd3 server_cookie=a350082b29531223 in_seq=0 out_seq=0 2024-09-19T06:51:14.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc897e00640 1 -- 172.21.15.114:0/2216089863 <== mds.4 v2:172.21.15.116:6836/2673389539 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5644bd8e7b30 con 0x5644bda052a0 2024-09-19T06:51:14.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc897e00640 10 client.24971 handle_command_reply: tid=0 2024-09-19T06:51:14.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 client.24971 shutdown 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2216089863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97a4e0 msgr2=0x5644bd9a04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97a4e0 0x5644bd9a04a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8009250 tx=0x7fc8a800f4d0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2216089863 shutdown_connections 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x5644bda052a0 0x5644bda03fb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644bd9856c0 0x5644bd999d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5644bd97ae00 0x5644bd9997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 --2- 172.21.15.114:0/2216089863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5644bd97a4e0 0x5644bd9a04a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.363+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2216089863 >> 172.21.15.114:0/2216089863 conn(0x5644bd976a70 msgr2=0x5644bd995110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.364+0000 7fc897e00640 0 client.24971 ms_handle_reset on v2:172.21.15.116:6836/2673389539 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.364+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2216089863 shutdown_connections 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.364+0000 7fc8bfc24740 1 -- 172.21.15.114:0/2216089863 wait complete. 2024-09-19T06:51:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.364+0000 7fc8bfc24740 20 client.24971 trim_cache size 0 max 16384 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 4, 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24482, 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 76.128600101000004, 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 149.083206561 2024-09-19T06:51:14.368 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 -- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc89c0a3350 msgr2=0x7fc89c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc89c0a3350 0x7fc89c0a5810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5644bd977df0 tx=0x7fc8a400b410 comp rx=0 tx=0).stop 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 -- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 msgr2=0x7fc8b819e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 0x7fc8b819e710 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc8b002aeb0 tx=0x7fc8b0003f10 comp rx=0 tx=0).stop 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 -- 172.21.15.114:0/569898993 shutdown_connections 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fc89c0a3350 0x7fc89c0a5810 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8b8112870 0x7fc8b819e710 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8b8077040 0x7fc8b81a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 --2- 172.21.15.114:0/569898993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8b8076720 0x7fc8b81a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:14.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 -- 172.21.15.114:0/569898993 >> 172.21.15.114:0/569898993 conn(0x7fc8b80ff8d0 msgr2=0x7fc8b8101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:14.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 -- 172.21.15.114:0/569898993 shutdown_connections 2024-09-19T06:51:14.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:14.365+0000 7fc8b7200640 1 -- 172.21.15.114:0/569898993 wait complete. 2024-09-19T06:51:15.206 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-19T06:51:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:15 smithi114 ceph-mon[30820]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:51:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:51:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:15.580 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:15 smithi116 ceph-mon[35411]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:51:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:51:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:15 smithi118 ceph-mon[35527]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:51:15.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:51:15.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:15 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:51:16.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 -- 172.21.15.114:0/3481088454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 msgr2=0x7fec6c110650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 --2- 172.21.15.114:0/3481088454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 0x7fec6c110650 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fec64009900 tx=0x7fec6402a480 comp rx=0 tx=0).stop 2024-09-19T06:51:16.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 -- 172.21.15.114:0/3481088454 shutdown_connections 2024-09-19T06:51:16.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 --2- 172.21.15.114:0/3481088454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 0x7fec6c110650 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 --2- 172.21.15.114:0/3481088454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c074ee0 0x7fec6c10dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 --2- 172.21.15.114:0/3481088454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec6c0745c0 0x7fec6c0749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 -- 172.21.15.114:0/3481088454 >> 172.21.15.114:0/3481088454 conn(0x7fec6c0fd740 msgr2=0x7fec6c0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 -- 172.21.15.114:0/3481088454 shutdown_connections 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.164+0000 7fec70e00640 1 -- 172.21.15.114:0/3481088454 wait complete. 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.165+0000 7fec70e00640 1 Processor -- start 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.165+0000 7fec70e00640 1 -- start start 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.165+0000 7fec70e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec6c0745c0 0x7fec6c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.165+0000 7fec70e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c074ee0 0x7fec6c19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.165+0000 7fec70e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 0x7fec6c19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.165+0000 7fec70e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c113fb0 con 0x7fec6c10e260 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.165+0000 7fec70e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c113e30 con 0x7fec6c074ee0 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.165+0000 7fec70e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c114130 con 0x7fec6c0745c0 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 0x7fec6c19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 0x7fec6c19ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38618/0 (socket says 172.21.15.114:38618) 2024-09-19T06:51:16.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec6c0745c0 0x7fec6c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 -- 172.21.15.114:0/1207791108 learned_addr learned my addr 172.21.15.114:0/1207791108 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec62a00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c074ee0 0x7fec6c19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 -- 172.21.15.114:0/1207791108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec6c0745c0 msgr2=0x7fec6c19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec6c0745c0 0x7fec6c19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 -- 172.21.15.114:0/1207791108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c074ee0 msgr2=0x7fec6c19a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c074ee0 0x7fec6c19a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 -- 172.21.15.114:0/1207791108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6c19f420 con 0x7fec6c10e260 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec62a00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c074ee0 0x7fec6c19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6b400640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec6c0745c0 0x7fec6c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.166+0000 7fec6be00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 0x7fec6c19ed40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fec64005ab0 tx=0x7fec64031b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:16.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.167+0000 7fec68c00640 1 -- 172.21.15.114:0/1207791108 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6403f030 con 0x7fec6c10e260 2024-09-19T06:51:16.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.167+0000 7fec68c00640 1 -- 172.21.15.114:0/1207791108 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec640043d0 con 0x7fec6c10e260 2024-09-19T06:51:16.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.167+0000 7fec70e00640 1 -- 172.21.15.114:0/1207791108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec6c19f6b0 con 0x7fec6c10e260 2024-09-19T06:51:16.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.167+0000 7fec68c00640 1 -- 172.21.15.114:0/1207791108 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec64035070 con 0x7fec6c10e260 2024-09-19T06:51:16.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.167+0000 7fec70e00640 1 -- 172.21.15.114:0/1207791108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec6c1a6ff0 con 0x7fec6c10e260 2024-09-19T06:51:16.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.170+0000 7fec68c00640 1 -- 172.21.15.114:0/1207791108 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7fec64042030 con 0x7fec6c10e260 2024-09-19T06:51:16.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.170+0000 7fec74ec0740 1 Processor -- start 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.170+0000 7fec74ec0740 1 -- start start 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a124d4e0 0x5584a124d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1258180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a12586c0 0x5584a125aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec74ec0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584a11b30b0 con 0x5584a124d4e0 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec74ec0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5584a115aab0 con 0x5584a124de00 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec74ec0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5584a11af070 con 0x5584a12586c0 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a124d4e0 0x5584a124d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec6b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1258180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a124d4e0 0x5584a124d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38620/0 (socket says 172.21.15.114:38620) 2024-09-19T06:51:16.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec62a00640 1 -- 172.21.15.114:0/2337096513 learned_addr learned my addr 172.21.15.114:0/2337096513 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec68c00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec500a32d0 0x7fec500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.171+0000 7fec62a00640 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a12586c0 0x5584a125aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec68c00640 1 -- 172.21.15.114:0/1207791108 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fec640f01a0 con 0x7fec6c10e260 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec6be00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec500a32d0 0x7fec500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec6b400640 1 -- 172.21.15.114:0/2337096513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a12586c0 msgr2=0x5584a125aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec6b400640 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a12586c0 0x5584a125aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec6b400640 1 -- 172.21.15.114:0/2337096513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a124d4e0 msgr2=0x5584a124d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec6b400640 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a124d4e0 0x5584a124d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec6b400640 1 -- 172.21.15.114:0/2337096513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584a11674a0 con 0x5584a124de00 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec62a00640 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a12586c0 0x5584a125aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec6b400640 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1258180 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fec5c005fd0 tx=0x7fec5c011c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5859ce317eaee20 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec6be00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec500a32d0 0x7fec500a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fec64032c00 tx=0x7fec64005780 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:16.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec60c00640 1 -- 172.21.15.114:0/2337096513 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c004d20 con 0x5584a124de00 2024-09-19T06:51:16.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec60c00640 1 -- 172.21.15.114:0/2337096513 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec5c00f9a0 con 0x5584a124de00 2024-09-19T06:51:16.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.172+0000 7fec60c00640 1 -- 172.21.15.114:0/2337096513 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c004490 con 0x5584a124de00 2024-09-19T06:51:16.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 -- 172.21.15.114:0/2337096513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 msgr2=0x5584a1258180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1258180 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fec5c005fd0 tx=0x7fec5c011c70 comp rx=0 tx=0).stop 2024-09-19T06:51:16.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 -- 172.21.15.114:0/2337096513 shutdown_connections 2024-09-19T06:51:16.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a12586c0 0x5584a125aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1258180 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2337096513 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a124d4e0 0x5584a124d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 -- 172.21.15.114:0/2337096513 >> 172.21.15.114:0/2337096513 conn(0x5584a1249a70 msgr2=0x5584a1249e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 -- 172.21.15.114:0/2337096513 shutdown_connections 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.173+0000 7fec74ec0740 1 -- 172.21.15.114:0/2337096513 wait complete. 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.174+0000 7fec74ec0740 1 -- start start 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.174+0000 7fec74ec0740 1 Processor -- start 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.174+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a1255540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.174+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1255a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.174+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1255fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.174+0000 7fec74ec0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584a125b330 con 0x5584a12586c0 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.174+0000 7fec74ec0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5584a113b920 con 0x5584a124de00 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.174+0000 7fec74ec0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5584a125b4b0 con 0x5584a124d4e0 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec74ec0740 10 client.0 _ll_register_callbacks cb 0x5584a115f8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1255fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec6b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1255a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec6b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1255a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:35984/0 (socket says 172.21.15.114:35984) 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1255fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38632/0 (socket says 172.21.15.114:38632) 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec6b400640 1 -- 172.21.15.114:0/640810684 learned_addr learned my addr 172.21.15.114:0/640810684 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec62a00640 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a1255540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec6b400640 1 -- 172.21.15.114:0/640810684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 msgr2=0x5584a1255540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec6b400640 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a1255540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec6b400640 1 -- 172.21.15.114:0/640810684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 msgr2=0x5584a1255fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec6b400640 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1255fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.175+0000 7fec6b400640 1 -- 172.21.15.114:0/640810684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584a0f242b0 con 0x5584a124de00 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec6b400640 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1255a80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00a010 tx=0x7fec5c003c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1430cecacfedfe0b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec59600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec59600640 1 -- 172.21.15.114:0/640810684 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c005000 con 0x5584a124de00 2024-09-19T06:51:16.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec74ec0740 1 -- 172.21.15.114:0/640810684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec5c00f9a0 con 0x5584a124de00 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec59600640 1 -- 172.21.15.114:0/640810684 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fec5c0051a0 con 0x5584a124de00 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec59600640 1 -- 172.21.15.114:0/640810684 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fec5c032050 con 0x5584a124de00 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec74ec0740 10 client.24983 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec74ec0740 1 -- 172.21.15.114:0/640810684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5584a1266f70 con 0x5584a124de00 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.176+0000 7fec59600640 1 -- 172.21.15.114:0/640810684 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec5c00f9a0 con 0x5584a124de00 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.177+0000 7fec59600640 1 -- 172.21.15.114:0/640810684 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fec5c018070 con 0x5584a124de00 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.177+0000 7fec74ec0740 10 client.24983 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.177+0000 7fec74ec0740 10 client.24983 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.177+0000 7fec74ec0740 20 client.24983 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.177+0000 7fec74ec0740 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x5584a1269330 0x5584a126a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.177+0000 7fec74ec0740 4 client.24983 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:16.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.177+0000 7fec74ec0740 1 -- 172.21.15.114:0/640810684 --> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584a11c0620 con 0x5584a1269330 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.177+0000 7fec62a00640 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x5584a1269330 0x5584a126a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.178+0000 7fec62a00640 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x5584a1269330 0x5584a126a7a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=260dffa529ddff07 server_cookie=93468a4a2b2a3a00 in_seq=0 out_seq=0 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.178+0000 7fec59600640 10 client.24983 ms_handle_connect on v2:172.21.15.118:6834/449495659 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec59600640 1 -- 172.21.15.114:0/640810684 <== mds.3 v2:172.21.15.118:6834/449495659 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5584a11c0620 con 0x5584a1269330 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec59600640 10 client.24983 handle_command_reply: tid=0 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 client.24983 shutdown 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 -- 172.21.15.114:0/640810684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 msgr2=0x5584a1255a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1255a80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00a010 tx=0x7fec5c003c60 comp rx=0 tx=0).stop 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 -- 172.21.15.114:0/640810684 shutdown_connections 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1255fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x5584a1269330 0x5584a126a7a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1255a80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 --2- 172.21.15.114:0/640810684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a1255540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec59600640 0 client.24983 ms_handle_reset on v2:172.21.15.118:6834/449495659 2024-09-19T06:51:16.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 -- 172.21.15.114:0/640810684 >> 172.21.15.114:0/640810684 conn(0x5584a1249a70 msgr2=0x5584a1248bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:16.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 -- 172.21.15.114:0/640810684 shutdown_connections 2024-09-19T06:51:16.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.179+0000 7fec74ec0740 1 -- 172.21.15.114:0/640810684 wait complete. 2024-09-19T06:51:16.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.180+0000 7fec74ec0740 20 client.24983 trim_cache size 0 max 16384 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.198+0000 7fec74ec0740 1 Processor -- start 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec74ec0740 1 -- start start 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124d4e0 0x5584a126a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124de00 0x5584a1255e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a12527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec74ec0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584a1268ae0 con 0x5584a12586c0 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec74ec0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5584a12691b0 con 0x5584a124d4e0 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec74ec0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5584a1269880 con 0x5584a124de00 2024-09-19T06:51:16.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a12527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a12527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38638/0 (socket says 172.21.15.114:38638) 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec62a00640 1 -- 172.21.15.114:0/3948425678 learned_addr learned my addr 172.21.15.114:0/3948425678 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec6b400640 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124de00 0x5584a1255e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.199+0000 7fec62a00640 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124d4e0 0x5584a126a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec62a00640 1 -- 172.21.15.114:0/3948425678 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124de00 msgr2=0x5584a1255e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec62a00640 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124de00 0x5584a1255e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec62a00640 1 -- 172.21.15.114:0/3948425678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124d4e0 msgr2=0x5584a126a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec62a00640 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124d4e0 0x5584a126a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec62a00640 1 -- 172.21.15.114:0/3948425678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584a124a740 con 0x5584a12586c0 2024-09-19T06:51:16.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec62a00640 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a12527c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fec54005890 tx=0x7fec540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500516d3bc282c50 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec5be00640 1 -- 172.21.15.114:0/3948425678 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5401c040 con 0x5584a12586c0 2024-09-19T06:51:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec5be00640 1 -- 172.21.15.114:0/3948425678 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec54003bc0 con 0x5584a12586c0 2024-09-19T06:51:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.200+0000 7fec5be00640 1 -- 172.21.15.114:0/3948425678 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54010430 con 0x5584a12586c0 2024-09-19T06:51:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 -- 172.21.15.114:0/3948425678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 msgr2=0x5584a12527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a12527c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fec54005890 tx=0x7fec540046c0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 -- 172.21.15.114:0/3948425678 shutdown_connections 2024-09-19T06:51:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a12527c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124de00 0x5584a1255e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 --2- 172.21.15.114:0/3948425678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124d4e0 0x5584a126a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 -- 172.21.15.114:0/3948425678 >> 172.21.15.114:0/3948425678 conn(0x5584a1249a70 msgr2=0x5584a1254630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 -- 172.21.15.114:0/3948425678 shutdown_connections 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 -- 172.21.15.114:0/3948425678 wait complete. 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.201+0000 7fec74ec0740 1 -- start start 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.202+0000 7fec74ec0740 1 Processor -- start 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.202+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a12526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.202+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1252be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.202+0000 7fec74ec0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1253120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.202+0000 7fec74ec0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584a126a620 con 0x5584a12586c0 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.202+0000 7fec74ec0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5584a1269f50 con 0x5584a124de00 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.202+0000 7fec74ec0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5584a126acf0 con 0x5584a124d4e0 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.202+0000 7fec74ec0740 10 client.0 _ll_register_callbacks cb 0x7fec5c00ff10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:16.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1253120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1252be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1252be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:35996/0 (socket says 172.21.15.114:35996) 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 -- 172.21.15.114:0/2581542443 learned_addr learned my addr 172.21.15.114:0/2581542443 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec62a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1253120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38642/0 (socket says 172.21.15.114:38642) 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec62a00640 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a12526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 -- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 msgr2=0x5584a12526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a12526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 -- 172.21.15.114:0/2581542443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 msgr2=0x5584a1253120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1253120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 -- 172.21.15.114:0/2581542443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584a1270f80 con 0x5584a124de00 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec62a00640 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a12526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec6b400640 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1252be0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fec5c006010 tx=0x7fec5c00fad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f49f22c924d0b628 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:16.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec59600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:16.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.203+0000 7fec59600640 1 -- 172.21.15.114:0/2581542443 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c010de0 con 0x5584a124de00 2024-09-19T06:51:16.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec59600640 1 -- 172.21.15.114:0/2581542443 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fec5c010430 con 0x5584a124de00 2024-09-19T06:51:16.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec74ec0740 1 -- 172.21.15.114:0/2581542443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec54003bc0 con 0x5584a124de00 2024-09-19T06:51:16.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec59600640 1 -- 172.21.15.114:0/2581542443 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fec5c010630 con 0x5584a124de00 2024-09-19T06:51:16.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec74ec0740 10 client.24989 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:16.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec74ec0740 1 -- 172.21.15.114:0/2581542443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5584a12d3cd0 con 0x5584a124de00 2024-09-19T06:51:16.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec59600640 1 -- 172.21.15.114:0/2581542443 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec54003bc0 con 0x5584a124de00 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec59600640 1 -- 172.21.15.114:0/2581542443 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fec5c018040 con 0x5584a124de00 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec74ec0740 10 client.24989 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec74ec0740 10 client.24989 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec74ec0740 20 client.24989 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.204+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x5584a12d87c0 0x5584a12d74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.205+0000 7fec74ec0740 4 client.24989 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.205+0000 7fec74ec0740 1 -- 172.21.15.114:0/2581542443 --> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] -- command(tid 0: {"prefix": "status"}) -- 0x5584a11bab30 con 0x5584a12d87c0 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.205+0000 7fec62a00640 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x5584a12d87c0 0x5584a12d74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.205+0000 7fec62a00640 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x5584a12d87c0 0x5584a12d74d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6f284403bc732dc6 server_cookie=d8660b1fc53405b6 in_seq=0 out_seq=0 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.205+0000 7fec59600640 10 client.24989 ms_handle_connect on v2:172.21.15.118:6834/449495659 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec59600640 1 -- 172.21.15.114:0/2581542443 <== mds.3 v2:172.21.15.118:6834/449495659 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5584a11bab30 con 0x5584a12d87c0 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec59600640 10 client.24989 handle_command_reply: tid=0 2024-09-19T06:51:16.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 client.24989 shutdown 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 -- 172.21.15.114:0/2581542443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 msgr2=0x5584a1252be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1252be0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fec5c006010 tx=0x7fec5c00fad0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 -- 172.21.15.114:0/2581542443 shutdown_connections 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584a12586c0 0x5584a1253120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x5584a12d87c0 0x5584a12d74d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5584a124de00 0x5584a1252be0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec59600640 0 client.24989 ms_handle_reset on v2:172.21.15.118:6834/449495659 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 --2- 172.21.15.114:0/2581542443 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5584a124d4e0 0x5584a12526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 -- 172.21.15.114:0/2581542443 >> 172.21.15.114:0/2581542443 conn(0x5584a1249a70 msgr2=0x5584a124ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 -- 172.21.15.114:0/2581542443 shutdown_connections 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 1 -- 172.21.15.114:0/2581542443 wait complete. 2024-09-19T06:51:16.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.206+0000 7fec74ec0740 20 client.24989 trim_cache size 0 max 16384 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 3, 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24508, 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 78.981481048999996, 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:16.212 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:16.213 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:16.213 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 156.24134691399999 2024-09-19T06:51:16.213 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:16.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 -- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec500a32d0 msgr2=0x7fec500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec500a32d0 0x7fec500a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fec64032c00 tx=0x7fec64005780 comp rx=0 tx=0).stop 2024-09-19T06:51:16.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 -- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 msgr2=0x7fec6c19ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:16.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 0x7fec6c19ed40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fec64005ab0 tx=0x7fec64031b50 comp rx=0 tx=0).stop 2024-09-19T06:51:16.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 -- 172.21.15.114:0/1207791108 shutdown_connections 2024-09-19T06:51:16.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fec500a32d0 0x7fec500a5790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec6c10e260 0x7fec6c19ed40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6c074ee0 0x7fec6c19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 --2- 172.21.15.114:0/1207791108 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec6c0745c0 0x7fec6c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:16.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 -- 172.21.15.114:0/1207791108 >> 172.21.15.114:0/1207791108 conn(0x7fec6c0fd740 msgr2=0x7fec6c0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:16.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 -- 172.21.15.114:0/1207791108 shutdown_connections 2024-09-19T06:51:16.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:16.207+0000 7fec70e00640 1 -- 172.21.15.114:0/1207791108 wait complete. 2024-09-19T06:51:17.063 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-19T06:51:17.432 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 -- 172.21.15.114:0/3910533062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 msgr2=0x7f59c4105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 --2- 172.21.15.114:0/3910533062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 0x7f59c4105f60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009900 tx=0x7f59b402a3f0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 -- 172.21.15.114:0/3910533062 shutdown_connections 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 --2- 172.21.15.114:0/3910533062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c4101fc0 0x7f59c410d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 --2- 172.21.15.114:0/3910533062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59c4101640 0x7f59c4101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 --2- 172.21.15.114:0/3910533062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 0x7f59c4105f60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 -- 172.21.15.114:0/3910533062 >> 172.21.15.114:0/3910533062 conn(0x7f59c40fba90 msgr2=0x7f59c40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 -- 172.21.15.114:0/3910533062 shutdown_connections 2024-09-19T06:51:18.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.021+0000 7f59ca400640 1 -- 172.21.15.114:0/3910533062 wait complete. 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.022+0000 7f59ca400640 1 Processor -- start 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.022+0000 7f59ca400640 1 -- start start 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.022+0000 7f59ca400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c4101640 0x7f59c419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59ca400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59c4101fc0 0x7f59c419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59ca400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 0x7f59c419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59ca400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f59c4110e30 con 0x7f59c4101640 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59ca400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59c4110cb0 con 0x7f59c4105b80 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59ca400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f59c4110fb0 con 0x7f59c4101fc0 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c4101640 0x7f59c419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c4101640 0x7f59c419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55282/0 (socket says 172.21.15.114:55282) 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3e00640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 0x7f59c419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3400640 1 -- 172.21.15.114:0/3625736956 learned_addr learned my addr 172.21.15.114:0/3625736956 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:18.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59baa00640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59c4101fc0 0x7f59c419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3e00640 1 -- 172.21.15.114:0/3625736956 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59c4101fc0 msgr2=0x7f59c419aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3e00640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59c4101fc0 0x7f59c419aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3e00640 1 -- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c4101640 msgr2=0x7f59c419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3e00640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c4101640 0x7f59c419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.023+0000 7f59c3e00640 1 -- 172.21.15.114:0/3625736956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c419f610 con 0x7f59c4105b80 2024-09-19T06:51:18.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.024+0000 7f59c3e00640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 0x7f59c419ef30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009fd0 tx=0x7f59bc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:18.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.024+0000 7f59c0c00640 1 -- 172.21.15.114:0/3625736956 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc00ce20 con 0x7f59c4105b80 2024-09-19T06:51:18.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.024+0000 7f59c0c00640 1 -- 172.21.15.114:0/3625736956 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59bc005020 con 0x7f59c4105b80 2024-09-19T06:51:18.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.024+0000 7f59ca400640 1 -- 172.21.15.114:0/3625736956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c419f900 con 0x7f59c4105b80 2024-09-19T06:51:18.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.024+0000 7f59c0c00640 1 -- 172.21.15.114:0/3625736956 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc010720 con 0x7f59c4105b80 2024-09-19T06:51:18.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.024+0000 7f59ca400640 1 -- 172.21.15.114:0/3625736956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c41a71e0 con 0x7f59c4105b80 2024-09-19T06:51:18.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.028+0000 7f59ccdfb740 1 Processor -- start 2024-09-19T06:51:18.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.028+0000 7f59c0c00640 1 -- 172.21.15.114:0/3625736956 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f59bc005540 con 0x7f59c4105b80 2024-09-19T06:51:18.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.028+0000 7f59ccdfb740 1 -- start start 2024-09-19T06:51:18.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8330 0x5557847d8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8c50 0x5557847e2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55578473e0b0 con 0x5557847d8330 2024-09-19T06:51:18.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5557846e5ab0 con 0x5557847e3510 2024-09-19T06:51:18.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55578473a070 con 0x5557847d8c50 2024-09-19T06:51:18.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59baa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8330 0x5557847d8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59c3400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8c50 0x5557847e2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59baa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8330 0x5557847d8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55288/0 (socket says 172.21.15.114:55288) 2024-09-19T06:51:18.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59c3400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8c50 0x5557847e2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:39442/0 (socket says 172.21.15.114:39442) 2024-09-19T06:51:18.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59baa00640 1 -- 172.21.15.114:0/222059658 learned_addr learned my addr 172.21.15.114:0/222059658 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.029+0000 7f59baa00640 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59c3400640 1 -- 172.21.15.114:0/222059658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 msgr2=0x5557847e5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59c3400640 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59c3400640 1 -- 172.21.15.114:0/222059658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8330 msgr2=0x5557847d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59c3400640 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8330 0x5557847d8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59c3400640 1 -- 172.21.15.114:0/222059658 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557846f24a0 con 0x5557847d8c50 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59baa00640 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8330 0x5557847d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59c0c00640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f59a80a32d0 0x7f59a80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59c3400640 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8c50 0x5557847e2fd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009230 tx=0x7f59b4005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d44b527b68fe7473 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59c0c00640 1 -- 172.21.15.114:0/3625736956 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f59bc0d0570 con 0x7f59c4105b80 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59baa00640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f59a80a32d0 0x7f59a80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.030+0000 7f59b8c00640 1 -- 172.21.15.114:0/222059658 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4040070 con 0x5557847d8c50 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59b8c00640 1 -- 172.21.15.114:0/222059658 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59b4004030 con 0x5557847d8c50 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59b8c00640 1 -- 172.21.15.114:0/222059658 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4035040 con 0x5557847d8c50 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59ccdfb740 1 -- 172.21.15.114:0/222059658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8c50 msgr2=0x5557847e2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59baa00640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f59a80a32d0 0x7f59a80a5790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x5557847d5c40 tx=0x7f59ac009850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8c50 0x5557847e2fd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009230 tx=0x7f59b4005dd0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59ccdfb740 1 -- 172.21.15.114:0/222059658 shutdown_connections 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8c50 0x5557847e2fd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/222059658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8330 0x5557847d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59ccdfb740 1 -- 172.21.15.114:0/222059658 >> 172.21.15.114:0/222059658 conn(0x5557847d48c0 msgr2=0x5557847d4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:18.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.031+0000 7f59ccdfb740 1 -- 172.21.15.114:0/222059658 shutdown_connections 2024-09-19T06:51:18.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.032+0000 7f59ccdfb740 1 -- 172.21.15.114:0/222059658 wait complete. 2024-09-19T06:51:18.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.032+0000 7f59ccdfb740 1 -- start start 2024-09-19T06:51:18.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.032+0000 7f59ccdfb740 1 Processor -- start 2024-09-19T06:51:18.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847e0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847e0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5557847e5fc0 con 0x5557847d8c50 2024-09-19T06:51:18.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5557846c6920 con 0x5557847e3510 2024-09-19T06:51:18.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5557847e6140 con 0x5557847d8330 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59ccdfb740 10 client.0 _ll_register_callbacks cb 0x5557846ea8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59baa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847e0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59baa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847e0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55290/0 (socket says 172.21.15.114:55290) 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59baa00640 1 -- 172.21.15.114:0/4119728867 learned_addr learned my addr 172.21.15.114:0/4119728867 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59c3400640 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847e0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59c3e00640 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59c3400640 1 -- 172.21.15.114:0/4119728867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 msgr2=0x5557847e1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59c3400640 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.033+0000 7f59c3400640 1 -- 172.21.15.114:0/4119728867 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 msgr2=0x5557847e0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59c3400640 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847e0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59c3400640 1 -- 172.21.15.114:0/4119728867 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557844af2b0 con 0x5557847d8330 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59baa00640 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847e0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59c3400640 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847e0610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009230 tx=0x7f59b40041e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a98b9d2f73d5efc server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:18.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59b1600640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59b1600640 1 -- 172.21.15.114:0/4119728867 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b404c070 con 0x5557847d8330 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59ccdfb740 1 -- 172.21.15.114:0/4119728867 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59b4004030 con 0x5557847d8330 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59b1600640 1 -- 172.21.15.114:0/4119728867 <== mon.1 v2:172.21.15.118:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f59b40317b0 con 0x5557847d8330 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59b1600640 1 -- 172.21.15.114:0/4119728867 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f59b4058050 con 0x5557847d8330 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59ccdfb740 10 client.25048 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.034+0000 7f59ccdfb740 1 -- 172.21.15.114:0/4119728867 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5557847e6010 con 0x5557847d8330 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.035+0000 7f59b1600640 1 -- 172.21.15.114:0/4119728867 <== mon.1 v2:172.21.15.118:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59b4004030 con 0x5557847d8330 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.035+0000 7f59b1600640 1 -- 172.21.15.114:0/4119728867 <== mon.1 v2:172.21.15.118:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f59b4003b40 con 0x5557847d8330 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.035+0000 7f59ccdfb740 10 client.25048 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.035+0000 7f59ccdfb740 10 client.25048 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.035+0000 7f59ccdfb740 20 client.25048 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:18.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.035+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5557847f3fd0 0x5557847f5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.035+0000 7f59ccdfb740 4 client.25048 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:18.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.035+0000 7f59ccdfb740 1 -- 172.21.15.114:0/4119728867 --> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55578474b620 con 0x5557847f3fd0 2024-09-19T06:51:18.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.036+0000 7f59baa00640 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5557847f3fd0 0x5557847f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.036+0000 7f59baa00640 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5557847f3fd0 0x5557847f5440 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=785394920d316bdf server_cookie=4d8f8330db9f5552 in_seq=0 out_seq=0 2024-09-19T06:51:18.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.036+0000 7f59b1600640 10 client.25048 ms_handle_connect on v2:172.21.15.118:6836/4109230511 2024-09-19T06:51:18.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59b1600640 1 -- 172.21.15.114:0/4119728867 <== mds.2 v2:172.21.15.118:6836/4109230511 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55578474b620 con 0x5557847f3fd0 2024-09-19T06:51:18.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59b1600640 10 client.25048 handle_command_reply: tid=0 2024-09-19T06:51:18.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 client.25048 shutdown 2024-09-19T06:51:18.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 -- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 msgr2=0x5557847e0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847e0610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009230 tx=0x7f59b40041e0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 -- 172.21.15.114:0/4119728867 shutdown_connections 2024-09-19T06:51:18.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5557847f3fd0 0x5557847f5440 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847e0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/4119728867 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847e0610 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59b1600640 0 client.25048 ms_handle_reset on v2:172.21.15.118:6836/4109230511 2024-09-19T06:51:18.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 -- 172.21.15.114:0/4119728867 >> 172.21.15.114:0/4119728867 conn(0x5557847d48c0 msgr2=0x5557847da5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:18.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.037+0000 7f59ccdfb740 1 -- 172.21.15.114:0/4119728867 shutdown_connections 2024-09-19T06:51:18.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.038+0000 7f59ccdfb740 1 -- 172.21.15.114:0/4119728867 wait complete. 2024-09-19T06:51:18.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.038+0000 7f59ccdfb740 20 client.25048 trim_cache size 0 max 16384 2024-09-19T06:51:18.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59ccdfb740 1 Processor -- start 2024-09-19T06:51:18.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59ccdfb740 1 -- start start 2024-09-19T06:51:18.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847f3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5557847f3780 con 0x5557847d8c50 2024-09-19T06:51:18.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5557847f3e50 con 0x5557847e3510 2024-09-19T06:51:18.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5557847f4520 con 0x5557847d8330 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59baa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59baa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55296/0 (socket says 172.21.15.114:55296) 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.056+0000 7f59baa00640 1 -- 172.21.15.114:0/3423208448 learned_addr learned my addr 172.21.15.114:0/3423208448 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59c3400640 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59c3e00640 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59baa00640 1 -- 172.21.15.114:0/3423208448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 msgr2=0x5557847f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59baa00640 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59baa00640 1 -- 172.21.15.114:0/3423208448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 msgr2=0x5557847e0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59baa00640 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59baa00640 1 -- 172.21.15.114:0/3423208448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557847df600 con 0x5557847d8c50 2024-09-19T06:51:18.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59baa00640 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dad90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x5557847e1bc0 tx=0x7f59ac015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ac9a966500683b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:18.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59b3e00640 1 -- 172.21.15.114:0/3423208448 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac022410 con 0x5557847d8c50 2024-09-19T06:51:18.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59b3e00640 1 -- 172.21.15.114:0/3423208448 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ac022a90 con 0x5557847d8c50 2024-09-19T06:51:18.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.057+0000 7f59b3e00640 1 -- 172.21.15.114:0/3423208448 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac02db20 con 0x5557847d8c50 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 -- 172.21.15.114:0/3423208448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 msgr2=0x5557847dad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dad90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x5557847e1bc0 tx=0x7f59ac015000 comp rx=0 tx=0).stop 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 -- 172.21.15.114:0/3423208448 shutdown_connections 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847e3510 0x5557847e0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dad90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/3423208448 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847d8330 0x5557847f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 -- 172.21.15.114:0/3423208448 >> 172.21.15.114:0/3423208448 conn(0x5557847d48c0 msgr2=0x5557847f2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 -- 172.21.15.114:0/3423208448 shutdown_connections 2024-09-19T06:51:18.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.058+0000 7f59ccdfb740 1 -- 172.21.15.114:0/3423208448 wait complete. 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 1 -- start start 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 1 Processor -- start 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847d8330 0x5557847dd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847e3510 0x5557847fc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5557847f52c0 con 0x5557847d8c50 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5557847f4bf0 con 0x5557847d8330 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5557847f5990 con 0x5557847e3510 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.059+0000 7f59ccdfb740 10 client.0 _ll_register_callbacks cb 0x7f59b4009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:18.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847d8330 0x5557847dd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59baa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847d8330 0x5557847dd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:47300/0 (socket says 172.21.15.114:47300) 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 -- 172.21.15.114:0/434037576 learned_addr learned my addr 172.21.15.114:0/434037576 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3e00640 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847e3510 0x5557847fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 -- 172.21.15.114:0/434037576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847e3510 msgr2=0x5557847fc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847e3510 0x5557847fc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 -- 172.21.15.114:0/434037576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 msgr2=0x5557847dd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 -- 172.21.15.114:0/434037576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557847fcbd0 con 0x5557847d8330 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59c3400640 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847d8330 0x5557847dd040 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f59b4005940 tx=0x7f59b40317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e94218fabfe7294f server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.060+0000 7f59b1600640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59b1600640 1 -- 172.21.15.114:0/434037576 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b404c050 con 0x5557847d8330 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59ccdfb740 1 -- 172.21.15.114:0/434037576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59ac022a90 con 0x5557847d8330 2024-09-19T06:51:18.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59b1600640 1 -- 172.21.15.114:0/434037576 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f59b40037f0 con 0x5557847d8330 2024-09-19T06:51:18.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59b1600640 1 -- 172.21.15.114:0/434037576 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f59b4040050 con 0x5557847d8330 2024-09-19T06:51:18.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59ccdfb740 10 client.25007 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:18.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59ccdfb740 1 -- 172.21.15.114:0/434037576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5557847fdeb0 con 0x5557847d8330 2024-09-19T06:51:18.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59b1600640 1 -- 172.21.15.114:0/434037576 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ac022a90 con 0x5557847d8330 2024-09-19T06:51:18.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59b1600640 1 -- 172.21.15.114:0/434037576 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f59b40518f0 con 0x5557847d8330 2024-09-19T06:51:18.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59ccdfb740 10 client.25007 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:18.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59ccdfb740 10 client.25007 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.061+0000 7f59ccdfb740 20 client.25007 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.062+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x555784862a30 0x555784861740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.062+0000 7f59ccdfb740 4 client.25007 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.062+0000 7f59ccdfb740 1 -- 172.21.15.114:0/434037576 --> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] -- command(tid 0: {"prefix": "status"}) -- 0x555784745b30 con 0x555784862a30 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.062+0000 7f59baa00640 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x555784862a30 0x555784861740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.062+0000 7f59baa00640 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x555784862a30 0x555784861740 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3eb32674cc77c06f server_cookie=cedc68851535aa1f in_seq=0 out_seq=0 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.062+0000 7f59b1600640 10 client.25007 ms_handle_connect on v2:172.21.15.118:6836/4109230511 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59b1600640 1 -- 172.21.15.114:0/434037576 <== mds.2 v2:172.21.15.118:6836/4109230511 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555784745b30 con 0x555784862a30 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59b1600640 10 client.25007 handle_command_reply: tid=0 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 client.25007 shutdown 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 -- 172.21.15.114:0/434037576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847d8330 msgr2=0x5557847dd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847d8330 0x5557847dd040 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f59b4005940 tx=0x7f59b40317e0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 -- 172.21.15.114:0/434037576 shutdown_connections 2024-09-19T06:51:18.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5557847e3510 0x5557847fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x555784862a30 0x555784861740 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5557847d8c50 0x5557847dd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 --2- 172.21.15.114:0/434037576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5557847d8330 0x5557847dd040 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59b1600640 0 client.25007 ms_handle_reset on v2:172.21.15.118:6836/4109230511 2024-09-19T06:51:18.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 -- 172.21.15.114:0/434037576 >> 172.21.15.114:0/434037576 conn(0x5557847d48c0 msgr2=0x5557847ee390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:18.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 -- 172.21.15.114:0/434037576 shutdown_connections 2024-09-19T06:51:18.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 1 -- 172.21.15.114:0/434037576 wait complete. 2024-09-19T06:51:18.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.063+0000 7f59ccdfb740 20 client.25007 trim_cache size 0 max 16384 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 2, 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24520, 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 81.847725535999999, 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 155.49316656600001 2024-09-19T06:51:18.069 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:18.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 -- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f59a80a32d0 msgr2=0x7f59a80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f59a80a32d0 0x7f59a80a5790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x5557847d5c40 tx=0x7f59ac009850 comp rx=0 tx=0).stop 2024-09-19T06:51:18.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 -- 172.21.15.114:0/3625736956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 msgr2=0x7f59c419ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:18.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 0x7f59c419ef30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009fd0 tx=0x7f59bc00ef10 comp rx=0 tx=0).stop 2024-09-19T06:51:18.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 -- 172.21.15.114:0/3625736956 shutdown_connections 2024-09-19T06:51:18.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f59a80a32d0 0x7f59a80a5790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59c4105b80 0x7f59c419ef30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59c4101fc0 0x7f59c419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 --2- 172.21.15.114:0/3625736956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f59c4101640 0x7f59c419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 -- 172.21.15.114:0/3625736956 >> 172.21.15.114:0/3625736956 conn(0x7f59c40fba90 msgr2=0x7f59c40feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 -- 172.21.15.114:0/3625736956 shutdown_connections 2024-09-19T06:51:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:18.064+0000 7f59ca400640 1 -- 172.21.15.114:0/3625736956 wait complete. 2024-09-19T06:51:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:18 smithi116 ceph-mon[35411]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:51:18.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:18 smithi118 ceph-mon[35527]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:51:18.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:18 smithi114 ceph-mon[30820]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:51:18.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-19T06:51:19.285 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:19.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.879+0000 7f61b7e00640 1 -- 172.21.15.114:0/4124098367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8108f60 msgr2=0x7f61b8110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.879+0000 7f61b7e00640 1 --2- 172.21.15.114:0/4124098367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8108f60 0x7f61b8110ae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009ca0 tx=0x7f61b002a8b0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.880+0000 7f61b7e00640 1 -- 172.21.15.114:0/4124098367 shutdown_connections 2024-09-19T06:51:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.880+0000 7f61b7e00640 1 --2- 172.21.15.114:0/4124098367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8108f60 0x7f61b8110ae0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.880+0000 7f61b7e00640 1 --2- 172.21.15.114:0/4124098367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61b81047e0 0x7f61b8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.880+0000 7f61b7e00640 1 --2- 172.21.15.114:0/4124098367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b8103e30 0x7f61b8104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.880+0000 7f61b7e00640 1 -- 172.21.15.114:0/4124098367 >> 172.21.15.114:0/4124098367 conn(0x7f61b80ff8d0 msgr2=0x7f61b8101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.880+0000 7f61b7e00640 1 -- 172.21.15.114:0/4124098367 shutdown_connections 2024-09-19T06:51:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.880+0000 7f61b7e00640 1 -- 172.21.15.114:0/4124098367 wait complete. 2024-09-19T06:51:19.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b7e00640 1 Processor -- start 2024-09-19T06:51:19.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b7e00640 1 -- start start 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8103e30 0x7f61b819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61b81047e0 0x7f61b819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b7e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b8108f60 0x7f61b819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8114010 con 0x7f61b8103e30 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b7e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8113e90 con 0x7f61b8108f60 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8114190 con 0x7f61b81047e0 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8103e30 0x7f61b819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8103e30 0x7f61b819a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55328/0 (socket says 172.21.15.114:55328) 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b6a00640 1 -- 172.21.15.114:0/2778374667 learned_addr learned my addr 172.21.15.114:0/2778374667 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.881+0000 7f61b6000640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61b81047e0 0x7f61b819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b7400640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b8108f60 0x7f61b819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b6a00640 1 -- 172.21.15.114:0/2778374667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61b81047e0 msgr2=0x7f61b819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b6a00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61b81047e0 0x7f61b819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b6a00640 1 -- 172.21.15.114:0/2778374667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b8108f60 msgr2=0x7f61b819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b6a00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b8108f60 0x7f61b819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b6a00640 1 -- 172.21.15.114:0/2778374667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b819f3c0 con 0x7f61b8103e30 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b7400640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b8108f60 0x7f61b819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b6a00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8103e30 0x7f61b819a410 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f61a800e9f0 tx=0x7f61a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61af400640 1 -- 172.21.15.114:0/2778374667 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a800cd80 con 0x7f61b8103e30 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61af400640 1 -- 172.21.15.114:0/2778374667 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61a8010910 con 0x7f61b8103e30 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b7e00640 1 -- 172.21.15.114:0/2778374667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b819f6b0 con 0x7f61b8103e30 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61af400640 1 -- 172.21.15.114:0/2778374667 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a8018990 con 0x7f61b8103e30 2024-09-19T06:51:19.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.882+0000 7f61b7e00640 1 -- 172.21.15.114:0/2778374667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b81a6f90 con 0x7f61b8103e30 2024-09-19T06:51:19.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61af400640 1 -- 172.21.15.114:0/2778374667 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f61a8010ac0 con 0x7f61b8103e30 2024-09-19T06:51:19.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61c0601740 1 Processor -- start 2024-09-19T06:51:19.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61c0601740 1 -- start start 2024-09-19T06:51:19.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61c0601740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f91b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61c0601740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f926180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61c0601740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f928ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61c0601740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f29f8810b0 con 0x55f29f91be00 2024-09-19T06:51:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61c0601740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f29f828ab0 con 0x55f29f91b4e0 2024-09-19T06:51:19.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.886+0000 7f61c0601740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f29f87d070 con 0x55f29f9266c0 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f926180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f91b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f91b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:47332/0 (socket says 172.21.15.114:47332) 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 -- 172.21.15.114:0/1427962885 learned_addr learned my addr 172.21.15.114:0/1427962885 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b6000640 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f928ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 -- 172.21.15.114:0/1427962885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 msgr2=0x55f29f928ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f928ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 -- 172.21.15.114:0/1427962885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 msgr2=0x55f29f926180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f926180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 -- 172.21.15.114:0/1427962885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29f8354a0 con 0x55f29f91b4e0 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b6000640 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f928ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:51:19.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.887+0000 7f61b7400640 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f91b8c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f61b0035000 tx=0x7f61b0009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cce4a760bbc2260d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.888+0000 7f61af400640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f619c0a3350 0x7f619c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.888+0000 7f61a3e00640 1 -- 172.21.15.114:0/1427962885 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0030750 con 0x55f29f91b4e0 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.888+0000 7f61a3e00640 1 -- 172.21.15.114:0/1427962885 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b0040070 con 0x55f29f91b4e0 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.888+0000 7f61a3e00640 1 -- 172.21.15.114:0/1427962885 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b00043a0 con 0x55f29f91b4e0 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.888+0000 7f61af400640 1 -- 172.21.15.114:0/2778374667 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f61a80d0210 con 0x7f61b8103e30 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.888+0000 7f61b6a00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f619c0a3350 0x7f619c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 -- 172.21.15.114:0/1427962885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 msgr2=0x55f29f91b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f91b8c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f61b0035000 tx=0x7f61b0009a20 comp rx=0 tx=0).stop 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61b6a00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f619c0a3350 0x7f619c0a5810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f61a80986e0 tx=0x7f61a80d3cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:19.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 -- 172.21.15.114:0/1427962885 shutdown_connections 2024-09-19T06:51:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f928ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f926180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 --2- 172.21.15.114:0/1427962885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f91b8c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 -- 172.21.15.114:0/1427962885 >> 172.21.15.114:0/1427962885 conn(0x55f29f917a70 msgr2=0x55f29f917e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 -- 172.21.15.114:0/1427962885 shutdown_connections 2024-09-19T06:51:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 -- 172.21.15.114:0/1427962885 wait complete. 2024-09-19T06:51:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.889+0000 7f61c0601740 1 -- start start 2024-09-19T06:51:19.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61c0601740 1 Processor -- start 2024-09-19T06:51:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61c0601740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f9235f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61c0601740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f923b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61c0601740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f924070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61c0601740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f29f929330 con 0x55f29f91be00 2024-09-19T06:51:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61c0601740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f29f809920 con 0x55f29f91b4e0 2024-09-19T06:51:19.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61c0601740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f29f9294b0 con 0x55f29f9266c0 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61c0601740 10 client.0 _ll_register_callbacks cb 0x55f29f82d8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61b6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f923b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61b6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f923b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55338/0 (socket says 172.21.15.114:55338) 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.890+0000 7f61b6000640 1 -- 172.21.15.114:0/2629593019 learned_addr learned my addr 172.21.15.114:0/2629593019 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61b7400640 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f9235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61b7400640 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f924070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61b6000640 1 -- 172.21.15.114:0/2629593019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 msgr2=0x55f29f924070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61b6000640 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f924070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61b6000640 1 -- 172.21.15.114:0/2629593019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 msgr2=0x55f29f9235f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61b6000640 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f9235f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61b6000640 1 -- 172.21.15.114:0/2629593019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29f5f22b0 con 0x55f29f91be00 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61b6000640 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f923b30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f61a4012730 tx=0x7f61a4012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2082b5844f0e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-09-19T06:51:19.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.891+0000 7f61a0c00640 1 -- 172.21.15.114:0/2629593019 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a400fa60 con 0x55f29f91be00 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.892+0000 7f61a0c00640 1 -- 172.21.15.114:0/2629593019 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f61a400fc00 con 0x55f29f91be00 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.892+0000 7f61c0601740 1 -- 172.21.15.114:0/2629593019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b0040070 con 0x55f29f91be00 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.892+0000 7f61a0c00640 1 -- 172.21.15.114:0/2629593019 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61a4024050 con 0x55f29f91be00 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.892+0000 7f61c0601740 10 client.15024 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.892+0000 7f61c0601740 1 -- 172.21.15.114:0/2629593019 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f29f9350e0 con 0x55f29f91be00 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.892+0000 7f61a0c00640 1 -- 172.21.15.114:0/2629593019 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b0040070 con 0x55f29f91be00 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.892+0000 7f61a0c00640 1 -- 172.21.15.114:0/2629593019 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f61a4010430 con 0x55f29f91be00 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.893+0000 7f61c0601740 10 client.15024 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.893+0000 7f61c0601740 10 client.15024 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.893+0000 7f61c0601740 20 client.15024 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.893+0000 7f61c0601740 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55f29f937790 0x55f29f938c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.893+0000 7f61c0601740 4 client.15024 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.893+0000 7f61c0601740 1 -- 172.21.15.114:0/2629593019 --> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f29f88e620 con 0x55f29f937790 2024-09-19T06:51:19.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.893+0000 7f61b7400640 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55f29f937790 0x55f29f938c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.894+0000 7f61b7400640 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55f29f937790 0x55f29f938c00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=89f45be46baad69a server_cookie=96d10d9212620872 in_seq=0 out_seq=0 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.894+0000 7f61a0c00640 10 client.15024 ms_handle_connect on v2:172.21.15.116:6838/2386737584 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.894+0000 7f61a0c00640 1 -- 172.21.15.114:0/2629593019 <== mds.1 v2:172.21.15.116:6838/2386737584 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f29f88e620 con 0x55f29f937790 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.894+0000 7f61a0c00640 10 client.15024 handle_command_reply: tid=0 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.894+0000 7f61c0601740 1 client.15024 shutdown 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 -- 172.21.15.114:0/2629593019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 msgr2=0x55f29f923b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f923b30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f61a4012730 tx=0x7f61a4012bf0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 -- 172.21.15.114:0/2629593019 shutdown_connections 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f924070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f923b30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55f29f937790 0x55f29f938c00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 --2- 172.21.15.114:0/2629593019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91b4e0 0x55f29f9235f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 -- 172.21.15.114:0/2629593019 >> 172.21.15.114:0/2629593019 conn(0x55f29f917a70 msgr2=0x55f29f916bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:19.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61a0c00640 0 client.15024 ms_handle_reset on v2:172.21.15.116:6838/2386737584 2024-09-19T06:51:19.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 -- 172.21.15.114:0/2629593019 shutdown_connections 2024-09-19T06:51:19.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 1 -- 172.21.15.114:0/2629593019 wait complete. 2024-09-19T06:51:19.899 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.895+0000 7f61c0601740 20 client.15024 trim_cache size 0 max 16384 2024-09-19T06:51:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.913+0000 7f61c0601740 1 Processor -- start 2024-09-19T06:51:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.913+0000 7f61c0601740 1 -- start start 2024-09-19T06:51:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61c0601740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f91b4e0 0x55f29f91df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61c0601740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f917e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61c0601740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f9266c0 0x55f29f920850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61c0601740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f29f936f40 con 0x55f29f91be00 2024-09-19T06:51:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61c0601740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f29f937610 con 0x55f29f9266c0 2024-09-19T06:51:19.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61c0601740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f29f937ce0 con 0x55f29f91b4e0 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f917e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f91b4e0 0x55f29f91df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f91b4e0 0x55f29f91df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:39482/0 (socket says 172.21.15.114:39482) 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 -- 172.21.15.114:0/732042482 learned_addr learned my addr 172.21.15.114:0/732042482 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 --2- 172.21.15.114:0/732042482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f9266c0 0x55f29f920850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 -- 172.21.15.114:0/732042482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f9266c0 msgr2=0x55f29f920850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 --2- 172.21.15.114:0/732042482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f9266c0 0x55f29f920850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 -- 172.21.15.114:0/732042482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 msgr2=0x55f29f917e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 --2- 172.21.15.114:0/732042482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f917e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.914+0000 7f61b7400640 1 -- 172.21.15.114:0/732042482 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29f925f60 con 0x55f29f91b4e0 2024-09-19T06:51:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.915+0000 7f61b7400640 1 --2- 172.21.15.114:0/732042482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f91b4e0 0x55f29f91df40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009430 tx=0x7f61b0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2803764d568b4c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.915+0000 7f61a3400640 1 -- 172.21.15.114:0/732042482 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b004e040 con 0x55f29f91b4e0 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.915+0000 7f61a3400640 1 -- 172.21.15.114:0/732042482 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b0040040 con 0x55f29f91b4e0 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.915+0000 7f61a3400640 1 -- 172.21.15.114:0/732042482 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0045470 con 0x55f29f91b4e0 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.915+0000 7f61c0601740 1 -- 172.21.15.114:0/732042482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f91b4e0 msgr2=0x55f29f91df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 --2- 172.21.15.114:0/732042482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f91b4e0 0x55f29f91df40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009430 tx=0x7f61b0004a40 comp rx=0 tx=0).stop 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 -- 172.21.15.114:0/732042482 shutdown_connections 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 --2- 172.21.15.114:0/732042482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f9266c0 0x55f29f920850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 --2- 172.21.15.114:0/732042482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91be00 0x55f29f917e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 --2- 172.21.15.114:0/732042482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f91b4e0 0x55f29f91df40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 -- 172.21.15.114:0/732042482 >> 172.21.15.114:0/732042482 conn(0x55f29f917a70 msgr2=0x55f29f921ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 -- 172.21.15.114:0/732042482 shutdown_connections 2024-09-19T06:51:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 -- 172.21.15.114:0/732042482 wait complete. 2024-09-19T06:51:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 -- start start 2024-09-19T06:51:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.916+0000 7f61c0601740 1 Processor -- start 2024-09-19T06:51:19.920 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61c0601740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91b4e0 0x55f29f93abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61c0601740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91be00 0x55f29f93b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61c0601740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f93b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61c0601740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f29f938a80 con 0x55f29f91b4e0 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61c0601740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f29f9383b0 con 0x55f29f91be00 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61c0601740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55f29f939150 con 0x55f29f9266c0 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61c0601740 10 client.0 _ll_register_callbacks cb 0x7f61a4009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61b7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91b4e0 0x55f29f93abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61b7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91b4e0 0x55f29f93abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55352/0 (socket says 172.21.15.114:55352) 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61b7400640 1 -- 172.21.15.114:0/4162746476 learned_addr learned my addr 172.21.15.114:0/4162746476 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61b6000640 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91be00 0x55f29f93b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.917+0000 7f61b7400640 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f93b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61b7400640 1 -- 172.21.15.114:0/4162746476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 msgr2=0x55f29f93b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:19.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61b7400640 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f93b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61b7400640 1 -- 172.21.15.114:0/4162746476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91be00 msgr2=0x55f29f93b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61b7400640 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91be00 0x55f29f93b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61b7400640 1 -- 172.21.15.114:0/4162746476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29f93f400 con 0x55f29f91b4e0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61b7400640 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91b4e0 0x55f29f93abe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f61b0005d50 tx=0x7f61b0003cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4177496af9e60a server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61a0c00640 1 -- 172.21.15.114:0/4162746476 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0045470 con 0x55f29f91b4e0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61c0601740 1 -- 172.21.15.114:0/4162746476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b0040040 con 0x55f29f91b4e0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61a0c00640 1 -- 172.21.15.114:0/4162746476 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f61b0045610 con 0x55f29f91b4e0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61a0c00640 1 -- 172.21.15.114:0/4162746476 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61b0045810 con 0x55f29f91b4e0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61c0601740 10 client.15030 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.918+0000 7f61c0601740 1 -- 172.21.15.114:0/4162746476 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f29f9a21c0 con 0x55f29f91b4e0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61a0c00640 1 -- 172.21.15.114:0/4162746476 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b0040040 con 0x55f29f91b4e0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61a0c00640 1 -- 172.21.15.114:0/4162746476 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f61b00628d0 con 0x55f29f91b4e0 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61c0601740 10 client.15030 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:19.922 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61c0601740 10 client.15030 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T06:51:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61c0601740 20 client.15030 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61c0601740 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55f29f9a6920 0x55f29f9a5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61c0601740 4 client.15030 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61c0601740 1 -- 172.21.15.114:0/4162746476 --> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] -- command(tid 0: {"prefix": "status"}) -- 0x55f29f888b30 con 0x55f29f9a6920 2024-09-19T06:51:19.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.919+0000 7f61b6000640 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55f29f9a6920 0x55f29f9a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61b6000640 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55f29f9a6920 0x55f29f9a5630 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=35e20961be2496e6 server_cookie=8428aa7002fae77b in_seq=0 out_seq=0 2024-09-19T06:51:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61a0c00640 10 client.15030 ms_handle_connect on v2:172.21.15.116:6838/2386737584 2024-09-19T06:51:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61a0c00640 1 -- 172.21.15.114:0/4162746476 <== mds.1 v2:172.21.15.116:6838/2386737584 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55f29f888b30 con 0x55f29f9a6920 2024-09-19T06:51:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61a0c00640 10 client.15030 handle_command_reply: tid=0 2024-09-19T06:51:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 client.15030 shutdown 2024-09-19T06:51:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 -- 172.21.15.114:0/4162746476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91b4e0 msgr2=0x55f29f93abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91b4e0 0x55f29f93abe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f61b0005d50 tx=0x7f61b0003cb0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 -- 172.21.15.114:0/4162746476 shutdown_connections 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f29f9266c0 0x55f29f93b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55f29f9a6920 0x55f29f9a5630 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f29f91be00 0x55f29f93b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 --2- 172.21.15.114:0/4162746476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f29f91b4e0 0x55f29f93abe0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61c0601740 1 -- 172.21.15.114:0/4162746476 >> 172.21.15.114:0/4162746476 conn(0x55f29f917a70 msgr2=0x55f29f931700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.920+0000 7f61a0c00640 0 client.15030 ms_handle_reset on v2:172.21.15.116:6838/2386737584 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.921+0000 7f61c0601740 1 -- 172.21.15.114:0/4162746476 shutdown_connections 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.921+0000 7f61c0601740 1 -- 172.21.15.114:0/4162746476 wait complete. 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.921+0000 7f61c0601740 20 client.15030 trim_cache size 0 max 16384 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:19.925 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 1, 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24524, 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 84.709984198000001, 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 141.29559576599999 2024-09-19T06:51:19.926 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:19.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.921+0000 7f61b7e00640 1 -- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f619c0a3350 msgr2=0x7f619c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.921+0000 7f61b7e00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f619c0a3350 0x7f619c0a5810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f61a80986e0 tx=0x7f61a80d3cc0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.921+0000 7f61b7e00640 1 -- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8103e30 msgr2=0x7f61b819a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:19.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8103e30 0x7f61b819a410 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f61a800e9f0 tx=0x7f61a800eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 -- 172.21.15.114:0/2778374667 shutdown_connections 2024-09-19T06:51:19.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f619c0a3350 0x7f619c0a5810 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61b8108f60 0x7f61b819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61b81047e0 0x7f61b819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 --2- 172.21.15.114:0/2778374667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61b8103e30 0x7f61b819a410 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:19.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 -- 172.21.15.114:0/2778374667 >> 172.21.15.114:0/2778374667 conn(0x7f61b80ff8d0 msgr2=0x7f61b8100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:19.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 -- 172.21.15.114:0/2778374667 shutdown_connections 2024-09-19T06:51:19.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:19.922+0000 7f61b7e00640 1 -- 172.21.15.114:0/2778374667 wait complete. 2024-09-19T06:51:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:20 smithi116 ceph-mon[35411]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:20.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:20 smithi118 ceph-mon[35527]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:20.600 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:20 smithi114 ceph-mon[30820]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:20.638 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-19T06:51:21.025 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.600+0000 7f0815a00640 1 -- 172.21.15.114:0/3223203930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 msgr2=0x7f0810101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.600+0000 7f0815a00640 1 --2- 172.21.15.114:0/3223203930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 0x7f0810101e20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f07fc009930 tx=0x7f07fc02a420 comp rx=0 tx=0).stop 2024-09-19T06:51:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.600+0000 7f0815a00640 1 -- 172.21.15.114:0/3223203930 shutdown_connections 2024-09-19T06:51:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.600+0000 7f0815a00640 1 --2- 172.21.15.114:0/3223203930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0810102360 0x7f081010e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.600+0000 7f0815a00640 1 --2- 172.21.15.114:0/3223203930 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 0x7f0810101e20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.600+0000 7f0815a00640 1 --2- 172.21.15.114:0/3223203930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08101079c0 0x7f0810107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.600+0000 7f0815a00640 1 -- 172.21.15.114:0/3223203930 >> 172.21.15.114:0/3223203930 conn(0x7f08100fd740 msgr2=0x7f08100ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.601+0000 7f0815a00640 1 -- 172.21.15.114:0/3223203930 shutdown_connections 2024-09-19T06:51:21.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.601+0000 7f0815a00640 1 -- 172.21.15.114:0/3223203930 wait complete. 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.602+0000 7f0815a00640 1 Processor -- start 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.602+0000 7f0815a00640 1 -- start start 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.602+0000 7f0815a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 0x7f081019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.602+0000 7f0815a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0810102360 0x7f081019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.602+0000 7f0815a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08101079c0 0x7f081019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.602+0000 7f0815a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0810111dc0 con 0x7f08101079c0 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.602+0000 7f0815a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0810111c40 con 0x7f0810102360 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.602+0000 7f0815a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0810111f40 con 0x7f08101019c0 2024-09-19T06:51:21.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 0x7f081019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 0x7f081019a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:39522/0 (socket says 172.21.15.114:39522) 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080fe00640 1 -- 172.21.15.114:0/1538218452 learned_addr learned my addr 172.21.15.114:0/1538218452 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f0815000640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08101079c0 0x7f081019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080f400640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0810102360 0x7f081019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080fe00640 1 -- 172.21.15.114:0/1538218452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0810102360 msgr2=0x7f081019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080fe00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0810102360 0x7f081019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080fe00640 1 -- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08101079c0 msgr2=0x7f081019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080fe00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08101079c0 0x7f081019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f080fe00640 1 -- 172.21.15.114:0/1538218452 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081019f4d0 con 0x7f08101019c0 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.603+0000 7f0815000640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08101079c0 0x7f081019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.604+0000 7f080fe00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 0x7f081019a520 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f080000e9f0 tx=0x7f080000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.604+0000 7f080cc00640 1 -- 172.21.15.114:0/1538218452 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080000cd80 con 0x7f08101019c0 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.604+0000 7f0815a00640 1 -- 172.21.15.114:0/1538218452 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f081019f7c0 con 0x7f08101019c0 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.604+0000 7f080cc00640 1 -- 172.21.15.114:0/1538218452 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0800005500 con 0x7f08101019c0 2024-09-19T06:51:21.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.604+0000 7f080cc00640 1 -- 172.21.15.114:0/1538218452 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0800010690 con 0x7f08101019c0 2024-09-19T06:51:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.604+0000 7f0815a00640 1 -- 172.21.15.114:0/1538218452 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08101a70a0 con 0x7f08101019c0 2024-09-19T06:51:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.607+0000 7f0819a87740 1 Processor -- start 2024-09-19T06:51:21.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.607+0000 7f0819a87740 1 -- start start 2024-09-19T06:51:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.607+0000 7f0819a87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdb58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.607+0000 7f0819a87740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 0x5572dfdc0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.607+0000 7f0819a87740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdc2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.607+0000 7f0819a87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5572dfd1b0b0 con 0x5572dfdb54e0 2024-09-19T06:51:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.607+0000 7f0819a87740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5572dfcc2ab0 con 0x5572dfdc06c0 2024-09-19T06:51:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.607+0000 7f0819a87740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5572dfd17070 con 0x5572dfdb5e00 2024-09-19T06:51:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f0815000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdb58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f0815000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdb58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55384/0 (socket says 172.21.15.114:55384) 2024-09-19T06:51:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f0815000640 1 -- 172.21.15.114:0/1092701795 learned_addr learned my addr 172.21.15.114:0/1092701795 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f080f400640 1 --2- 172.21.15.114:0/1092701795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdc2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f0815000640 1 -- 172.21.15.114:0/1092701795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 msgr2=0x5572dfdc0180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-19T06:51:21.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f0815000640 1 --2- 172.21.15.114:0/1092701795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 0x5572dfdc0180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f0815000640 1 -- 172.21.15.114:0/1092701795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 msgr2=0x5572dfdc2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f0815000640 1 --2- 172.21.15.114:0/1092701795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdc2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f0815000640 1 -- 172.21.15.114:0/1092701795 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572dfccf4a0 con 0x5572dfdb54e0 2024-09-19T06:51:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.608+0000 7f080cc00640 1 -- 172.21.15.114:0/1538218452 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) ==== 138407+0+0 (secure 0 0 0) 0x7f08000046c0 con 0x7f08101019c0 2024-09-19T06:51:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.609+0000 7f0815000640 1 --2- 172.21.15.114:0/1092701795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdb58c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0808009fd0 tx=0x7f08080108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de76a0f582ab9c25 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.609+0000 7f0805600640 1 -- 172.21.15.114:0/1092701795 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0808011760 con 0x5572dfdb54e0 2024-09-19T06:51:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.609+0000 7f0805600640 1 -- 172.21.15.114:0/1092701795 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0808011de0 con 0x5572dfdb54e0 2024-09-19T06:51:21.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.609+0000 7f0805600640 1 -- 172.21.15.114:0/1092701795 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080800e3c0 con 0x5572dfdb54e0 2024-09-19T06:51:21.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f0819a87740 1 -- 172.21.15.114:0/1092701795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 msgr2=0x5572dfdb58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f0819a87740 1 --2- 172.21.15.114:0/1092701795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdb58c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0808009fd0 tx=0x7f08080108e0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f080cc00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f07f40a3350 0x7f07f40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f080cc00640 1 -- 172.21.15.114:0/1538218452 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f08000d0330 con 0x7f08101019c0 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f0819a87740 1 -- 172.21.15.114:0/1092701795 shutdown_connections 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f0819a87740 1 --2- 172.21.15.114:0/1092701795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdc2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f0819a87740 1 --2- 172.21.15.114:0/1092701795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 0x5572dfdc0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f0819a87740 1 --2- 172.21.15.114:0/1092701795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdb58c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f0819a87740 1 -- 172.21.15.114:0/1092701795 >> 172.21.15.114:0/1092701795 conn(0x5572dfdb1a70 msgr2=0x5572dfdb1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.610+0000 7f0819a87740 1 -- 172.21.15.114:0/1092701795 shutdown_connections 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.611+0000 7f0815000640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f07f40a3350 0x7f07f40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.611+0000 7f0819a87740 1 -- 172.21.15.114:0/1092701795 wait complete. 2024-09-19T06:51:21.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 1 -- start start 2024-09-19T06:51:21.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0815000640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f07f40a3350 0x7f07f40a5810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f080801ba90 tx=0x7f080801baf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:21.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 1 Processor -- start 2024-09-19T06:51:21.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdbd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdb5e00 0x5572dfdbdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdc06c0 0x5572dfdbe030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5572dfdc3330 con 0x5572dfdb54e0 2024-09-19T06:51:21.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5572dfca3920 con 0x5572dfdb5e00 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5572dfdc34b0 con 0x5572dfdc06c0 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.612+0000 7f0819a87740 10 client.0 _ll_register_callbacks cb 0x5572dfcc78a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdb5e00 0x5572dfdbdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080f400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdb5e00 0x5572dfdbdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:47408/0 (socket says 172.21.15.114:47408) 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080f400640 1 -- 172.21.15.114:0/2257428289 learned_addr learned my addr 172.21.15.114:0/2257428289 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f0815000640 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdc06c0 0x5572dfdbe030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080fe00640 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdbd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080f400640 1 -- 172.21.15.114:0/2257428289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdc06c0 msgr2=0x5572dfdbe030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080f400640 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdc06c0 0x5572dfdbe030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080f400640 1 -- 172.21.15.114:0/2257428289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 msgr2=0x5572dfdbd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080f400640 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdbd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.613+0000 7f080f400640 1 -- 172.21.15.114:0/2257428289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572dfa8c2b0 con 0x5572dfdb5e00 2024-09-19T06:51:21.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.614+0000 7f080f400640 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdb5e00 0x5572dfdbdaf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f07fc005940 tx=0x7f07fc0317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e815c5983a6df50 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.614+0000 7f07fa000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.614+0000 7f07fa000640 1 -- 172.21.15.114:0/2257428289 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc035040 con 0x5572dfdb5e00 2024-09-19T06:51:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.614+0000 7f0819a87740 1 -- 172.21.15.114:0/2257428289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0808011de0 con 0x5572dfdb5e00 2024-09-19T06:51:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.614+0000 7f07fa000640 1 -- 172.21.15.114:0/2257428289 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f07fc004040 con 0x5572dfdb5e00 2024-09-19T06:51:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.614+0000 7f07fa000640 1 -- 172.21.15.114:0/2257428289 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f07fc004240 con 0x5572dfdb5e00 2024-09-19T06:51:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.614+0000 7f0819a87740 10 client.25037 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.614+0000 7f0819a87740 1 -- 172.21.15.114:0/2257428289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5572dfdcf030 con 0x5572dfdb5e00 2024-09-19T06:51:21.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f07fa000640 1 -- 172.21.15.114:0/2257428289 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0808011de0 con 0x5572dfdb5e00 2024-09-19T06:51:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f07fa000640 1 -- 172.21.15.114:0/2257428289 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f07fc004a20 con 0x5572dfdb5e00 2024-09-19T06:51:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f0819a87740 10 client.25037 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f0819a87740 10 client.25037 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-19T06:51:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f0819a87740 20 client.25037 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f0819a87740 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x5572dfdd15e0 0x5572dfdd2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f0819a87740 4 client.25037 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:21.620 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f0819a87740 1 -- 172.21.15.114:0/2257428289 --> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572dfd28620 con 0x5572dfdd15e0 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.615+0000 7f080fe00640 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x5572dfdd15e0 0x5572dfdd2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.616+0000 7f080fe00640 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x5572dfdd15e0 0x5572dfdd2a50 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=635f1c0964d863c6 server_cookie=d8a126d4157723fa in_seq=0 out_seq=0 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.616+0000 7f07fa000640 10 client.25037 ms_handle_connect on v2:172.21.15.118:6838/345383252 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f07fa000640 1 -- 172.21.15.114:0/2257428289 <== mds.0 v2:172.21.15.118:6838/345383252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5572dfd28620 con 0x5572dfdd15e0 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f07fa000640 10 client.25037 handle_command_reply: tid=0 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 client.25037 shutdown 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 -- 172.21.15.114:0/2257428289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdb5e00 msgr2=0x5572dfdbdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdb5e00 0x5572dfdbdaf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f07fc005940 tx=0x7f07fc0317e0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 -- 172.21.15.114:0/2257428289 shutdown_connections 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x5572dfdd15e0 0x5572dfdd2a50 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdc06c0 0x5572dfdbe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdb5e00 0x5572dfdbdaf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 --2- 172.21.15.114:0/2257428289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfdbd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 -- 172.21.15.114:0/2257428289 >> 172.21.15.114:0/2257428289 conn(0x5572dfdb1a70 msgr2=0x5572dfdb7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f07fa000640 0 client.25037 ms_handle_reset on v2:172.21.15.118:6838/345383252 2024-09-19T06:51:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 -- 172.21.15.114:0/2257428289 shutdown_connections 2024-09-19T06:51:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.617+0000 7f0819a87740 1 -- 172.21.15.114:0/2257428289 wait complete. 2024-09-19T06:51:21.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.618+0000 7f0819a87740 20 client.25037 trim_cache size 0 max 16384 2024-09-19T06:51:21.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.636+0000 7f0819a87740 1 Processor -- start 2024-09-19T06:51:21.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f0819a87740 1 -- start start 2024-09-19T06:51:21.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f0819a87740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb54e0 0x5572dfdb7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f0819a87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb5e00 0x5572dfdbde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f0819a87740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdba870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f0819a87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5572dfdd0d90 con 0x5572dfdb5e00 2024-09-19T06:51:21.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f0819a87740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5572dfdd1460 con 0x5572dfdc06c0 2024-09-19T06:51:21.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f0819a87740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5572dfdd1b30 con 0x5572dfdb54e0 2024-09-19T06:51:21.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f080f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb5e00 0x5572dfdbde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f080f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb5e00 0x5572dfdbde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55396/0 (socket says 172.21.15.114:55396) 2024-09-19T06:51:21.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f080f400640 1 -- 172.21.15.114:0/2332112186 learned_addr learned my addr 172.21.15.114:0/2332112186 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:21.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f080fe00640 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb54e0 0x5572dfdb7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.637+0000 7f0815000640 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdba870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f080fe00640 1 -- 172.21.15.114:0/2332112186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 msgr2=0x5572dfdba870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:21.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f080fe00640 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdba870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f080fe00640 1 -- 172.21.15.114:0/2332112186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb5e00 msgr2=0x5572dfdbde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:21.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f080fe00640 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb5e00 0x5572dfdbde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f080fe00640 1 -- 172.21.15.114:0/2332112186 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572dfdbbb40 con 0x5572dfdb54e0 2024-09-19T06:51:21.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f080f400640 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb5e00 0x5572dfdbde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:21.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f080fe00640 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb54e0 0x5572dfdb7f40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f08000cd890 tx=0x7f0800009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6036d46a5e8d4f93 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:21.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f0804c00640 1 -- 172.21.15.114:0/2332112186 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080009c3a0 con 0x5572dfdb54e0 2024-09-19T06:51:21.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.638+0000 7f0804c00640 1 -- 172.21.15.114:0/2332112186 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08000a0750 con 0x5572dfdb54e0 2024-09-19T06:51:21.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.639+0000 7f0804c00640 1 -- 172.21.15.114:0/2332112186 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08000a4b80 con 0x5572dfdb54e0 2024-09-19T06:51:21.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.639+0000 7f0819a87740 1 -- 172.21.15.114:0/2332112186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb54e0 msgr2=0x5572dfdb7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.639+0000 7f0819a87740 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb54e0 0x5572dfdb7f40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f08000cd890 tx=0x7f0800009720 comp rx=0 tx=0).stop 2024-09-19T06:51:21.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.639+0000 7f0819a87740 1 -- 172.21.15.114:0/2332112186 shutdown_connections 2024-09-19T06:51:21.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.639+0000 7f0819a87740 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdba870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.639+0000 7f0819a87740 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb5e00 0x5572dfdbde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.639+0000 7f0819a87740 1 --2- 172.21.15.114:0/2332112186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb54e0 0x5572dfdb7f40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.639+0000 7f0819a87740 1 -- 172.21.15.114:0/2332112186 >> 172.21.15.114:0/2332112186 conn(0x5572dfdb1a70 msgr2=0x5572dfdbbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:21.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.640+0000 7f0819a87740 1 -- 172.21.15.114:0/2332112186 shutdown_connections 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.640+0000 7f0819a87740 1 -- 172.21.15.114:0/2332112186 wait complete. 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.640+0000 7f0819a87740 1 -- start start 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.640+0000 7f0819a87740 1 Processor -- start 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f0819a87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfddb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f0819a87740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 0x5572dfdba580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f0819a87740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdbacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f0819a87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5572dfdd28d0 con 0x5572dfdb54e0 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f0819a87740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5572dfdd2200 con 0x5572dfdc06c0 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f0819a87740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5572dfdd2fa0 con 0x5572dfdb5e00 2024-09-19T06:51:21.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f0819a87740 10 client.0 _ll_register_callbacks cb 0x7f07fc009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f080fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfddb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f080fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfddb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55404/0 (socket says 172.21.15.114:55404) 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f080fe00640 1 -- 172.21.15.114:0/1017631340 learned_addr learned my addr 172.21.15.114:0/1017631340 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f080f400640 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 0x5572dfdba580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.641+0000 7f0815000640 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdbacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f080f400640 1 -- 172.21.15.114:0/1017631340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 msgr2=0x5572dfdbacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f080f400640 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdbacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f080f400640 1 -- 172.21.15.114:0/1017631340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 msgr2=0x5572dfddb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f080f400640 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfddb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f080f400640 1 -- 172.21.15.114:0/1017631340 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572dfdd9080 con 0x5572dfdb5e00 2024-09-19T06:51:21.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f080fe00640 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfddb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f080f400640 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 0x5572dfdba580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07fc006010 tx=0x7f07fc004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61132d615fc56cdb server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f07fa000640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f07fa000640 1 -- 172.21.15.114:0/1017631340 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc035070 con 0x5572dfdb5e00 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f0819a87740 1 -- 172.21.15.114:0/1017631340 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08000a0750 con 0x5572dfdb5e00 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.642+0000 7f07fa000640 1 -- 172.21.15.114:0/1017631340 <== mon.1 v2:172.21.15.118:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f07fc04b020 con 0x5572dfdb5e00 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.643+0000 7f07fa000640 1 -- 172.21.15.114:0/1017631340 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f07fc003800 con 0x5572dfdb5e00 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.643+0000 7f0819a87740 10 client.25087 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.643+0000 7f0819a87740 1 -- 172.21.15.114:0/1017631340 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5572dfe3be10 con 0x5572dfdb5e00 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.643+0000 7f07fa000640 1 -- 172.21.15.114:0/1017631340 <== mon.1 v2:172.21.15.118:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08000a0750 con 0x5572dfdb5e00 2024-09-19T06:51:21.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.643+0000 7f07fa000640 1 -- 172.21.15.114:0/1017631340 <== mon.1 v2:172.21.15.118:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f07fc003f20 con 0x5572dfdb5e00 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.643+0000 7f0819a87740 10 client.25087 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.644+0000 7f0819a87740 10 client.25087 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.644+0000 7f0819a87740 20 client.25087 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.644+0000 7f0819a87740 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x5572dfe409e0 0x5572dfe3f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.644+0000 7f0819a87740 4 client.25087 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.644+0000 7f0819a87740 1 -- 172.21.15.114:0/1017631340 --> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] -- command(tid 0: {"prefix": "status"}) -- 0x5572dfd22b30 con 0x5572dfe409e0 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.644+0000 7f080fe00640 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x5572dfe409e0 0x5572dfe3f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.644+0000 7f07fa000640 10 client.25087 ms_handle_connect on v2:172.21.15.118:6838/345383252 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.644+0000 7f080fe00640 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x5572dfe409e0 0x5572dfe3f6f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d3e1478386cfa93 server_cookie=6f8271fefc0c2b56 in_seq=0 out_seq=0 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f07fa000640 1 -- 172.21.15.114:0/1017631340 <== mds.0 v2:172.21.15.118:6838/345383252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5572dfd22b30 con 0x5572dfe409e0 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f07fa000640 10 client.25087 handle_command_reply: tid=0 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 client.25087 shutdown 2024-09-19T06:51:21.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 -- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 msgr2=0x5572dfdba580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 0x5572dfdba580 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07fc006010 tx=0x7f07fc004610 comp rx=0 tx=0).stop 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 -- 172.21.15.114:0/1017631340 shutdown_connections 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x5572dfe409e0 0x5572dfe3f6f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572dfdc06c0 0x5572dfdbacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5572dfdb5e00 0x5572dfdba580 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 --2- 172.21.15.114:0/1017631340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5572dfdb54e0 0x5572dfddb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f07fa000640 0 client.25087 ms_handle_reset on v2:172.21.15.118:6838/345383252 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 -- 172.21.15.114:0/1017631340 >> 172.21.15.114:0/1017631340 conn(0x5572dfdb1a70 msgr2=0x5572dfdb03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 -- 172.21.15.114:0/1017631340 shutdown_connections 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 1 -- 172.21.15.114:0/1017631340 wait complete. 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.645+0000 7f0819a87740 20 client.25087 trim_cache size 0 max 16384 2024-09-19T06:51:21.650 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24538, 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 129.280196816, 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 153.735939008 2024-09-19T06:51:21.651 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 -- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f07f40a3350 msgr2=0x7f07f40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f07f40a3350 0x7f07f40a5810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f080801ba90 tx=0x7f080801baf0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 -- 172.21.15.114:0/1538218452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 msgr2=0x7f081019a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 0x7f081019a520 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f080000e9f0 tx=0x7f080000edf0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 -- 172.21.15.114:0/1538218452 shutdown_connections 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f07f40a3350 0x7f07f40a5810 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08101079c0 0x7f081019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0810102360 0x7f081019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 --2- 172.21.15.114:0/1538218452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08101019c0 0x7f081019a520 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:21.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.646+0000 7f0815a00640 1 -- 172.21.15.114:0/1538218452 >> 172.21.15.114:0/1538218452 conn(0x7f08100fd740 msgr2=0x7f081010cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:21.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.647+0000 7f0815a00640 1 -- 172.21.15.114:0/1538218452 shutdown_connections 2024-09-19T06:51:21.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:21.647+0000 7f0815a00640 1 -- 172.21.15.114:0/1538218452 wait complete. 2024-09-19T06:51:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:22 smithi114 ceph-mon[30820]: pgmap v370: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:22 smithi116 ceph-mon[35411]: pgmap v370: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:22.455 INFO:tasks.cephadm:Setup complete, yielding 2024-09-19T06:51:22.455 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-19T06:51:22.465 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-19T06:51:22.465 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- bash -c 'ceph orch status' 2024-09-19T06:51:22.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:22 smithi118 ceph-mon[35527]: pgmap v370: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:22.834 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:23.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:23 smithi114 ceph-mon[30820]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-19T06:51:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.391+0000 7f92e1000640 1 -- 172.21.15.114:0/4077805335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 msgr2=0x7f92dc074fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.391+0000 7f92e1000640 1 --2- 172.21.15.114:0/4077805335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 0x7f92dc074fa0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009900 tx=0x7f92c802a410 comp rx=0 tx=0).stop 2024-09-19T06:51:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.392+0000 7f92e1000640 1 -- 172.21.15.114:0/4077805335 shutdown_connections 2024-09-19T06:51:23.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.392+0000 7f92e1000640 1 --2- 172.21.15.114:0/4077805335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92dc109a70 0x7f92dc10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.392+0000 7f92e1000640 1 --2- 172.21.15.114:0/4077805335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 0x7f92dc074fa0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.392+0000 7f92e1000640 1 --2- 172.21.15.114:0/4077805335 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92dc0761a0 0x7f92dc074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.392+0000 7f92e1000640 1 -- 172.21.15.114:0/4077805335 >> 172.21.15.114:0/4077805335 conn(0x7f92dc0fd390 msgr2=0x7f92dc0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.392+0000 7f92e1000640 1 -- 172.21.15.114:0/4077805335 shutdown_connections 2024-09-19T06:51:23.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.392+0000 7f92e1000640 1 -- 172.21.15.114:0/4077805335 wait complete. 2024-09-19T06:51:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92e1000640 1 Processor -- start 2024-09-19T06:51:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92e1000640 1 -- start start 2024-09-19T06:51:23.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92e1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 0x7f92dc19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92e1000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92dc0761a0 0x7f92dc19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92e1000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92dc109a70 0x7f92dc1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92e1000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc10faa0 con 0x7f92dc074b40 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92e1000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc10f920 con 0x7f92dc109a70 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92e1000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc10fc20 con 0x7f92dc0761a0 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92d9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 0x7f92dc19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92d9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 0x7f92dc19a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55416/0 (socket says 172.21.15.114:55416) 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92d9e00640 1 -- 172.21.15.114:0/2172796953 learned_addr learned my addr 172.21.15.114:0/2172796953 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.393+0000 7f92d9400640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92dc0761a0 0x7f92dc19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.394+0000 7f92da800640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92dc109a70 0x7f92dc1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.394+0000 7f92d9400640 1 -- 172.21.15.114:0/2172796953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92dc109a70 msgr2=0x7f92dc1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:23.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.394+0000 7f92d9400640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92dc109a70 0x7f92dc1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.394+0000 7f92d9400640 1 -- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 msgr2=0x7f92dc19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.394+0000 7f92d9400640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 0x7f92dc19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.394+0000 7f92d9400640 1 -- 172.21.15.114:0/2172796953 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc1a1d10 con 0x7f92dc0761a0 2024-09-19T06:51:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.394+0000 7f92d9e00640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 0x7f92dc19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.394+0000 7f92d9400640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92dc0761a0 0x7f92dc19a620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f92c802a930 tx=0x7f92c8032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:23.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.395+0000 7f92d2a00640 1 -- 172.21.15.114:0/2172796953 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c803f030 con 0x7f92dc0761a0 2024-09-19T06:51:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.395+0000 7f92d2a00640 1 -- 172.21.15.114:0/2172796953 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92c8003ce0 con 0x7f92dc0761a0 2024-09-19T06:51:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.395+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92dc1a1fa0 con 0x7f92dc0761a0 2024-09-19T06:51:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.395+0000 7f92d2a00640 1 -- 172.21.15.114:0/2172796953 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c8035070 con 0x7f92dc0761a0 2024-09-19T06:51:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.395+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92dc1a23f0 con 0x7f92dc0761a0 2024-09-19T06:51:23.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.396+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92dc108880 con 0x7f92dc0761a0 2024-09-19T06:51:23.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.398+0000 7f92d2a00640 1 -- 172.21.15.114:0/2172796953 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f92c8042030 con 0x7f92dc0761a0 2024-09-19T06:51:23.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.399+0000 7f92d2a00640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f92c00a3270 0x7f92c00a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:23.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.399+0000 7f92d2a00640 1 -- 172.21.15.114:0/2172796953 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f92c80f0350 con 0x7f92dc0761a0 2024-09-19T06:51:23.400 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.399+0000 7f92d9e00640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f92c00a3270 0x7f92c00a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:23.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.400+0000 7f92d9e00640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f92c00a3270 0x7f92c00a5730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f92cc011550 tx=0x7f92cc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:23.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.401+0000 7f92d2a00640 1 -- 172.21.15.114:0/2172796953 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f92c80bc5d0 con 0x7f92dc0761a0 2024-09-19T06:51:23.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:23 smithi118 ceph-mon[35527]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-19T06:51:23.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.591+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f92dc062630 con 0x7f92c00a3270 2024-09-19T06:51:23.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.592+0000 7f92d2a00640 1 -- 172.21.15.114:0/2172796953 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f92dc062630 con 0x7f92c00a3270 2024-09-19T06:51:23.594 INFO:teuthology.orchestra.run.smithi114.stdout:Backend: cephadm 2024-09-19T06:51:23.594 INFO:teuthology.orchestra.run.smithi114.stdout:Available: Yes 2024-09-19T06:51:23.594 INFO:teuthology.orchestra.run.smithi114.stdout:Paused: No 2024-09-19T06:51:23.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f92c00a3270 msgr2=0x7f92c00a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:23.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f92c00a3270 0x7f92c00a5730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f92cc011550 tx=0x7f92cc005f90 comp rx=0 tx=0).stop 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92dc0761a0 msgr2=0x7f92dc19a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92dc0761a0 0x7f92dc19a620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f92c802a930 tx=0x7f92c8032ce0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 shutdown_connections 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f92c00a3270 0x7f92c00a5730 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92dc109a70 0x7f92dc1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92dc0761a0 0x7f92dc19a620 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 --2- 172.21.15.114:0/2172796953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92dc074b40 0x7f92dc19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 >> 172.21.15.114:0/2172796953 conn(0x7f92dc0fd390 msgr2=0x7f92dc105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 shutdown_connections 2024-09-19T06:51:23.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:23.595+0000 7f92e1000640 1 -- 172.21.15.114:0/2172796953 wait complete. 2024-09-19T06:51:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:23 smithi116 ceph-mon[35411]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-19T06:51:24.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-19T06:51:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:24 smithi118 ceph-mon[35527]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:24.631 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:24 smithi114 ceph-mon[30820]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:24 smithi116 ceph-mon[35411]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:25.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 -- 172.21.15.114:0/912053471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 msgr2=0x7f9d081073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:25.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 --2- 172.21.15.114:0/912053471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 0x7f9d081073a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009900 tx=0x7f9cf002a410 comp rx=0 tx=0).stop 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 -- 172.21.15.114:0/912053471 shutdown_connections 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 --2- 172.21.15.114:0/912053471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d081078e0 0x7f9d08109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 --2- 172.21.15.114:0/912053471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 0x7f9d081073a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 --2- 172.21.15.114:0/912053471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d08102680 0x7f9d08104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 -- 172.21.15.114:0/912053471 >> 172.21.15.114:0/912053471 conn(0x7f9d080fc280 msgr2=0x7f9d080fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 -- 172.21.15.114:0/912053471 shutdown_connections 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.180+0000 7f9d0e800640 1 -- 172.21.15.114:0/912053471 wait complete. 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.181+0000 7f9d0e800640 1 Processor -- start 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.181+0000 7f9d0e800640 1 -- start start 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9d0e800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d08102680 0x7f9d08197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9d0e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 0x7f9d08198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9d0e800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d081078e0 0x7f9d0819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:25.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9d0e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0810d860 con 0x7f9d08104fb0 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9d0e800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0810d6e0 con 0x7f9d081078e0 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9d0e800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0810d9e0 con 0x7f9d08102680 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9cfea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 0x7f9d08198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9cfea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 0x7f9d08198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55446/0 (socket says 172.21.15.114:55446) 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9cfea00640 1 -- 172.21.15.114:0/554051156 learned_addr learned my addr 172.21.15.114:0/554051156 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9d07400640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d08102680 0x7f9d08197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.182+0000 7f9d07e00640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d081078e0 0x7f9d0819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9cfea00640 1 -- 172.21.15.114:0/554051156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d08102680 msgr2=0x7f9d08197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9cfea00640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d08102680 0x7f9d08197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9cfea00640 1 -- 172.21.15.114:0/554051156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d081078e0 msgr2=0x7f9d0819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9cfea00640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d081078e0 0x7f9d0819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9cfea00640 1 -- 172.21.15.114:0/554051156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0819fc00 con 0x7f9d08104fb0 2024-09-19T06:51:25.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9cfea00640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 0x7f9d08198510 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0006010 tx=0x7f9cf0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9d04c00640 1 -- 172.21.15.114:0/554051156 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf003f030 con 0x7f9d08104fb0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9d04c00640 1 -- 172.21.15.114:0/554051156 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cf0031a20 con 0x7f9d08104fb0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0819fe90 con 0x7f9d08104fb0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9d04c00640 1 -- 172.21.15.114:0/554051156 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0035070 con 0x7f9d08104fb0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.183+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d081a02e0 con 0x7f9d08104fb0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.184+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d080fdd70 con 0x7f9d08104fb0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.186+0000 7f9d04c00640 1 -- 172.21.15.114:0/554051156 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f9cf0042030 con 0x7f9d08104fb0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.187+0000 7f9d04c00640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cec0a3270 0x7f9cec0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.187+0000 7f9d04c00640 1 -- 172.21.15.114:0/554051156 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f9cf00f00d0 con 0x7f9d08104fb0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.187+0000 7f9d07400640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cec0a3270 0x7f9cec0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.188+0000 7f9d07400640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cec0a3270 0x7f9cec0a5730 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009bd0 tx=0x7f9cf8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:25.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.192+0000 7f9d04c00640 1 -- 172.21.15.114:0/554051156 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f9cf00bc350 con 0x7f9d08104fb0 2024-09-19T06:51:25.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.376+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f9d08062630 con 0x7f9cec0a3270 2024-09-19T06:51:25.377 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:25 smithi114 ceph-mon[30820]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:25.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.388+0000 7f9d04c00640 1 -- 172.21.15.114:0/554051156 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f9d08062630 con 0x7f9cec0a3270 2024-09-19T06:51:25.389 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-19T06:51:25.389 INFO:teuthology.orchestra.run.smithi114.stdout:mds.a smithi114 running (2m) 10s ago 2m 16.6M - 19.1.1-361-g2f599a46 d2baa855c067 8f1ede13d215 2024-09-19T06:51:25.389 INFO:teuthology.orchestra.run.smithi114.stdout:mds.b smithi116 running (2m) 12s ago 2m 19.4M - 19.1.1-361-g2f599a46 d2baa855c067 fdafe1263299 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.c smithi118 running (2m) 12s ago 2m 18.5M - 19.1.1-361-g2f599a46 d2baa855c067 499bdf3469e0 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.d smithi114 running (2m) 10s ago 2m 13.2M - 19.1.1-361-g2f599a46 d2baa855c067 b0213f5b7fab 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.e smithi116 running (2m) 12s ago 2m 20.7M - 19.1.1-361-g2f599a46 d2baa855c067 603a1c1e9bfe 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.f smithi118 running (2m) 12s ago 2m 19.2M - 19.1.1-361-g2f599a46 d2baa855c067 1f75ea9ff44c 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.g smithi114 running (2m) 10s ago 2m 13.0M - 19.1.1-361-g2f599a46 d2baa855c067 9d0acfebd10c 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.h smithi116 running (2m) 12s ago 2m 18.0M - 19.1.1-361-g2f599a46 d2baa855c067 b3c95259708b 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.i smithi118 running (2m) 12s ago 2m 29.3M - 19.1.1-361-g2f599a46 d2baa855c067 8f43fef95b92 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.j smithi114 running (2m) 10s ago 2m 16.8M - 19.1.1-361-g2f599a46 d2baa855c067 00ed245ba669 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.k smithi116 running (2m) 12s ago 2m 19.8M - 19.1.1-361-g2f599a46 d2baa855c067 12d8a50226c5 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mds.l smithi118 running (3m) 12s ago 3m 22.0M - 19.1.1-361-g2f599a46 d2baa855c067 cbb3da9be1cf 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi114 *:9283,8765 running (10m) 10s ago 10m 510M - 19.1.1-361-g2f599a46 d2baa855c067 f9558e3ea802 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi116 *:8443,8765 running (8m) 12s ago 8m 444M - 19.1.1-361-g2f599a46 d2baa855c067 003d5c462a3f 2024-09-19T06:51:25.390 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.z smithi118 *:8443,8765 running (8m) 12s ago 8m 442M - 19.1.1-361-g2f599a46 d2baa855c067 99a0012696ac 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (10m) 10s ago 10m 50.1M 2048M 19.1.1-361-g2f599a46 d2baa855c067 a7a113530492 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi116 running (8m) 12s ago 8m 42.8M 2048M 19.1.1-361-g2f599a46 d2baa855c067 26689fb910f8 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi118 running (8m) 12s ago 8m 41.0M 2048M 19.1.1-361-g2f599a46 d2baa855c067 c01cce961036 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (8m) 10s ago 8m 45.7M 4272M 19.1.1-361-g2f599a46 d2baa855c067 8c312c48a012 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi116 running (7m) 12s ago 7m 48.7M 4272M 19.1.1-361-g2f599a46 d2baa855c067 0481a2f2e89b 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi118 running (7m) 12s ago 7m 47.0M 4272M 19.1.1-361-g2f599a46 d2baa855c067 e1e93a6f9985 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (6m) 10s ago 6m 43.3M 4272M 19.1.1-361-g2f599a46 d2baa855c067 a2c4d1eeaafa 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi116 running (6m) 12s ago 6m 44.7M 4272M 19.1.1-361-g2f599a46 d2baa855c067 553c1c6d6d7f 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi118 running (6m) 12s ago 5m 46.6M 4272M 19.1.1-361-g2f599a46 d2baa855c067 ae7ea0103be6 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi114 running (5m) 10s ago 5m 51.6M 4272M 19.1.1-361-g2f599a46 d2baa855c067 29b8487d697e 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi116 running (5m) 12s ago 5m 45.6M 4272M 19.1.1-361-g2f599a46 d2baa855c067 937aa61010a3 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.8 smithi118 running (4m) 12s ago 4m 44.2M 4272M 19.1.1-361-g2f599a46 d2baa855c067 badd6b1c8594 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.9 smithi114 running (4m) 10s ago 4m 52.2M 4272M 19.1.1-361-g2f599a46 d2baa855c067 c7cdddbd5351 2024-09-19T06:51:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:osd.10 smithi116 running (3m) 12s ago 3m 46.3M 4272M 19.1.1-361-g2f599a46 d2baa855c067 492cad6125f2 2024-09-19T06:51:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:osd.11 smithi118 running (3m) 12s ago 3m 43.6M 4272M 19.1.1-361-g2f599a46 d2baa855c067 a81672fd9286 2024-09-19T06:51:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cec0a3270 msgr2=0x7f9cec0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cec0a3270 0x7f9cec0a5730 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8009bd0 tx=0x7f9cf8005f90 comp rx=0 tx=0).stop 2024-09-19T06:51:25.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 msgr2=0x7f9d08198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 0x7f9d08198510 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0006010 tx=0x7f9cf0004270 comp rx=0 tx=0).stop 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 shutdown_connections 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f9cec0a3270 0x7f9cec0a5730 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d081078e0 0x7f9d0819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d08104fb0 0x7f9d08198510 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 --2- 172.21.15.114:0/554051156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d08102680 0x7f9d08197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 >> 172.21.15.114:0/554051156 conn(0x7f9d080fc280 msgr2=0x7f9d08102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 shutdown_connections 2024-09-19T06:51:25.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:25.390+0000 7f9d0e800640 1 -- 172.21.15.114:0/554051156 wait complete. 2024-09-19T06:51:25.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:25 smithi118 ceph-mon[35527]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:25 smithi116 ceph-mon[35411]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:26.146 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-19T06:51:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:26 smithi114 ceph-mon[30820]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-19T06:51:26.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:26 smithi118 ceph-mon[35527]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-19T06:51:26.517 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:26 smithi116 ceph-mon[35411]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-19T06:51:27.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.105+0000 7fbc54c00640 1 -- 172.21.15.114:0/2496808884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc501037c0 msgr2=0x7fbc5010a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:27.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.105+0000 7fbc54c00640 1 --2- 172.21.15.114:0/2496808884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc501037c0 0x7fbc5010a050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009ca0 tx=0x7fbc4802a8c0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.106+0000 7fbc54c00640 1 -- 172.21.15.114:0/2496808884 shutdown_connections 2024-09-19T06:51:27.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.106+0000 7fbc54c00640 1 --2- 172.21.15.114:0/2496808884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc501037c0 0x7fbc5010a050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.106+0000 7fbc54c00640 1 --2- 172.21.15.114:0/2496808884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc50102e00 0x7fbc50103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.106+0000 7fbc54c00640 1 --2- 172.21.15.114:0/2496808884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc50101c00 0x7fbc50102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.106+0000 7fbc54c00640 1 -- 172.21.15.114:0/2496808884 >> 172.21.15.114:0/2496808884 conn(0x7fbc500fd390 msgr2=0x7fbc500ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:27.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.106+0000 7fbc54c00640 1 -- 172.21.15.114:0/2496808884 shutdown_connections 2024-09-19T06:51:27.108 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.106+0000 7fbc54c00640 1 -- 172.21.15.114:0/2496808884 wait complete. 2024-09-19T06:51:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.107+0000 7fbc54c00640 1 Processor -- start 2024-09-19T06:51:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.107+0000 7fbc54c00640 1 -- start start 2024-09-19T06:51:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.107+0000 7fbc54c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc50101c00 0x7fbc50197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.107+0000 7fbc54c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc50102e00 0x7fbc501983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:27.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.107+0000 7fbc54c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc501037c0 0x7fbc5019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc54c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5010d8d0 con 0x7fbc501037c0 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc54c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5010d750 con 0x7fbc50102e00 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc54c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5010da50 con 0x7fbc50101c00 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc50101c00 0x7fbc50197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc47e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc50102e00 0x7fbc501983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc50101c00 0x7fbc50197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:50752/0 (socket says 172.21.15.114:50752) 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 -- 172.21.15.114:0/1319417549 learned_addr learned my addr 172.21.15.114:0/1319417549 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4e800640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc501037c0 0x7fbc5019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 -- 172.21.15.114:0/1319417549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc50102e00 msgr2=0x7fbc501983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc50102e00 0x7fbc501983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 -- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc501037c0 msgr2=0x7fbc5019f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc501037c0 0x7fbc5019f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 -- 172.21.15.114:0/1319417549 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5019faa0 con 0x7fbc50101c00 2024-09-19T06:51:27.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.108+0000 7fbc4de00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc50101c00 0x7fbc50197e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000e9c0 tx=0x7fbc4000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.109+0000 7fbc46a00640 1 -- 172.21.15.114:0/1319417549 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4000cd80 con 0x7fbc50101c00 2024-09-19T06:51:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.109+0000 7fbc46a00640 1 -- 172.21.15.114:0/1319417549 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc400035f0 con 0x7fbc50101c00 2024-09-19T06:51:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.109+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5019fd90 con 0x7fbc50101c00 2024-09-19T06:51:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.109+0000 7fbc46a00640 1 -- 172.21.15.114:0/1319417549 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40010690 con 0x7fbc50101c00 2024-09-19T06:51:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.109+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc501a0210 con 0x7fbc50101c00 2024-09-19T06:51:27.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.110+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc50108f30 con 0x7fbc50101c00 2024-09-19T06:51:27.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.112+0000 7fbc46a00640 1 -- 172.21.15.114:0/1319417549 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7fbc400037a0 con 0x7fbc50101c00 2024-09-19T06:51:27.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.113+0000 7fbc46a00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbc340a32f0 0x7fbc340a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.113+0000 7fbc46a00640 1 -- 172.21.15.114:0/1319417549 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fbc400d02c0 con 0x7fbc50101c00 2024-09-19T06:51:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.114+0000 7fbc47e00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbc340a32f0 0x7fbc340a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:27.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.114+0000 7fbc47e00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbc340a32f0 0x7fbc340a57b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbc50199390 tx=0x7fbc38005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:27.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.115+0000 7fbc46a00640 1 -- 172.21.15.114:0/1319417549 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fbc4009cb90 con 0x7fbc50101c00 2024-09-19T06:51:27.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.305+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fbc50062630 con 0x7fbc340a32f0 2024-09-19T06:51:27.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:27 smithi114 ceph-mon[30820]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:27.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.310+0000 7fbc46a00640 1 -- 172.21.15.114:0/1319417549 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fbc50062630 con 0x7fbc340a32f0 2024-09-19T06:51:27.312 INFO:teuthology.orchestra.run.smithi114.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-19T06:51:27.312 INFO:teuthology.orchestra.run.smithi114.stdout:mds.all 12/12 14s ago 3m smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi116=b;smithi116=e;smithi116=h;smithi116=k;smithi118=c;smithi118=f;smithi118=i;smithi118=l;count:12 2024-09-19T06:51:27.312 INFO:teuthology.orchestra.run.smithi114.stdout:mgr 3/3 14s ago 8m smithi114=x;smithi116=y;smithi118=z;count:3 2024-09-19T06:51:27.312 INFO:teuthology.orchestra.run.smithi114.stdout:mon 3/3 14s ago 9m smithi114:172.21.15.114=a;smithi116:172.21.15.116=b;smithi118:172.21.15.118=c;count:3 2024-09-19T06:51:27.312 INFO:teuthology.orchestra.run.smithi114.stdout:osd 12 14s ago - 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbc340a32f0 msgr2=0x7fbc340a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbc340a32f0 0x7fbc340a57b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbc50199390 tx=0x7fbc38005f90 comp rx=0 tx=0).stop 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc50101c00 msgr2=0x7fbc50197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc50101c00 0x7fbc50197e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000e9c0 tx=0x7fbc4000ee80 comp rx=0 tx=0).stop 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 shutdown_connections 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fbc340a32f0 0x7fbc340a57b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc501037c0 0x7fbc5019f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc50102e00 0x7fbc501983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 --2- 172.21.15.114:0/1319417549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc50101c00 0x7fbc50197e70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 >> 172.21.15.114:0/1319417549 conn(0x7fbc500fd390 msgr2=0x7fbc50106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 shutdown_connections 2024-09-19T06:51:27.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:27.313+0000 7fbc54c00640 1 -- 172.21.15.114:0/1319417549 wait complete. 2024-09-19T06:51:27.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:27 smithi118 ceph-mon[35527]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:27 smithi116 ceph-mon[35411]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:28.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-19T06:51:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:28 smithi114 ceph-mon[30820]: pgmap v373: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-19T06:51:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:28 smithi114 ceph-mon[30820]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:28.467 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:28.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:28 smithi118 ceph-mon[35527]: pgmap v373: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-19T06:51:28.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:28 smithi118 ceph-mon[35527]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:28 smithi116 ceph-mon[35411]: pgmap v373: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-19T06:51:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:28 smithi116 ceph-mon[35411]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 -- 172.21.15.114:0/647564380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0968102c20 msgr2=0x7f0968105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 --2- 172.21.15.114:0/647564380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0968102c20 0x7f0968105090 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0954009900 tx=0x7f095402a410 comp rx=0 tx=0).stop 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 -- 172.21.15.114:0/647564380 shutdown_connections 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 --2- 172.21.15.114:0/647564380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09681055d0 0x7f0968105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 --2- 172.21.15.114:0/647564380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0968102c20 0x7f0968105090 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 --2- 172.21.15.114:0/647564380 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09681002d0 0x7f09681026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 -- 172.21.15.114:0/647564380 >> 172.21.15.114:0/647564380 conn(0x7f09680fc0e0 msgr2=0x7f09680fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 -- 172.21.15.114:0/647564380 shutdown_connections 2024-09-19T06:51:29.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.049+0000 7f096e200640 1 -- 172.21.15.114:0/647564380 wait complete. 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.050+0000 7f096e200640 1 Processor -- start 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.050+0000 7f096e200640 1 -- start start 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.050+0000 7f096e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09681002d0 0x7f096819e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f096e200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0968102c20 0x7f096819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f096e200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09681055d0 0x7f09681a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f096e200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0968113d30 con 0x7f09681002d0 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f096e200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0968113bb0 con 0x7f09681055d0 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f096e200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f0968113eb0 con 0x7f0968102c20 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f096ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09681002d0 0x7f096819e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f0967e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0968102c20 0x7f096819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f096d800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09681055d0 0x7f09681a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f0967e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0968102c20 0x7f096819e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:50782/0 (socket says 172.21.15.114:50782) 2024-09-19T06:51:29.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f096ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09681002d0 0x7f096819e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56052/0 (socket says 172.21.15.114:56052) 2024-09-19T06:51:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f0967e00640 1 -- 172.21.15.114:0/3083515323 learned_addr learned my addr 172.21.15.114:0/3083515323 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f0967e00640 1 -- 172.21.15.114:0/3083515323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09681055d0 msgr2=0x7f09681a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f0967e00640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09681055d0 0x7f09681a58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f0967e00640 1 -- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09681002d0 msgr2=0x7f096819e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f0967e00640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09681002d0 0x7f096819e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.051+0000 7f0967e00640 1 -- 172.21.15.114:0/3083515323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09681a5f10 con 0x7f0968102c20 2024-09-19T06:51:29.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.052+0000 7f0967e00640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0968102c20 0x7f096819e890 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0954031440 tx=0x7f0954032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.052+0000 7f0965600640 1 -- 172.21.15.114:0/3083515323 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095403f030 con 0x7f0968102c20 2024-09-19T06:51:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.052+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09681a61a0 con 0x7f0968102c20 2024-09-19T06:51:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.052+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09681a66d0 con 0x7f0968102c20 2024-09-19T06:51:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.052+0000 7f0965600640 1 -- 172.21.15.114:0/3083515323 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0954003ce0 con 0x7f0968102c20 2024-09-19T06:51:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.052+0000 7f0965600640 1 -- 172.21.15.114:0/3083515323 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954049060 con 0x7f0968102c20 2024-09-19T06:51:29.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.053+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09680fdbd0 con 0x7f0968102c20 2024-09-19T06:51:29.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.055+0000 7f0965600640 1 -- 172.21.15.114:0/3083515323 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f0954042030 con 0x7f0968102c20 2024-09-19T06:51:29.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.056+0000 7f0965600640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f09500a3270 0x7f09500a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:29.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.057+0000 7f0965600640 1 -- 172.21.15.114:0/3083515323 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f09540f02d0 con 0x7f0968102c20 2024-09-19T06:51:29.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.057+0000 7f096ce00640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f09500a3270 0x7f09500a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:29.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.057+0000 7f096ce00640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f09500a3270 0x7f09500a5730 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f095c009bd0 tx=0x7f095c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:29.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.058+0000 7f0965600640 1 -- 172.21.15.114:0/3083515323 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f09540bc550 con 0x7f0968102c20 2024-09-19T06:51:29.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.238+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f0968062630 con 0x7f09500a3270 2024-09-19T06:51:29.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.239+0000 7f0965600640 1 -- 172.21.15.114:0/3083515323 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f0968062630 con 0x7f09500a3270 2024-09-19T06:51:29.241 INFO:teuthology.orchestra.run.smithi114.stdout:HOST ADDR LABELS STATUS 2024-09-19T06:51:29.241 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 172.21.15.114 2024-09-19T06:51:29.241 INFO:teuthology.orchestra.run.smithi114.stdout:smithi116 172.21.15.116 2024-09-19T06:51:29.241 INFO:teuthology.orchestra.run.smithi114.stdout:smithi118 172.21.15.118 2024-09-19T06:51:29.241 INFO:teuthology.orchestra.run.smithi114.stdout:3 hosts in cluster 2024-09-19T06:51:29.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f09500a3270 msgr2=0x7f09500a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:29.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f09500a3270 0x7f09500a5730 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f095c009bd0 tx=0x7f095c005f90 comp rx=0 tx=0).stop 2024-09-19T06:51:29.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0968102c20 msgr2=0x7f096819e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0968102c20 0x7f096819e890 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0954031440 tx=0x7f0954032ce0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 shutdown_connections 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f09500a3270 0x7f09500a5730 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09681055d0 0x7f09681a58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0968102c20 0x7f096819e890 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 --2- 172.21.15.114:0/3083515323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f09681002d0 0x7f096819e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 >> 172.21.15.114:0/3083515323 conn(0x7f09680fc0e0 msgr2=0x7f0968100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 shutdown_connections 2024-09-19T06:51:29.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:29.242+0000 7f096e200640 1 -- 172.21.15.114:0/3083515323 wait complete. 2024-09-19T06:51:29.970 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-19T06:51:30.339 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:30.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:30 smithi114 ceph-mon[30820]: pgmap v374: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-19T06:51:30.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:30 smithi114 ceph-mon[30820]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:30.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:30 smithi118 ceph-mon[35527]: pgmap v374: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-19T06:51:30.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:30 smithi118 ceph-mon[35527]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:30 smithi116 ceph-mon[35411]: pgmap v374: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-19T06:51:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:30 smithi116 ceph-mon[35411]: from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:30.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 -- 172.21.15.114:0/1577745562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 msgr2=0x7f45cc074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:30.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 --2- 172.21.15.114:0/1577745562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 0x7f45cc074680 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009900 tx=0x7f45bc02a3f0 comp rx=0 tx=0).stop 2024-09-19T06:51:30.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 -- 172.21.15.114:0/1577745562 shutdown_connections 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 --2- 172.21.15.114:0/1577745562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45cc10e1e0 0x7f45cc1105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 --2- 172.21.15.114:0/1577745562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45cc074c50 0x7f45cc0750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 --2- 172.21.15.114:0/1577745562 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 0x7f45cc074680 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 -- 172.21.15.114:0/1577745562 >> 172.21.15.114:0/1577745562 conn(0x7f45cc0fd390 msgr2=0x7f45cc0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 -- 172.21.15.114:0/1577745562 shutdown_connections 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.907+0000 7f45d3200640 1 -- 172.21.15.114:0/1577745562 wait complete. 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.908+0000 7f45d3200640 1 Processor -- start 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.908+0000 7f45d3200640 1 -- start start 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45d3200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45cc074c50 0x7f45cc19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:30.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45d3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 0x7f45cc19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45d3200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45cc10e1e0 0x7f45cc1a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45d3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc113ea0 con 0x7f45cc076220 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45d3200640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc113d20 con 0x7f45cc10e1e0 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45d3200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc114020 con 0x7f45cc074c50 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45cb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 0x7f45cc19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45cb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 0x7f45cc19ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56076/0 (socket says 172.21.15.114:56076) 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45cb400640 1 -- 172.21.15.114:0/2588970060 learned_addr learned my addr 172.21.15.114:0/2588970060 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45cbe00640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45cc074c50 0x7f45cc19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.909+0000 7f45d1200640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45cc10e1e0 0x7f45cc1a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45cb400640 1 -- 172.21.15.114:0/2588970060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45cc074c50 msgr2=0x7f45cc19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45cb400640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45cc074c50 0x7f45cc19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45cb400640 1 -- 172.21.15.114:0/2588970060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45cc10e1e0 msgr2=0x7f45cc1a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:30.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45cb400640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45cc10e1e0 0x7f45cc1a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45cb400640 1 -- 172.21.15.114:0/2588970060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc1a6140 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45cbe00640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45cc074c50 0x7f45cc19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45cb400640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 0x7f45cc19ea50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f45b800e9f0 tx=0x7f45b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45c8c00640 1 -- 172.21.15.114:0/2588970060 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b800cd80 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45c8c00640 1 -- 172.21.15.114:0/2588970060 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45b8005500 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45cc1a6430 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45c8c00640 1 -- 172.21.15.114:0/2588970060 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8010690 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.910+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45cc1a6920 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.911+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45cc103b80 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.914+0000 7f45c8c00640 1 -- 172.21.15.114:0/2588970060 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f45b80046c0 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.915+0000 7f45c8c00640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f45b00a3270 0x7f45b00a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.915+0000 7f45c8c00640 1 -- 172.21.15.114:0/2588970060 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f45b80d0130 con 0x7f45cc076220 2024-09-19T06:51:30.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.915+0000 7f45cbe00640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f45b00a3270 0x7f45b00a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:30.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.916+0000 7f45cbe00640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f45b00a3270 0x7f45b00a5730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f45bc005ab0 tx=0x7f45bc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:30.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:30.919+0000 7f45c8c00640 1 -- 172.21.15.114:0/2588970060 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f45b809ca00 con 0x7f45cc076220 2024-09-19T06:51:31.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.100+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 --> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f45cc062630 con 0x7f45b00a3270 2024-09-19T06:51:31.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.102+0000 7f45c8c00640 1 -- 172.21.15.114:0/2588970060 <== mgr.14150 v2:172.21.15.114:6800/3586930729 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f45cc062630 con 0x7f45b00a3270 2024-09-19T06:51:31.103 INFO:teuthology.orchestra.run.smithi114.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-19T06:51:31.104 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-09-19T06:51:31.104 INFO:teuthology.orchestra.run.smithi114.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-19T06:51:31.104 INFO:teuthology.orchestra.run.smithi114.stdout:smithi118 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747500JS480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-09-19T06:51:31.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f45b00a3270 msgr2=0x7f45b00a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:31.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f45b00a3270 0x7f45b00a5730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f45bc005ab0 tx=0x7f45bc000bc0 comp rx=0 tx=0).stop 2024-09-19T06:51:31.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 msgr2=0x7f45cc19ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:31.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 0x7f45cc19ea50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f45b800e9f0 tx=0x7f45b800edf0 comp rx=0 tx=0).stop 2024-09-19T06:51:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 shutdown_connections 2024-09-19T06:51:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f45b00a3270 0x7f45b00a5730 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45cc10e1e0 0x7f45cc1a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f45cc076220 0x7f45cc19ea50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 --2- 172.21.15.114:0/2588970060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45cc074c50 0x7f45cc19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 >> 172.21.15.114:0/2588970060 conn(0x7f45cc0fd390 msgr2=0x7f45cc0ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 shutdown_connections 2024-09-19T06:51:31.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:31.105+0000 7f45d3200640 1 -- 172.21.15.114:0/2588970060 wait complete. 2024-09-19T06:51:31.913 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-19T06:51:31.923 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-19T06:51:31.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-19T06:51:32.298 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:32 smithi118 ceph-mon[35527]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:32 smithi118 ceph-mon[35527]: pgmap v375: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-19T06:51:32.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:32 smithi114 ceph-mon[30820]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:32.573 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:32 smithi114 ceph-mon[30820]: pgmap v375: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-19T06:51:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:32 smithi116 ceph-mon[35411]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-19T06:51:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:32 smithi116 ceph-mon[35411]: pgmap v375: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-19T06:51:32.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 -- 172.21.15.114:0/3060383962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8105810 msgr2=0x7fdae810c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:32.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 --2- 172.21.15.114:0/3060383962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8105810 0x7fdae810c0a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009ca0 tx=0x7fdae002a8c0 comp rx=0 tx=0).stop 2024-09-19T06:51:32.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 -- 172.21.15.114:0/3060383962 shutdown_connections 2024-09-19T06:51:32.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 --2- 172.21.15.114:0/3060383962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8105810 0x7fdae810c0a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:32.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 --2- 172.21.15.114:0/3060383962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae8104e50 0x7fdae81052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:32.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 --2- 172.21.15.114:0/3060383962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae8103c50 0x7fdae8104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:32.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 -- 172.21.15.114:0/3060383962 >> 172.21.15.114:0/3060383962 conn(0x7fdae80ff3e0 msgr2=0x7fdae8101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:32.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 -- 172.21.15.114:0/3060383962 shutdown_connections 2024-09-19T06:51:32.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.852+0000 7fdaecc00640 1 -- 172.21.15.114:0/3060383962 wait complete. 2024-09-19T06:51:32.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.853+0000 7fdaecc00640 1 Processor -- start 2024-09-19T06:51:32.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.853+0000 7fdaecc00640 1 -- start start 2024-09-19T06:51:32.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdaecc00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae8103c50 0x7fdae8199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:32.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdaecc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8104e50 0x7fdae819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:32.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdaecc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae8105810 0x7fdae81a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:32.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdaecc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdae810f900 con 0x7fdae8104e50 2024-09-19T06:51:32.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdaecc00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdae810f780 con 0x7fdae8103c50 2024-09-19T06:51:32.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdaecc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fdae810fa80 con 0x7fdae8105810 2024-09-19T06:51:32.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdae6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8104e50 0x7fdae819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:32.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdae6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8104e50 0x7fdae819a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56086/0 (socket says 172.21.15.114:56086) 2024-09-19T06:51:32.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdae6a00640 1 -- 172.21.15.114:0/1695899185 learned_addr learned my addr 172.21.15.114:0/1695899185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:32.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdae7400640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae8103c50 0x7fdae8199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:32.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.854+0000 7fdae7e00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae8105810 0x7fdae81a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:32.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdae6a00640 1 -- 172.21.15.114:0/1695899185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae8105810 msgr2=0x7fdae81a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:32.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdae6a00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae8105810 0x7fdae81a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:32.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdae6a00640 1 -- 172.21.15.114:0/1695899185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae8103c50 msgr2=0x7fdae8199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:32.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdae6a00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae8103c50 0x7fdae8199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:32.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdae6a00640 1 -- 172.21.15.114:0/1695899185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae81a1af0 con 0x7fdae8104e50 2024-09-19T06:51:32.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdae7400640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae8103c50 0x7fdae8199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:32.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdae6a00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8104e50 0x7fdae819a400 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdad400e9f0 tx=0x7fdad400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:32.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdadfe00640 1 -- 172.21.15.114:0/1695899185 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad400cd80 con 0x7fdae8104e50 2024-09-19T06:51:32.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdadfe00640 1 -- 172.21.15.114:0/1695899185 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdad4010910 con 0x7fdae8104e50 2024-09-19T06:51:32.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae81a1de0 con 0x7fdae8104e50 2024-09-19T06:51:32.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdadfe00640 1 -- 172.21.15.114:0/1695899185 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad4018990 con 0x7fdae8104e50 2024-09-19T06:51:32.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.855+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae81a22e0 con 0x7fdae8104e50 2024-09-19T06:51:32.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.857+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae810b000 con 0x7fdae8104e50 2024-09-19T06:51:32.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.859+0000 7fdadfe00640 1 -- 172.21.15.114:0/1695899185 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7fdad4010ac0 con 0x7fdae8104e50 2024-09-19T06:51:32.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.860+0000 7fdadfe00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdacc0a32f0 0x7fdacc0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:32.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.860+0000 7fdadfe00640 1 -- 172.21.15.114:0/1695899185 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fdad40cff20 con 0x7fdae8104e50 2024-09-19T06:51:32.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.860+0000 7fdae7400640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdacc0a32f0 0x7fdacc0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:32.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.861+0000 7fdae7400640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdacc0a32f0 0x7fdacc0a57b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdad8005fd0 tx=0x7fdad800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:32.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:32.864+0000 7fdadfe00640 1 -- 172.21.15.114:0/1695899185 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fdad409c1a0 con 0x7fdae8104e50 2024-09-19T06:51:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.073+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fdae8062df0 con 0x7fdae8104e50 2024-09-19T06:51:33.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.073+0000 7fdadfe00640 1 -- 172.21.15.114:0/1695899185 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) ==== 76+0+3146 (secure 0 0 0) 0x7fdad40a04a0 con 0x7fdae8104e50 2024-09-19T06:51:33.075 INFO:teuthology.orchestra.run.smithi114.stdout:e29 2024-09-19T06:51:33.075 INFO:teuthology.orchestra.run.smithi114.stdout:btime 2024-09-19T06:50:00:244614+0000 2024-09-19T06:51:33.075 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-19T06:51:33.075 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-19T06:51:33.075 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-09-19T06:51:33.075 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:33.075 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-09-19T06:51:33.075 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 29 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-09-19T06:49:11.450651+0000 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-09-19T06:49:59.255237+0000 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 300 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-09-19T06:51:33.076 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 0 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 5 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1,2,3,4 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=24538,1=24524,2=24520,3=24508,4=24482} 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3,4] 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data disabled 2024-09-19T06:51:33.077 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 0 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:qdb_cluster leader: 24538 members: 24482,24508,24520,24524,24538 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.i{0:24538} state up:active seq 8 addr [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.l{0:24460} state up:standby-replay seq 1 addr [v2:172.21.15.118:6832/3230062684,v1:172.21.15.118:6833/3230062684] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b{1:24524} state up:active seq 16 addr [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.e{1:24455} state up:standby-replay seq 1 addr [v2:172.21.15.116:6834/2147208941,v1:172.21.15.116:6835/2147208941] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c{2:24520} state up:active seq 20 addr [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.j{2:24431} state up:standby-replay seq 1 addr [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.f{3:24508} state up:active seq 21 addr [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.078 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.k{3:24419} state up:standby-replay seq 1 addr [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.079 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.h{4:24482} state up:active seq 20 addr [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.079 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a{4:14520} state up:standby-replay seq 1 addr [v2:172.21.15.114:6840/2459666351,v1:172.21.15.114:6841/2459666351] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:33.079 INFO:teuthology.orchestra.run.smithi114.stdout:Standby daemons: 2024-09-19T06:51:33.079 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:33.079 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d{-1:14469} state up:standby seq 1 addr [v2:172.21.15.114:6836/1558253171,v1:172.21.15.114:6837/1558253171] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.079 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.g{-1:14508} state up:standby seq 1 addr [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] compat {c=[1],r=[1],i=[1fff]}] 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdacc0a32f0 msgr2=0x7fdacc0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdacc0a32f0 0x7fdacc0a57b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdad8005fd0 tx=0x7fdad800b410 comp rx=0 tx=0).stop 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8104e50 msgr2=0x7fdae819a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8104e50 0x7fdae819a400 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdad400e9f0 tx=0x7fdad400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 shutdown_connections 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fdacc0a32f0 0x7fdacc0a57b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdae8105810 0x7fdae81a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:33.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdae8104e50 0x7fdae819a400 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:33.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 --2- 172.21.15.114:0/1695899185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae8103c50 0x7fdae8199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:33.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 >> 172.21.15.114:0/1695899185 conn(0x7fdae80ff3e0 msgr2=0x7fdae81081d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:33.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 shutdown_connections 2024-09-19T06:51:33.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:33.076+0000 7fdaecc00640 1 -- 172.21.15.114:0/1695899185 wait complete. 2024-09-19T06:51:33.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:33 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1695899185' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-19T06:51:33.509 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:33 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1695899185' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-19T06:51:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:33 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1695899185' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-19T06:51:33.891 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fface640-7651-11ef-bceb-c7b262605968 -- bash -c 'ceph osd dump' 2024-09-19T06:51:34.258 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:34 smithi118 ceph-mon[35527]: pgmap v376: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-19T06:51:34.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:34 smithi114 ceph-mon[30820]: pgmap v376: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-19T06:51:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:34 smithi116 ceph-mon[35411]: pgmap v376: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-19T06:51:34.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 -- 172.21.15.114:0/3039002042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70103e00 msgr2=0x7fab70104200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 --2- 172.21.15.114:0/3039002042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70103e00 0x7fab70104200 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fab60009900 tx=0x7fab6002a410 comp rx=0 tx=0).stop 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 -- 172.21.15.114:0/3039002042 shutdown_connections 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 --2- 172.21.15.114:0/3039002042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70105920 0x7fab7010c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 --2- 172.21.15.114:0/3039002042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab70104f60 0x7fab701053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 --2- 172.21.15.114:0/3039002042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70103e00 0x7fab70104200 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 -- 172.21.15.114:0/3039002042 >> 172.21.15.114:0/3039002042 conn(0x7fab700ff5b0 msgr2=0x7fab701019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 -- 172.21.15.114:0/3039002042 shutdown_connections 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.832+0000 7fab76600640 1 -- 172.21.15.114:0/3039002042 wait complete. 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.833+0000 7fab76600640 1 Processor -- start 2024-09-19T06:51:34.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab76600640 1 -- start start 2024-09-19T06:51:34.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab76600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70103e00 0x7fab7019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:34.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab76600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70104f60 0x7fab7019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:34.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab76600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab70105920 0x7fab701a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:34.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab76600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fab7010fb00 con 0x7fab70104f60 2024-09-19T06:51:34.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab76600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab7010f980 con 0x7fab70103e00 2024-09-19T06:51:34.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab76600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fab7010fc80 con 0x7fab70105920 2024-09-19T06:51:34.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab6ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70104f60 0x7fab7019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:34.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab6ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70104f60 0x7fab7019a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56098/0 (socket says 172.21.15.114:56098) 2024-09-19T06:51:34.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab6ea00640 1 -- 172.21.15.114:0/3260523116 learned_addr learned my addr 172.21.15.114:0/3260523116 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:34.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab6f400640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70103e00 0x7fab7019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:34.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.834+0000 7fab6fe00640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab70105920 0x7fab701a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:34.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab6ea00640 1 -- 172.21.15.114:0/3260523116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab70105920 msgr2=0x7fab701a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab6ea00640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab70105920 0x7fab701a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab6ea00640 1 -- 172.21.15.114:0/3260523116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70103e00 msgr2=0x7fab7019a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab6ea00640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70103e00 0x7fab7019a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab6ea00640 1 -- 172.21.15.114:0/3260523116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab701a1d00 con 0x7fab70104f60 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab6f400640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70103e00 0x7fab7019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab6ea00640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70104f60 0x7fab7019a610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fab5c00e9c0 tx=0x7fab5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab67e00640 1 -- 172.21.15.114:0/3260523116 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c00cd80 con 0x7fab70104f60 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab67e00640 1 -- 172.21.15.114:0/3260523116 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab5c0035f0 con 0x7fab70104f60 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab701a1ff0 con 0x7fab70104f60 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.835+0000 7fab67e00640 1 -- 172.21.15.114:0/3260523116 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c010690 con 0x7fab70104f60 2024-09-19T06:51:34.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.836+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab701a2450 con 0x7fab70104f60 2024-09-19T06:51:34.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.837+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7010b1a0 con 0x7fab70104f60 2024-09-19T06:51:34.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.839+0000 7fab67e00640 1 -- 172.21.15.114:0/3260523116 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7fab5c0037a0 con 0x7fab70104f60 2024-09-19T06:51:34.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.840+0000 7fab67e00640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fab540a3270 0x7fab540a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:34.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.840+0000 7fab67e00640 1 -- 172.21.15.114:0/3260523116 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fab5c0cfff0 con 0x7fab70104f60 2024-09-19T06:51:34.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.841+0000 7fab6f400640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fab540a3270 0x7fab540a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:34.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.841+0000 7fab6f400640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fab540a3270 0x7fab540a5730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab60031440 tx=0x7fab600315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:34.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:34.845+0000 7fab67e00640 1 -- 172.21.15.114:0/3260523116 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fab5c09c920 con 0x7fab70104f60 2024-09-19T06:51:35.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.020+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fab7002d7c0 con 0x7fab70104f60 2024-09-19T06:51:35.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.021+0000 7fab67e00640 1 -- 172.21.15.114:0/3260523116 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v95) ==== 56+0+5755 (secure 0 0 0) 0x7fab5c09c220 con 0x7fab70104f60 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 95 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-09-19T06:40:58.566770+0000 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-09-19T06:50:52.207228+0000 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:crush_version 27 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:full_ratio 0.95 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:backfillfull_ratio 0.9 2024-09-19T06:51:35.023 INFO:teuthology.orchestra.run.smithi114.stdout:nearfull_ratio 0.85 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:require_min_compat_client luminous 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:min_compat_client luminous 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:require_osd_release squid 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:stretch_mode_enabled false 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.69 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:max_osd 12 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6802/1641241463,v1:172.21.15.114:6803/1641241463] [v2:172.21.15.114:6804/1641241463,v1:172.21.15.114:6805/1641241463] exists,up 0a97979d-58ef-4b4b-a9a1-f703d05eca0d 2024-09-19T06:51:35.024 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6800/2939029063,v1:172.21.15.116:6801/2939029063] [v2:172.21.15.116:6802/2939029063,v1:172.21.15.116:6803/2939029063] exists,up 97cfe3c4-42fe-4873-a2ba-058f81d9c7d9 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 up in weight 1 up_from 18 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6800/657498996,v1:172.21.15.118:6801/657498996] [v2:172.21.15.118:6802/657498996,v1:172.21.15.118:6803/657498996] exists,up 6db87a1e-8734-4c62-86bb-b3c2297b7ef6 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6810/21390467,v1:172.21.15.114:6811/21390467] [v2:172.21.15.114:6812/21390467,v1:172.21.15.114:6813/21390467] exists,up d5c36094-adb2-4adb-8aa0-dd87da73028e 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 up in weight 1 up_from 33 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6808/2553198252,v1:172.21.15.116:6809/2553198252] [v2:172.21.15.116:6810/2553198252,v1:172.21.15.116:6811/2553198252] exists,up 3898d363-6848-46d1-b062-93850b994735 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 up in weight 1 up_from 38 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6808/2474489127,v1:172.21.15.118:6809/2474489127] [v2:172.21.15.118:6810/2474489127,v1:172.21.15.118:6811/2474489127] exists,up 7fd64637-ce99-4ea8-bee2-049d202c4720 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 up in weight 1 up_from 46 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6818/1336611047,v1:172.21.15.114:6819/1336611047] [v2:172.21.15.114:6820/1336611047,v1:172.21.15.114:6821/1336611047] exists,up cc797e6c-f130-4fd1-ba32-cb8a989cfd81 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6816/2839705948,v1:172.21.15.116:6817/2839705948] [v2:172.21.15.116:6818/2839705948,v1:172.21.15.116:6819/2839705948] exists,up 39e63a0c-eb89-434a-977e-2efd436ee732 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.8 up in weight 1 up_from 58 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6816/3182934697,v1:172.21.15.118:6817/3182934697] [v2:172.21.15.118:6818/3182934697,v1:172.21.15.118:6819/3182934697] exists,up 7c245abe-b78b-46a0-8d12-943863e93026 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.9 up in weight 1 up_from 66 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6826/2129482086,v1:172.21.15.114:6827/2129482086] [v2:172.21.15.114:6828/2129482086,v1:172.21.15.114:6829/2129482086] exists,up e29bdc9d-0f06-4b54-ba01-d15193a6ab0e 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.10 up in weight 1 up_from 71 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6824/1825955684,v1:172.21.15.116:6825/1825955684] [v2:172.21.15.116:6826/1825955684,v1:172.21.15.116:6827/1825955684] exists,up 12da1b00-07fc-49c3-96ec-450fd6568732 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:osd.11 up in weight 1 up_from 78 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6824/694640175,v1:172.21.15.118:6825/694640175] [v2:172.21.15.118:6826/694640175,v1:172.21.15.118:6827/694640175] exists,up b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.5 [8,11] 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.10 [2,4] 2024-09-19T06:51:35.025 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.12 [5,11] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.1a [8,1] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.1b [2,4] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.b [3,9] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.1d [4,6] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.1e [3,9] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.24 [4,9] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.28 [5,7] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.31 [4,8] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.6 [5,8] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.f [9,8] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.14 [9,8] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 4.18 [9,10] 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/3886387391 expires 2024-09-20T06:41:50.994561+0000 2024-09-19T06:51:35.026 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3653733983 expires 2024-09-20T06:41:50.994561+0000 2024-09-19T06:51:35.027 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1660437080 expires 2024-09-20T06:41:50.994561+0000 2024-09-19T06:51:35.027 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2975903254 expires 2024-09-20T06:41:50.994561+0000 2024-09-19T06:51:35.027 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/3244383942 expires 2024-09-20T06:41:23.468444+0000 2024-09-19T06:51:35.027 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/3886387391 expires 2024-09-20T06:41:50.994561+0000 2024-09-19T06:51:35.027 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/967314322 expires 2024-09-20T06:41:23.468444+0000 2024-09-19T06:51:35.027 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/2384341327 expires 2024-09-20T06:41:23.468444+0000 2024-09-19T06:51:35.027 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/2384341327 expires 2024-09-20T06:41:23.468444+0000 2024-09-19T06:51:35.027 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1724858398 expires 2024-09-20T06:41:23.468444+0000 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fab540a3270 msgr2=0x7fab540a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fab540a3270 0x7fab540a5730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab60031440 tx=0x7fab600315b0 comp rx=0 tx=0).stop 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70104f60 msgr2=0x7fab7019a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70104f60 0x7fab7019a610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fab5c00e9c0 tx=0x7fab5c00ee80 comp rx=0 tx=0).stop 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 shutdown_connections 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fab540a3270 0x7fab540a5730 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab70105920 0x7fab701a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fab70104f60 0x7fab7019a610 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 --2- 172.21.15.114:0/3260523116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab70103e00 0x7fab7019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 >> 172.21.15.114:0/3260523116 conn(0x7fab700ff5b0 msgr2=0x7fab70108370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:35.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 shutdown_connections 2024-09-19T06:51:35.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:35.024+0000 7fab76600640 1 -- 172.21.15.114:0/3260523116 wait complete. 2024-09-19T06:51:35.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:35 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3260523116' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-19T06:51:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:35 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3260523116' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-19T06:51:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:35 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3260523116' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-19T06:51:35.754 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-19T06:51:35.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:51:36.133 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:36.493 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:36 smithi114 ceph-mon[30820]: pgmap v377: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-19T06:51:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:36 smithi118 ceph-mon[35527]: pgmap v377: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-19T06:51:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:36 smithi116 ceph-mon[35411]: pgmap v377: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-09-19T06:51:36.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.720+0000 7fea3ba00640 1 -- 172.21.15.114:0/2369657441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074620 msgr2=0x7fea34074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:36.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.720+0000 7fea3ba00640 1 --2- 172.21.15.114:0/2369657441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074620 0x7fea34074a00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fea28009900 tx=0x7fea2802a480 comp rx=0 tx=0).stop 2024-09-19T06:51:36.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.721+0000 7fea3ba00640 1 -- 172.21.15.114:0/2369657441 shutdown_connections 2024-09-19T06:51:36.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.721+0000 7fea3ba00640 1 --2- 172.21.15.114:0/2369657441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea3410e3c0 0x7fea34110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.721+0000 7fea3ba00640 1 --2- 172.21.15.114:0/2369657441 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea34074f40 0x7fea3410de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.721+0000 7fea3ba00640 1 --2- 172.21.15.114:0/2369657441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074620 0x7fea34074a00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.721+0000 7fea3ba00640 1 -- 172.21.15.114:0/2369657441 >> 172.21.15.114:0/2369657441 conn(0x7fea340fd8c0 msgr2=0x7fea340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:36.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.721+0000 7fea3ba00640 1 -- 172.21.15.114:0/2369657441 shutdown_connections 2024-09-19T06:51:36.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.721+0000 7fea3ba00640 1 -- 172.21.15.114:0/2369657441 wait complete. 2024-09-19T06:51:36.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea3ba00640 1 Processor -- start 2024-09-19T06:51:36.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea3ba00640 1 -- start start 2024-09-19T06:51:36.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea3ba00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea34074620 0x7fea3419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea3ba00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074f40 0x7fea3419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea3ba00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea3410e3c0 0x7fea3419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea3ba00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fea341141c0 con 0x7fea34074f40 2024-09-19T06:51:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea3ba00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea34114040 con 0x7fea3410e3c0 2024-09-19T06:51:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea3ba00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fea34114340 con 0x7fea34074620 2024-09-19T06:51:36.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea33e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074f40 0x7fea3419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:36.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea39000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea34074620 0x7fea3419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:36.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea39000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea34074620 0x7fea3419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:50850/0 (socket says 172.21.15.114:50850) 2024-09-19T06:51:36.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea39000640 1 -- 172.21.15.114:0/3479328163 learned_addr learned my addr 172.21.15.114:0/3479328163 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:36.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.722+0000 7fea33e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074f40 0x7fea3419abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56110/0 (socket says 172.21.15.114:56110) 2024-09-19T06:51:36.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.723+0000 7fea39a00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea3410e3c0 0x7fea3419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:36.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.723+0000 7fea33e00640 1 -- 172.21.15.114:0/3479328163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea34074620 msgr2=0x7fea3419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.723+0000 7fea33e00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea34074620 0x7fea3419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.723+0000 7fea33e00640 1 -- 172.21.15.114:0/3479328163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea3410e3c0 msgr2=0x7fea3419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.723+0000 7fea33e00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea3410e3c0 0x7fea3419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.723+0000 7fea33e00640 1 -- 172.21.15.114:0/3479328163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3419f650 con 0x7fea34074f40 2024-09-19T06:51:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.723+0000 7fea39000640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea34074620 0x7fea3419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:36.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.723+0000 7fea33e00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074f40 0x7fea3419abe0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fea2000e9f0 tx=0x7fea2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.724+0000 7fea31600640 1 -- 172.21.15.114:0/3479328163 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2000ccc0 con 0x7fea34074f40 2024-09-19T06:51:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.724+0000 7fea31600640 1 -- 172.21.15.114:0/3479328163 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea20005020 con 0x7fea34074f40 2024-09-19T06:51:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.724+0000 7fea31600640 1 -- 172.21.15.114:0/3479328163 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea200049e0 con 0x7fea34074f40 2024-09-19T06:51:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.724+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea3419f940 con 0x7fea34074f40 2024-09-19T06:51:36.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.724+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea341a7220 con 0x7fea34074f40 2024-09-19T06:51:36.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.725+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea34074a00 con 0x7fea34074f40 2024-09-19T06:51:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.727+0000 7fea31600640 1 -- 172.21.15.114:0/3479328163 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7fea20005560 con 0x7fea34074f40 2024-09-19T06:51:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.728+0000 7fea31600640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fea1c0a3240 0x7fea1c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:36.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.728+0000 7fea31600640 1 -- 172.21.15.114:0/3479328163 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fea200cfce0 con 0x7fea34074f40 2024-09-19T06:51:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.729+0000 7fea39000640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fea1c0a3240 0x7fea1c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.729+0000 7fea39000640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fea1c0a3240 0x7fea1c0a5700 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fea280096c0 tx=0x7fea280046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:36.734 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.732+0000 7fea31600640 1 -- 172.21.15.114:0/3479328163 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7fea200051d0 con 0x7fea34074f40 2024-09-19T06:51:36.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.943+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea3402d7c0 con 0x7fea34074f40 2024-09-19T06:51:36.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.944+0000 7fea31600640 1 -- 172.21.15.114:0/3479328163 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11999 (secure 0 0 0) 0x7fea2009c5b0 con 0x7fea34074f40 2024-09-19T06:51:36.946 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:36.946 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"btime":"2024-09-19T06:50:00:244614+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T06:49:11.450651+0000","modified":"2024-09-19T06:49:59.255237+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24538,"mds_1":24524,"mds_2":24520,"mds_3":24508,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14520":{"gid":14520,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":21,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":20,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24538,"qdb_cluster":[24482,24508,24520,24524,24538]},"id":1}]} 2024-09-19T06:51:36.946 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.946+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fea1c0a3240 msgr2=0x7fea1c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.946+0000 7fea3ba00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fea1c0a3240 0x7fea1c0a5700 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fea280096c0 tx=0x7fea280046c0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.946+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074f40 msgr2=0x7fea3419abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.946+0000 7fea3ba00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074f40 0x7fea3419abe0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fea2000e9f0 tx=0x7fea2000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.947+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 shutdown_connections 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.947+0000 7fea3ba00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fea1c0a3240 0x7fea1c0a5700 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.947+0000 7fea3ba00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea3410e3c0 0x7fea3419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.947+0000 7fea3ba00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fea34074f40 0x7fea3419abe0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.947+0000 7fea3ba00640 1 --2- 172.21.15.114:0/3479328163 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fea34074620 0x7fea3419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:36.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.947+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 >> 172.21.15.114:0/3479328163 conn(0x7fea340fd8c0 msgr2=0x7fea340fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:36.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.947+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 shutdown_connections 2024-09-19T06:51:36.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:36.947+0000 7fea3ba00640 1 -- 172.21.15.114:0/3479328163 wait complete. 2024-09-19T06:51:37.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:37 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3479328163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:37 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3479328163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:37 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3479328163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:37.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-19T06:51:38.096 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:38.518 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:38 smithi118 ceph-mon[35527]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-19T06:51:38.518 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:38 smithi114 ceph-mon[30820]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-19T06:51:38.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:38 smithi116 ceph-mon[35411]: pgmap v378: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 -- 172.21.15.114:0/2761227231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e81049a0 msgr2=0x7f31e81089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 --2- 172.21.15.114:0/2761227231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e81049a0 0x7f31e81089f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009900 tx=0x7f31d002a400 comp rx=0 tx=0).stop 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 -- 172.21.15.114:0/2761227231 shutdown_connections 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 --2- 172.21.15.114:0/2761227231 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31e8109120 0x7f31e8110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 --2- 172.21.15.114:0/2761227231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e81049a0 0x7f31e81089f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 --2- 172.21.15.114:0/2761227231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e8103ff0 0x7f31e81043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 -- 172.21.15.114:0/2761227231 >> 172.21.15.114:0/2761227231 conn(0x7f31e80ffa90 msgr2=0x7f31e8101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 -- 172.21.15.114:0/2761227231 shutdown_connections 2024-09-19T06:51:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.645+0000 7f31eda00640 1 -- 172.21.15.114:0/2761227231 wait complete. 2024-09-19T06:51:38.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.646+0000 7f31eda00640 1 Processor -- start 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.646+0000 7f31eda00640 1 -- start start 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31eda00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31e8103ff0 0x7f31e819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31eda00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e81049a0 0x7f31e819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31eda00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e8109120 0x7f31e819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31eda00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f31e8114190 con 0x7f31e8109120 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31eda00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31e8114010 con 0x7f31e81049a0 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31eda00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f31e8114310 con 0x7f31e8103ff0 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31dfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e81049a0 0x7f31e819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31e7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e8109120 0x7f31e819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31dfe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e81049a0 0x7f31e819ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:43324/0 (socket says 172.21.15.114:43324) 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31dfe00640 1 -- 172.21.15.114:0/1097435346 learned_addr learned my addr 172.21.15.114:0/1097435346 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:38.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31e6a00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31e8103ff0 0x7f31e819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31e7400640 1 -- 172.21.15.114:0/1097435346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31e8103ff0 msgr2=0x7f31e819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31e7400640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31e8103ff0 0x7f31e819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31e7400640 1 -- 172.21.15.114:0/1097435346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e81049a0 msgr2=0x7f31e819ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31e7400640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e81049a0 0x7f31e819ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.647+0000 7f31e7400640 1 -- 172.21.15.114:0/1097435346 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e819f5a0 con 0x7f31e8109120 2024-09-19T06:51:38.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.648+0000 7f31e7400640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e8109120 0x7f31e819eec0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009fd0 tx=0x7f31e000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.648+0000 7f31df400640 1 -- 172.21.15.114:0/1097435346 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e000ce20 con 0x7f31e8109120 2024-09-19T06:51:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.648+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e819f890 con 0x7f31e8109120 2024-09-19T06:51:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.648+0000 7f31df400640 1 -- 172.21.15.114:0/1097435346 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31e0005020 con 0x7f31e8109120 2024-09-19T06:51:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.648+0000 7f31df400640 1 -- 172.21.15.114:0/1097435346 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e0010720 con 0x7f31e8109120 2024-09-19T06:51:38.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.648+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e81a7170 con 0x7f31e8109120 2024-09-19T06:51:38.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.649+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e81043d0 con 0x7f31e8109120 2024-09-19T06:51:38.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.651+0000 7f31df400640 1 -- 172.21.15.114:0/1097435346 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f31e0005540 con 0x7f31e8109120 2024-09-19T06:51:38.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.653+0000 7f31df400640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f31cc0a3240 0x7f31cc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:38.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.653+0000 7f31df400640 1 -- 172.21.15.114:0/1097435346 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f31e00d02f0 con 0x7f31e8109120 2024-09-19T06:51:38.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.653+0000 7f31e6a00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f31cc0a3240 0x7f31cc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:38.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.654+0000 7f31e6a00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f31cc0a3240 0x7f31cc0a5700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009bd0 tx=0x7f31d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:38.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.657+0000 7f31df400640 1 -- 172.21.15.114:0/1097435346 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f31e009cbc0 con 0x7f31e8109120 2024-09-19T06:51:38.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.875+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31e81a7410 con 0x7f31e8109120 2024-09-19T06:51:38.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.876+0000 7f31df400640 1 -- 172.21.15.114:0/1097435346 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11999 (secure 0 0 0) 0x7f31e009c4c0 con 0x7f31e8109120 2024-09-19T06:51:38.877 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:38.878 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":29,"btime":"2024-09-19T06:50:00:244614+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14469,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1558253171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1558253171},{"type":"v1","addr":"172.21.15.114:6837","nonce":1558253171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14508,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3315910714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3315910714},{"type":"v1","addr":"172.21.15.114:6839","nonce":3315910714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-19T06:49:11.450651+0000","modified":"2024-09-19T06:49:59.255237+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24538,"mds_1":24524,"mds_2":24520,"mds_3":24508,"mds_4":24482},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14520":{"gid":14520,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6841/2459666351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":2459666351},{"type":"v1","addr":"172.21.15.114:6841","nonce":2459666351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24419":{"gid":24419,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6833/1884054485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1884054485},{"type":"v1","addr":"172.21.15.116:6833","nonce":1884054485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2797655862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2797655862},{"type":"v1","addr":"172.21.15.114:6835","nonce":2797655862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6835/2147208941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2147208941},{"type":"v1","addr":"172.21.15.116:6835","nonce":2147208941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6833/3230062684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3230062684},{"type":"v1","addr":"172.21.15.118:6833","nonce":3230062684}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24482":{"gid":24482,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6837/2673389539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2673389539},{"type":"v1","addr":"172.21.15.116:6837","nonce":2673389539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":24,"state":"up:active","state_seq":21,"addr":"172.21.15.118:6835/449495659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":449495659},{"type":"v1","addr":"172.21.15.118:6835","nonce":449495659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":20,"addr":"172.21.15.118:6837/4109230511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":4109230511},{"type":"v1","addr":"172.21.15.118:6837","nonce":4109230511}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.116:6839/2386737584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":2386737584},{"type":"v1","addr":"172.21.15.116:6839","nonce":2386737584}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24538":{"gid":24538,"name":"i","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.118:6839/345383252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":345383252},{"type":"v1","addr":"172.21.15.118:6839","nonce":345383252}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24538,"qdb_cluster":[24482,24508,24520,24524,24538]},"id":1}]} 2024-09-19T06:51:38.878 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 29 2024-09-19T06:51:38.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f31cc0a3240 msgr2=0x7f31cc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:38.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f31cc0a3240 0x7f31cc0a5700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009bd0 tx=0x7f31d8005f90 comp rx=0 tx=0).stop 2024-09-19T06:51:38.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e8109120 msgr2=0x7f31e819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:38.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e8109120 0x7f31e819eec0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009fd0 tx=0x7f31e000ef10 comp rx=0 tx=0).stop 2024-09-19T06:51:38.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 shutdown_connections 2024-09-19T06:51:38.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f31cc0a3240 0x7f31cc0a5700 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f31e8109120 0x7f31e819eec0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e81049a0 0x7f31e819ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 --2- 172.21.15.114:0/1097435346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31e8103ff0 0x7f31e819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 >> 172.21.15.114:0/1097435346 conn(0x7f31e80ffa90 msgr2=0x7f31e8101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 shutdown_connections 2024-09-19T06:51:38.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:38.879+0000 7f31eda00640 1 -- 172.21.15.114:0/1097435346 wait complete. 2024-09-19T06:51:39.337 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:39 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/1097435346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:39.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:39 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/1097435346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:39 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/1097435346' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-19T06:51:39.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-19T06:51:40.128 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:40.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:40 smithi118 ceph-mon[35527]: pgmap v379: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:40.555 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:40 smithi114 ceph-mon[30820]: pgmap v379: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 -- 172.21.15.114:0/3032764632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 msgr2=0x7f4acc104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/3032764632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 0x7f4acc104210 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4abc009900 tx=0x7f4abc02a3f0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 -- 172.21.15.114:0/3032764632 shutdown_connections 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/3032764632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4acc108f60 0x7f4acc110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/3032764632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4acc1047e0 0x7f4acc108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/3032764632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 0x7f4acc104210 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 -- 172.21.15.114:0/3032764632 >> 172.21.15.114:0/3032764632 conn(0x7f4acc0ff8d0 msgr2=0x7f4acc101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 -- 172.21.15.114:0/3032764632 shutdown_connections 2024-09-19T06:51:40.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.664+0000 7f4ad0c00640 1 -- 172.21.15.114:0/3032764632 wait complete. 2024-09-19T06:51:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.665+0000 7f4ad0c00640 1 Processor -- start 2024-09-19T06:51:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.665+0000 7f4ad0c00640 1 -- start start 2024-09-19T06:51:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ad0c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 0x7f4acc19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ad0c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4acc1047e0 0x7f4acc19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:40.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ad0c00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4acc108f60 0x7f4acc19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ad0c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc113fc0 con 0x7f4acc103e30 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ad0c00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc113e40 con 0x7f4acc108f60 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ad0c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc114140 con 0x7f4acc1047e0 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4acb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 0x7f4acc19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4acb400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 0x7f4acc19a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43858/0 (socket says 172.21.15.114:43858) 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4acb400640 1 -- 172.21.15.114:0/795491185 learned_addr learned my addr 172.21.15.114:0/795491185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ac3e00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4acc1047e0 0x7f4acc19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4acbe00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4acc108f60 0x7f4acc19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ac3e00640 1 -- 172.21.15.114:0/795491185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4acc108f60 msgr2=0x7f4acc19ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ac3e00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4acc108f60 0x7f4acc19ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ac3e00640 1 -- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 msgr2=0x7f4acc19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ac3e00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 0x7f4acc19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.666+0000 7f4ac3e00640 1 -- 172.21.15.114:0/795491185 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc19f460 con 0x7f4acc1047e0 2024-09-19T06:51:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.667+0000 7f4ac3e00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4acc1047e0 0x7f4acc19a9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400e9f0 tx=0x7f4ab400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.667+0000 7f4ac8c00640 1 -- 172.21.15.114:0/795491185 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab400ccc0 con 0x7f4acc1047e0 2024-09-19T06:51:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.667+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4acc19f750 con 0x7f4acc1047e0 2024-09-19T06:51:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.667+0000 7f4ac8c00640 1 -- 172.21.15.114:0/795491185 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ab4005020 con 0x7f4acc1047e0 2024-09-19T06:51:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.667+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4acc1a7030 con 0x7f4acc1047e0 2024-09-19T06:51:40.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.667+0000 7f4ac8c00640 1 -- 172.21.15.114:0/795491185 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab401c070 con 0x7f4acc1047e0 2024-09-19T06:51:40.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.668+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4acc104210 con 0x7f4acc1047e0 2024-09-19T06:51:40.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.671+0000 7f4ac8c00640 1 -- 172.21.15.114:0/795491185 <== mon.1 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f4ab4005540 con 0x7f4acc1047e0 2024-09-19T06:51:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.672+0000 7f4ac8c00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4ab00a3240 0x7f4ab00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.672+0000 7f4ac8c00640 1 -- 172.21.15.114:0/795491185 <== mon.1 v2:172.21.15.118:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f4ab40cfd10 con 0x7f4acc1047e0 2024-09-19T06:51:40.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.672+0000 7f4acb400640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4ab00a3240 0x7f4ab00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:40.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.673+0000 7f4acb400640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4ab00a3240 0x7f4ab00a5700 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0096c0 tx=0x7f4abc005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:40.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.676+0000 7f4ac8c00640 1 -- 172.21.15.114:0/795491185 <== mon.1 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f4ab40d5020 con 0x7f4acc1047e0 2024-09-19T06:51:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:40 smithi116 ceph-mon[35411]: pgmap v379: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:40.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.850+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4acc10bbb0 con 0x7f4acc1047e0 2024-09-19T06:51:40.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.852+0000 7f4ac8c00640 1 -- 172.21.15.114:0/795491185 <== mon.1 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+25089 (secure 0 0 0) 0x7f4ab409c5e0 con 0x7f4acc1047e0 2024-09-19T06:51:40.854 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-09-19T06:51:40.854 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":95,"fsid":"fface640-7651-11ef-bceb-c7b262605968","created":"2024-09-19T06:40:58.566770+0000","modified":"2024-09-19T06:50:52.207228+0000","last_up_change":"2024-09-19T06:48:15.289193+0000","last_in_change":"2024-09-19T06:47:52.148145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-19T06:44:13.612575+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-19T06:49:06.468527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-19T06:49:09.344570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-19T06:49:18.899288+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a97979d-58ef-4b4b-a9a1-f703d05eca0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6803","nonce":1641241463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6805","nonce":1641241463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6809","nonce":1641241463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1641241463},{"type":"v1","addr":"172.21.15.114:6807","nonce":1641241463}]},"public_addr":"172.21.15.114:6803/1641241463","cluster_addr":"172.21.15.114:6805/1641241463","heartbeat_back_addr":"172.21.15.114:6809/1641241463","heartbeat_front_addr":"172.21.15.114:6807/1641241463","state":["exists","up"]},{"osd":1,"uuid":"97cfe3c4-42fe-4873-a2ba-058f81d9c7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6801","nonce":2939029063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6803","nonce":2939029063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6807","nonce":2939029063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2939029063},{"type":"v1","addr":"172.21.15.116:6805","nonce":2939029063}]},"public_addr":"172.21.15.116:6801/2939029063","cluster_addr":"172.21.15.116:6803/2939029063","heartbeat_back_addr":"172.21.15.116:6807/2939029063","heartbeat_front_addr":"172.21.15.116:6805/2939029063","state":["exists","up"]},{"osd":2,"uuid":"6db87a1e-8734-4c62-86bb-b3c2297b7ef6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6801","nonce":657498996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6803","nonce":657498996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6807","nonce":657498996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":657498996},{"type":"v1","addr":"172.21.15.118:6805","nonce":657498996}]},"public_addr":"172.21.15.118:6801/657498996","cluster_addr":"172.21.15.118:6803/657498996","heartbeat_back_addr":"172.21.15.118:6807/657498996","heartbeat_front_addr":"172.21.15.118:6805/657498996","state":["exists","up"]},{"osd":3,"uuid":"d5c36094-adb2-4adb-8aa0-dd87da73028e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6811","nonce":21390467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6813","nonce":21390467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6817","nonce":21390467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":21390467},{"type":"v1","addr":"172.21.15.114:6815","nonce":21390467}]},"public_addr":"172.21.15.114:6811/21390467","cluster_addr":"172.21.15.114:6813/21390467","heartbeat_back_addr":"172.21.15.114:6817/21390467","heartbeat_front_addr":"172.21.15.114:6815/21390467","state":["exists","up"]},{"osd":4,"uuid":"3898d363-6848-46d1-b062-93850b994735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6809","nonce":2553198252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6811","nonce":2553198252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6815","nonce":2553198252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2553198252},{"type":"v1","addr":"172.21.15.116:6813","nonce":2553198252}]},"public_addr":"172.21.15.116:6809/2553198252","cluster_addr":"172.21.15.116:6811/2553198252","heartbeat_back_addr":"172.21.15.116:6815/2553198252","heartbeat_front_addr":"172.21.15.116:6813/2553198252","state":["exists","up"]},{"osd":5,"uuid":"7fd64637-ce99-4ea8-bee2-049d202c4720","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6809","nonce":2474489127}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6811","nonce":2474489127}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6815","nonce":2474489127}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2474489127},{"type":"v1","addr":"172.21.15.118:6813","nonce":2474489127}]},"public_addr":"172.21.15.118:6809/2474489127","cluster_addr":"172.21.15.118:6811/2474489127","heartbeat_back_addr":"172.21.15.118:6815/2474489127","heartbeat_front_addr":"172.21.15.118:6813/2474489127","state":["exists","up"]},{"osd":6,"uuid":"cc797e6c-f130-4fd1-ba32-cb8a989cfd81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6819","nonce":1336611047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6821","nonce":1336611047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6825","nonce":1336611047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1336611047},{"type":"v1","addr":"172.21.15.114:6823","nonce":1336611047}]},"public_addr":"172.21.15.114:6819/1336611047","cluster_addr":"172.21.15.114:6821/1336611047","heartbeat_back_addr":"172.21.15.114:6825/1336611047","heartbeat_front_addr":"172.21.15.114:6823/1336611047","state":["exists","up"]},{"osd":7,"uuid":"39e63a0c-eb89-434a-977e-2efd436ee732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6817","nonce":2839705948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6819","nonce":2839705948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6823","nonce":2839705948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2839705948},{"type":"v1","addr":"172.21.15.116:6821","nonce":2839705948}]},"public_addr":"172.21.15.116:6817/2839705948","cluster_addr":"172.21.15.116:6819/2839705948","heartbeat_back_addr":"172.21.15.116:6823/2839705948","heartbeat_front_addr":"172.21.15.116:6821/2839705948","state":["exists","up"]},{"osd":8,"uuid":"7c245abe-b78b-46a0-8d12-943863e93026","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6817","nonce":3182934697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6819","nonce":3182934697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6823","nonce":3182934697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3182934697},{"type":"v1","addr":"172.21.15.118:6821","nonce":3182934697}]},"public_addr":"172.21.15.118:6817/3182934697","cluster_addr":"172.21.15.118:6819/3182934697","heartbeat_back_addr":"172.21.15.118:6823/3182934697","heartbeat_front_addr":"172.21.15.118:6821/3182934697","state":["exists","up"]},{"osd":9,"uuid":"e29bdc9d-0f06-4b54-ba01-d15193a6ab0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6827","nonce":2129482086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6829","nonce":2129482086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6833","nonce":2129482086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2129482086},{"type":"v1","addr":"172.21.15.114:6831","nonce":2129482086}]},"public_addr":"172.21.15.114:6827/2129482086","cluster_addr":"172.21.15.114:6829/2129482086","heartbeat_back_addr":"172.21.15.114:6833/2129482086","heartbeat_front_addr":"172.21.15.114:6831/2129482086","state":["exists","up"]},{"osd":10,"uuid":"12da1b00-07fc-49c3-96ec-450fd6568732","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6825","nonce":1825955684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6827","nonce":1825955684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6831","nonce":1825955684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1825955684},{"type":"v1","addr":"172.21.15.116:6829","nonce":1825955684}]},"public_addr":"172.21.15.116:6825/1825955684","cluster_addr":"172.21.15.116:6827/1825955684","heartbeat_back_addr":"172.21.15.116:6831/1825955684","heartbeat_front_addr":"172.21.15.116:6829/1825955684","state":["exists","up"]},{"osd":11,"uuid":"b5ee8b7a-7aa4-4520-87d0-141e1fb8ab71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6825","nonce":694640175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6827","nonce":694640175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6831","nonce":694640175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":694640175},{"type":"v1","addr":"172.21.15.118:6829","nonce":694640175}]},"public_addr":"172.21.15.118:6825/694640175","cluster_addr":"172.21.15.118:6827/694640175","heartbeat_back_addr":"172.21.15.118:6831/694640175","heartbeat_front_addr":"172.21.15.118:6829/694640175","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:20.401709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:43:45.581120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:10.284688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:44:40.350948+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:06.772512+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:45:32.279704+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:01.952498+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:27.699913+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:46:53.027210+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:22.774816+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:47:48.436980+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-19T06:48:13.672050+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3653733983":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/1660437080":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/2975903254":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/3244383942":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/3886387391":"2024-09-20T06:41:50.994561+0000","172.21.15.114:0/967314322":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6801/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:6800/2384341327":"2024-09-20T06:41:23.468444+0000","172.21.15.114:0/1724858398":"2024-09-20T06:41:23.468444+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.854+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4ab00a3240 msgr2=0x7f4ab00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.854+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4ab00a3240 0x7f4ab00a5700 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0096c0 tx=0x7f4abc005a20 comp rx=0 tx=0).stop 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.854+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4acc1047e0 msgr2=0x7f4acc19a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.854+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4acc1047e0 0x7f4acc19a9f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400e9f0 tx=0x7f4ab400edf0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.855+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 shutdown_connections 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.855+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f4ab00a3240 0x7f4ab00a5700 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.855+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4acc108f60 0x7f4acc19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.855+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4acc1047e0 0x7f4acc19a9f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.855+0000 7f4ad0c00640 1 --2- 172.21.15.114:0/795491185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4acc103e30 0x7f4acc19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.855+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 >> 172.21.15.114:0/795491185 conn(0x7f4acc0ff8d0 msgr2=0x7f4acc100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.855+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 shutdown_connections 2024-09-19T06:51:40.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:40.855+0000 7f4ad0c00640 1 -- 172.21.15.114:0/795491185 wait complete. 2024-09-19T06:51:41.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:41 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/795491185' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:51:41.634 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:41 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/795491185' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:51:41.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-19T06:49:11.450651+0000', 'modified': '2024-09-19T06:49:59.255237+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24538, 'mds_1': 24524, 'mds_2': 24520, 'mds_3': 24508, 'mds_4': 24482}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14520': {'gid': 14520, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6841/2459666351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6840', 'nonce': 2459666351}, {'type': 'v1', 'addr': '172.21.15.114:6841', 'nonce': 2459666351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24419': {'gid': 24419, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6833/1884054485', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6832', 'nonce': 1884054485}, {'type': 'v1', 'addr': '172.21.15.116:6833', 'nonce': 1884054485}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/2797655862', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 2797655862}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 2797655862}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6835/2147208941', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 2147208941}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 2147208941}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'l', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.118:6833/3230062684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6832', 'nonce': 3230062684}, {'type': 'v1', 'addr': '172.21.15.118:6833', 'nonce': 3230062684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24482': {'gid': 24482, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.116:6837/2673389539', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 2673389539}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 2673389539}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 21, 'addr': '172.21.15.118:6835/449495659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6834', 'nonce': 449495659}, {'type': 'v1', 'addr': '172.21.15.118:6835', 'nonce': 449495659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.118:6837/4109230511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6836', 'nonce': 4109230511}, {'type': 'v1', 'addr': '172.21.15.118:6837', 'nonce': 4109230511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.116:6839/2386737584', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 2386737584}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 2386737584}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24538': {'gid': 24538, 'name': 'i', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.118:6839/345383252', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6838', 'nonce': 345383252}, {'type': 'v1', 'addr': '172.21.15.118:6839', 'nonce': 345383252}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24538, 'qdb_cluster': [24482, 24508, 24520, 24524, 24538]} 2024-09-19T06:51:41.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-19T06:51:41.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-19T06:51:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:41 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/795491185' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-19T06:51:42.039 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:42 smithi118 ceph-mon[35527]: pgmap v380: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:42.515 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:42 smithi114 ceph-mon[30820]: pgmap v380: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:42.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 -- 172.21.15.114:0/1269658433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 msgr2=0x7f8f24102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 --2- 172.21.15.114:0/1269658433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 0x7f8f24102f70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8f18009ca0 tx=0x7f8f1802a950 comp rx=0 tx=0).stop 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 -- 172.21.15.114:0/1269658433 shutdown_connections 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 --2- 172.21.15.114:0/1269658433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f2410bfc0 0x7f8f2410e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 --2- 172.21.15.114:0/1269658433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f241034b0 0x7f8f2410b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 --2- 172.21.15.114:0/1269658433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 0x7f8f24102f70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 -- 172.21.15.114:0/1269658433 >> 172.21.15.114:0/1269658433 conn(0x7f8f240fc610 msgr2=0x7f8f240fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 -- 172.21.15.114:0/1269658433 shutdown_connections 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.626+0000 7f8f2b000640 1 -- 172.21.15.114:0/1269658433 wait complete. 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.627+0000 7f8f2b000640 1 Processor -- start 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.627+0000 7f8f2b000640 1 -- start start 2024-09-19T06:51:42.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.627+0000 7f8f2b000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 0x7f8f2419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.627+0000 7f8f2b000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f241034b0 0x7f8f2419a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.627+0000 7f8f2b000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f2410bfc0 0x7f8f2419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.627+0000 7f8f2b000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24111e10 con 0x7f8f24102b90 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.627+0000 7f8f2b000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24111c90 con 0x7f8f241034b0 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.627+0000 7f8f2b000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f8f24111f90 con 0x7f8f2410bfc0 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f241034b0 0x7f8f2419a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 0x7f8f2419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f241034b0 0x7f8f2419a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:43366/0 (socket says 172.21.15.114:43366) 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 0x7f8f2419a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43862/0 (socket says 172.21.15.114:43862) 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29200640 1 -- 172.21.15.114:0/3412352999 learned_addr learned my addr 172.21.15.114:0/3412352999 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:42.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f2a600640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f2410bfc0 0x7f8f2419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29c00640 1 -- 172.21.15.114:0/3412352999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f2410bfc0 msgr2=0x7f8f2419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29c00640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f2410bfc0 0x7f8f2419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29c00640 1 -- 172.21.15.114:0/3412352999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f241034b0 msgr2=0x7f8f2419a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29c00640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f241034b0 0x7f8f2419a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29c00640 1 -- 172.21.15.114:0/3412352999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2419f460 con 0x7f8f24102b90 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29200640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f241034b0 0x7f8f2419a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.628+0000 7f8f29c00640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 0x7f8f2419a4b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8f18005e30 tx=0x7f8f18003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.629+0000 7f8f22000640 1 -- 172.21.15.114:0/3412352999 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18034040 con 0x7f8f24102b90 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.629+0000 7f8f22000640 1 -- 172.21.15.114:0/3412352999 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f1802fe20 con 0x7f8f24102b90 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.629+0000 7f8f2b000640 1 -- 172.21.15.114:0/3412352999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f2419f6f0 con 0x7f8f24102b90 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.629+0000 7f8f22000640 1 -- 172.21.15.114:0/3412352999 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18004830 con 0x7f8f24102b90 2024-09-19T06:51:42.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.629+0000 7f8f2b000640 1 -- 172.21.15.114:0/3412352999 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f241a7030 con 0x7f8f24102b90 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f2efc1740 1 Processor -- start 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f2efc1740 1 -- start start 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f22000640 1 -- 172.21.15.114:0/3412352999 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f8f18037020 con 0x7f8f24102b90 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3eb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3eb6c0 0x558dca3edab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558dca3460b0 con 0x558dca3eb6c0 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558dca2edab0 con 0x558dca3e0e00 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.633+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558dca342070 con 0x558dca3e04e0 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f29200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3eb6c0 0x558dca3edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:48452/0 (socket says 172.21.15.114:48452) 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 -- 172.21.15.114:0/2959920763 learned_addr learned my addr 172.21.15.114:0/2959920763 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:42.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f29c00640 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 -- 172.21.15.114:0/2959920763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 msgr2=0x558dca3eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:42.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3eb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 -- 172.21.15.114:0/2959920763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3eb6c0 msgr2=0x558dca3edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:42.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3eb6c0 0x558dca3edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 -- 172.21.15.114:0/2959920763 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dca2fa4a0 con 0x558dca3e04e0 2024-09-19T06:51:42.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f29c00640 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:51:42.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.634+0000 7f8f2a600640 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e08c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c004620 tx=0x7f8f1c00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=726c22f8692902f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:42.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.635+0000 7f8f16a00640 1 -- 172.21.15.114:0/2959920763 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c003a70 con 0x558dca3e04e0 2024-09-19T06:51:42.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.635+0000 7f8f16a00640 1 -- 172.21.15.114:0/2959920763 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f1c015070 con 0x558dca3e04e0 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.635+0000 7f8f22000640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8f0c0a32c0 0x7f8f0c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.635+0000 7f8f16a00640 1 -- 172.21.15.114:0/2959920763 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c011460 con 0x558dca3e04e0 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.635+0000 7f8f22000640 1 -- 172.21.15.114:0/3412352999 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f8f180f0b00 con 0x7f8f24102b90 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.635+0000 7f8f29c00640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8f0c0a32c0 0x7f8f0c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 -- 172.21.15.114:0/2959920763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 msgr2=0x558dca3e08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e08c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c004620 tx=0x7f8f1c00ba50 comp rx=0 tx=0).stop 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f29c00640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8f0c0a32c0 0x7f8f0c0a5780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8f180bcc60 tx=0x7f8f180095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 -- 172.21.15.114:0/2959920763 shutdown_connections 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3eb6c0 0x558dca3edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/2959920763 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e08c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 -- 172.21.15.114:0/2959920763 >> 172.21.15.114:0/2959920763 conn(0x558dca3dca70 msgr2=0x558dca3dce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:42.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 -- 172.21.15.114:0/2959920763 shutdown_connections 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 -- 172.21.15.114:0/2959920763 wait complete. 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.636+0000 7f8f2efc1740 1 -- start start 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.637+0000 7f8f2efc1740 1 Processor -- start 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.637+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e04e0 0x558dca3e85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.637+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.637+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 0x558dca3e9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.637+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558dca3ee330 con 0x558dca3e04e0 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.637+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558dca2ce920 con 0x558dca3e0e00 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.637+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558dca3ee4b0 con 0x558dca3eb6c0 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.637+0000 7f8f2efc1740 10 client.0 _ll_register_callbacks cb 0x558dca2f28a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f2a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e04e0 0x558dca3e85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3e8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:43396/0 (socket says 172.21.15.114:43396) 2024-09-19T06:51:42.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 -- 172.21.15.114:0/1618217005 learned_addr learned my addr 172.21.15.114:0/1618217005 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f2a600640 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 0x558dca3e9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 -- 172.21.15.114:0/1618217005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 msgr2=0x558dca3e9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 0x558dca3e9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 -- 172.21.15.114:0/1618217005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e04e0 msgr2=0x558dca3e85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e04e0 0x558dca3e85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 -- 172.21.15.114:0/1618217005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dca0b72b0 con 0x558dca3e0e00 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f29200640 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3e8ae0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8f10011850 tx=0x7f8f10011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd17c1e151173ed6 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.638+0000 7f8f0b400640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f0b400640 1 -- 172.21.15.114:0/1618217005 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10004260 con 0x558dca3e0e00 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f0b400640 1 -- 172.21.15.114:0/1618217005 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f8f10004400 con 0x558dca3e0e00 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f2efc1740 1 -- 172.21.15.114:0/1618217005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f1c015070 con 0x558dca3e0e00 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f0b400640 1 -- 172.21.15.114:0/1618217005 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8f10010990 con 0x558dca3e0e00 2024-09-19T06:51:42.658 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f2efc1740 10 client.25115 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f2efc1740 1 -- 172.21.15.114:0/1618217005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558dca3fa0f0 con 0x558dca3e0e00 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f0b400640 1 -- 172.21.15.114:0/1618217005 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f1c015070 con 0x558dca3e0e00 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f0b400640 1 -- 172.21.15.114:0/1618217005 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8f1001b420 con 0x558dca3e0e00 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.639+0000 7f8f2efc1740 10 client.25115 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.640+0000 7f8f2efc1740 10 client.25115 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.640+0000 7f8f2efc1740 20 client.25115 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.640+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x558dca3fc750 0x558dca3fdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.640+0000 7f8f2efc1740 4 client.25115 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.640+0000 7f8f2efc1740 1 -- 172.21.15.114:0/1618217005 --> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558dca353620 con 0x558dca3fc750 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.640+0000 7f8f2a600640 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x558dca3fc750 0x558dca3fdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.641+0000 7f8f2a600640 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x558dca3fc750 0x558dca3fdbc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=beb9edeac680d7c2 server_cookie=87eb3fccaba86da in_seq=0 out_seq=0 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.641+0000 7f8f0b400640 10 client.25115 ms_handle_connect on v2:172.21.15.116:6836/2673389539 2024-09-19T06:51:42.659 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.641+0000 7f8f0b400640 1 -- 172.21.15.114:0/1618217005 <== mds.4 v2:172.21.15.116:6836/2673389539 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558dca353620 con 0x558dca3fc750 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.641+0000 7f8f0b400640 10 client.25115 handle_command_reply: tid=0 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.641+0000 7f8f2efc1740 1 client.25115 shutdown 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 -- 172.21.15.114:0/1618217005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 msgr2=0x558dca3e8ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3e8ae0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8f10011850 tx=0x7f8f10011d10 comp rx=0 tx=0).stop 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 -- 172.21.15.114:0/1618217005 shutdown_connections 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x558dca3fc750 0x558dca3fdbc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 0x558dca3e9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e0e00 0x558dca3e8ae0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/1618217005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e04e0 0x558dca3e85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 -- 172.21.15.114:0/1618217005 >> 172.21.15.114:0/1618217005 conn(0x558dca3dca70 msgr2=0x558dca3e2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f0b400640 0 client.25115 ms_handle_reset on v2:172.21.15.116:6836/2673389539 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 -- 172.21.15.114:0/1618217005 shutdown_connections 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 1 -- 172.21.15.114:0/1618217005 wait complete. 2024-09-19T06:51:42.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.642+0000 7f8f2efc1740 20 client.25115 trim_cache size 0 max 16384 2024-09-19T06:51:42.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.660+0000 7f8f2efc1740 1 Processor -- start 2024-09-19T06:51:42.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.660+0000 7f8f2efc1740 1 -- start start 2024-09-19T06:51:42.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3e8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3eb6c0 0x558dca3ea510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558dca3fbf00 con 0x558dca3e0e00 2024-09-19T06:51:42.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558dca3fc5d0 con 0x558dca3eb6c0 2024-09-19T06:51:42.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558dca3fcca0 con 0x558dca3e04e0 2024-09-19T06:51:42.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f29200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3e8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f29200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3e8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43902/0 (socket says 172.21.15.114:43902) 2024-09-19T06:51:42.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f29200640 1 -- 172.21.15.114:0/528948711 learned_addr learned my addr 172.21.15.114:0/528948711 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:42.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2a600640 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3eb6c0 0x558dca3ea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2a600640 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f29200640 1 -- 172.21.15.114:0/528948711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 msgr2=0x558dca3e2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f29200640 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f29200640 1 -- 172.21.15.114:0/528948711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3eb6c0 msgr2=0x558dca3ea510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f29200640 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3eb6c0 0x558dca3ea510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f29200640 1 -- 172.21.15.114:0/528948711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dca3e0310 con 0x558dca3e0e00 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.661+0000 7f8f2a600640 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.662+0000 7f8f29200640 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3e8e70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f1001ba30 tx=0x7f8f1000bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0d44b804041e97 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.662+0000 7f8f16000640 1 -- 172.21.15.114:0/528948711 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1002e030 con 0x558dca3e0e00 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.662+0000 7f8f16000640 1 -- 172.21.15.114:0/528948711 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f1001b420 con 0x558dca3e0e00 2024-09-19T06:51:42.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.662+0000 7f8f16000640 1 -- 172.21.15.114:0/528948711 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f100035f0 con 0x558dca3e0e00 2024-09-19T06:51:42.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 -- 172.21.15.114:0/528948711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 msgr2=0x558dca3e8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3e8e70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f1001ba30 tx=0x7f8f1000bf20 comp rx=0 tx=0).stop 2024-09-19T06:51:42.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 -- 172.21.15.114:0/528948711 shutdown_connections 2024-09-19T06:51:42.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3eb6c0 0x558dca3ea510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3e8e70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/528948711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3e04e0 0x558dca3e2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 -- 172.21.15.114:0/528948711 >> 172.21.15.114:0/528948711 conn(0x558dca3dca70 msgr2=0x558dca3e9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:42.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 -- 172.21.15.114:0/528948711 shutdown_connections 2024-09-19T06:51:42.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 -- 172.21.15.114:0/528948711 wait complete. 2024-09-19T06:51:42.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.663+0000 7f8f2efc1740 1 -- start start 2024-09-19T06:51:42.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2efc1740 1 Processor -- start 2024-09-19T06:51:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e04e0 0x558dca406660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3ff990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2efc1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 0x558dca3fff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558dca3fda40 con 0x558dca3e0e00 2024-09-19T06:51:42.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558dca3fd370 con 0x558dca3e04e0 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2efc1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x558dca3fe110 con 0x558dca3eb6c0 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2efc1740 10 client.0 _ll_register_callbacks cb 0x7f8f10011460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f29200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3ff990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2a600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e04e0 0x558dca406660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f29200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3ff990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43918/0 (socket says 172.21.15.114:43918) 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f29200640 1 -- 172.21.15.114:0/3529924732 learned_addr learned my addr 172.21.15.114:0/3529924732 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.664+0000 7f8f2a600640 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 0x558dca3fff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f29200640 1 -- 172.21.15.114:0/3529924732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 msgr2=0x558dca3fff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f29200640 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 0x558dca3fff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f29200640 1 -- 172.21.15.114:0/3529924732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e04e0 msgr2=0x558dca406660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f29200640 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e04e0 0x558dca406660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f29200640 1 -- 172.21.15.114:0/3529924732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dca400740 con 0x558dca3e0e00 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f2a600640 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e04e0 0x558dca406660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:42.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f29200640 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3ff990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8f10011d50 tx=0x7f8f10010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3410032141ed43 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f0b400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f0b400640 1 -- 172.21.15.114:0/3529924732 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10038020 con 0x558dca3e0e00 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f0b400640 1 -- 172.21.15.114:0/3529924732 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f8f100054e0 con 0x558dca3e0e00 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.665+0000 7f8f2efc1740 1 -- 172.21.15.114:0/3529924732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f1001b420 con 0x558dca3e0e00 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f0b400640 1 -- 172.21.15.114:0/3529924732 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8f1002e030 con 0x558dca3e0e00 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f2efc1740 10 client.15126 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f2efc1740 1 -- 172.21.15.114:0/3529924732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558dca466f20 con 0x558dca3e0e00 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f0b400640 1 -- 172.21.15.114:0/3529924732 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f1001b420 con 0x558dca3e0e00 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f0b400640 1 -- 172.21.15.114:0/3529924732 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8f10034610 con 0x558dca3e0e00 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f2efc1740 10 client.15126 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f2efc1740 10 client.15126 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f2efc1740 20 client.15126 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x558dca46b8d0 0x558dca46a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:42.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f2efc1740 4 client.15126 mds_command: new command op to 24482 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.666+0000 7f8f2efc1740 1 -- 172.21.15.114:0/3529924732 --> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] -- command(tid 0: {"prefix": "status"}) -- 0x558dca34db30 con 0x558dca46b8d0 2024-09-19T06:51:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.667+0000 7f8f2a600640 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x558dca46b8d0 0x558dca46a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.667+0000 7f8f2a600640 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x558dca46b8d0 0x558dca46a5e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a855010a281d24bf server_cookie=cef4ddc0a044ffeb in_seq=0 out_seq=0 2024-09-19T06:51:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.667+0000 7f8f0b400640 10 client.15126 ms_handle_connect on v2:172.21.15.116:6836/2673389539 2024-09-19T06:51:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f0b400640 1 -- 172.21.15.114:0/3529924732 <== mds.4 v2:172.21.15.116:6836/2673389539 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558dca34db30 con 0x558dca46b8d0 2024-09-19T06:51:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f0b400640 10 client.15126 handle_command_reply: tid=0 2024-09-19T06:51:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 client.15126 shutdown 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 -- 172.21.15.114:0/3529924732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 msgr2=0x558dca3ff990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3ff990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8f10011d50 tx=0x7f8f10010800 comp rx=0 tx=0).stop 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 -- 172.21.15.114:0/3529924732 shutdown_connections 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.116:6836/2673389539,v1:172.21.15.116:6837/2673389539] conn(0x558dca46b8d0 0x558dca46a5e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558dca3eb6c0 0x558dca3fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558dca3e0e00 0x558dca3ff990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 --2- 172.21.15.114:0/3529924732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558dca3e04e0 0x558dca406660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 -- 172.21.15.114:0/3529924732 >> 172.21.15.114:0/3529924732 conn(0x558dca3dca70 msgr2=0x558dca3f6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f0b400640 0 client.15126 ms_handle_reset on v2:172.21.15.116:6836/2673389539 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 -- 172.21.15.114:0/3529924732 shutdown_connections 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 1 -- 172.21.15.114:0/3529924732 wait complete. 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.668+0000 7f8f2efc1740 20 client.15126 trim_cache size 0 max 16384 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 4, 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24482, 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:42.674 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:42.675 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 104.43406441899999, 2024-09-19T06:51:42.675 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:42.675 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:42.675 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:42.675 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 177.38866808200001 2024-09-19T06:51:42.675 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:42.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 -- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8f0c0a32c0 msgr2=0x7f8f0c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8f0c0a32c0 0x7f8f0c0a5780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8f180bcc60 tx=0x7f8f180095c0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 -- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 msgr2=0x7f8f2419a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 0x7f8f2419a4b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8f18005e30 tx=0x7f8f18003d60 comp rx=0 tx=0).stop 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 -- 172.21.15.114:0/3412352999 shutdown_connections 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f8f0c0a32c0 0x7f8f0c0a5780 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f2410bfc0 0x7f8f2419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f241034b0 0x7f8f2419a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 --2- 172.21.15.114:0/3412352999 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f24102b90 0x7f8f2419a4b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 -- 172.21.15.114:0/3412352999 >> 172.21.15.114:0/3412352999 conn(0x7f8f240fc610 msgr2=0x7f8f2410d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 -- 172.21.15.114:0/3412352999 shutdown_connections 2024-09-19T06:51:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:42.669+0000 7f8f2b000640 1 -- 172.21.15.114:0/3412352999 wait complete. 2024-09-19T06:51:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:42 smithi116 ceph-mon[35411]: pgmap v380: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:43.410 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-19T06:51:43.787 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:44.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 -- 172.21.15.114:0/3765183181 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd528102360 msgr2=0x7fd52810e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 --2- 172.21.15.114:0/3765183181 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd528102360 0x7fd52810e890 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd520009cd0 tx=0x7fd52002a8e0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 -- 172.21.15.114:0/3765183181 shutdown_connections 2024-09-19T06:51:44.382 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 --2- 172.21.15.114:0/3765183181 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd528102360 0x7fd52810e890 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 --2- 172.21.15.114:0/3765183181 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5281019c0 0x7fd528101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 --2- 172.21.15.114:0/3765183181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5281079c0 0x7fd528107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 -- 172.21.15.114:0/3765183181 >> 172.21.15.114:0/3765183181 conn(0x7fd5280fd740 msgr2=0x7fd5280ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:44.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 -- 172.21.15.114:0/3765183181 shutdown_connections 2024-09-19T06:51:44.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.380+0000 7fd527e00640 1 -- 172.21.15.114:0/3765183181 wait complete. 2024-09-19T06:51:44.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.381+0000 7fd527e00640 1 Processor -- start 2024-09-19T06:51:44.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.381+0000 7fd527e00640 1 -- start start 2024-09-19T06:51:44.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5281019c0 0x7fd52819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd528102360 0x7fd52819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5281079c0 0x7fd52819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd528111dc0 con 0x7fd5281079c0 2024-09-19T06:51:44.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527e00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd528111c40 con 0x7fd528102360 2024-09-19T06:51:44.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7fd528111f40 con 0x7fd5281019c0 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5281079c0 0x7fd52819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd526a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5281019c0 0x7fd52819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5281079c0 0x7fd52819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43928/0 (socket says 172.21.15.114:43928) 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd527400640 1 -- 172.21.15.114:0/1198258364 learned_addr learned my addr 172.21.15.114:0/1198258364 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.382+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd528102360 0x7fd52819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd51fe00640 1 -- 172.21.15.114:0/1198258364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5281019c0 msgr2=0x7fd52819a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5281019c0 0x7fd52819a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd51fe00640 1 -- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5281079c0 msgr2=0x7fd52819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5281079c0 0x7fd52819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd51fe00640 1 -- 172.21.15.114:0/1198258364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52819f440 con 0x7fd528102360 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd527400640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5281079c0 0x7fd52819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:44.385 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd528102360 0x7fd52819a9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd51000e9f0 tx=0x7fd51000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:44.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd52ca00640 1 -- 172.21.15.114:0/1198258364 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51000ccc0 con 0x7fd528102360 2024-09-19T06:51:44.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd527e00640 1 -- 172.21.15.114:0/1198258364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52819f730 con 0x7fd528102360 2024-09-19T06:51:44.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.383+0000 7fd52ca00640 1 -- 172.21.15.114:0/1198258364 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd510005020 con 0x7fd528102360 2024-09-19T06:51:44.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.384+0000 7fd527e00640 1 -- 172.21.15.114:0/1198258364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5281a7010 con 0x7fd528102360 2024-09-19T06:51:44.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.384+0000 7fd52ca00640 1 -- 172.21.15.114:0/1198258364 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51001c070 con 0x7fd528102360 2024-09-19T06:51:44.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.386+0000 7fd52ca00640 1 -- 172.21.15.114:0/1198258364 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7fd510004a00 con 0x7fd528102360 2024-09-19T06:51:44.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.387+0000 7fd5307d2740 1 Processor -- start 2024-09-19T06:51:44.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.387+0000 7fd5307d2740 1 -- start start 2024-09-19T06:51:44.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.387+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb80424e0 0x556eb80428c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.387+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb8042e00 0x556eb804d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.387+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb804fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.387+0000 7fd5307d2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556eb7fa80b0 con 0x556eb804d6c0 2024-09-19T06:51:44.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.387+0000 7fd5307d2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556eb7f4fab0 con 0x556eb8042e00 2024-09-19T06:51:44.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.387+0000 7fd5307d2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556eb7fa4070 con 0x556eb80424e0 2024-09-19T06:51:44.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd51fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb804fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd51fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb804fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43942/0 (socket says 172.21.15.114:43942) 2024-09-19T06:51:44.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd51fe00640 1 -- 172.21.15.114:0/251149928 learned_addr learned my addr 172.21.15.114:0/251149928 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:44.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd526a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb8042e00 0x556eb804d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd527400640 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb80424e0 0x556eb80428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd51fe00640 1 -- 172.21.15.114:0/251149928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb80424e0 msgr2=0x556eb80428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:44.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd51fe00640 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb80424e0 0x556eb80428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd51fe00640 1 -- 172.21.15.114:0/251149928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb8042e00 msgr2=0x556eb804d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd51fe00640 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb8042e00 0x556eb804d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd51fe00640 1 -- 172.21.15.114:0/251149928 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eb7f5c4a0 con 0x556eb804d6c0 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd526a00640 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb8042e00 0x556eb804d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.388+0000 7fd52ca00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd50c0a3240 0x7fd50c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.389+0000 7fd51fe00640 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb804fab0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd5100d3da0 tx=0x7fd5100d86c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8df3df84d7a1e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.389+0000 7fd52ca00640 1 -- 172.21.15.114:0/1198258364 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fd5100cfc70 con 0x7fd528102360 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.389+0000 7fd527400640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd50c0a3240 0x7fd50c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.389+0000 7fd517e00640 1 -- 172.21.15.114:0/251149928 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51009e070 con 0x556eb804d6c0 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.389+0000 7fd517e00640 1 -- 172.21.15.114:0/251149928 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51009c540 con 0x556eb804d6c0 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.389+0000 7fd517e00640 1 -- 172.21.15.114:0/251149928 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51009b7a0 con 0x556eb804d6c0 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 -- 172.21.15.114:0/251149928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 msgr2=0x556eb804fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb804fab0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd5100d3da0 tx=0x7fd5100d86c0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd527400640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd50c0a3240 0x7fd50c0a5700 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd520031860 tx=0x7fd520005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 -- 172.21.15.114:0/251149928 shutdown_connections 2024-09-19T06:51:44.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb804fab0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb8042e00 0x556eb804d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 --2- 172.21.15.114:0/251149928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb80424e0 0x556eb80428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 -- 172.21.15.114:0/251149928 >> 172.21.15.114:0/251149928 conn(0x556eb803ea70 msgr2=0x556eb803ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 -- 172.21.15.114:0/251149928 shutdown_connections 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 -- 172.21.15.114:0/251149928 wait complete. 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.390+0000 7fd5307d2740 1 -- start start 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd5307d2740 1 Processor -- start 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb804a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb8042e00 0x556eb804ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb804d6c0 0x556eb804b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd5307d2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556eb8050330 con 0x556eb8042e00 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd5307d2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556eb7f30920 con 0x556eb80424e0 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd5307d2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556eb80504b0 con 0x556eb804d6c0 2024-09-19T06:51:44.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd5307d2740 10 client.0 _ll_register_callbacks cb 0x556eb7f548a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd51fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb8042e00 0x556eb804ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd51fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb8042e00 0x556eb804ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43954/0 (socket says 172.21.15.114:43954) 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd51fe00640 1 -- 172.21.15.114:0/3242902338 learned_addr learned my addr 172.21.15.114:0/3242902338 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd527400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb804d6c0 0x556eb804b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.391+0000 7fd526a00640 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb804a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd51fe00640 1 -- 172.21.15.114:0/3242902338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb804d6c0 msgr2=0x556eb804b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd51fe00640 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb804d6c0 0x556eb804b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd51fe00640 1 -- 172.21.15.114:0/3242902338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 msgr2=0x556eb804a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:44.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd51fe00640 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb804a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd51fe00640 1 -- 172.21.15.114:0/3242902338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eb7d192b0 con 0x556eb8042e00 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd527400640 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb804d6c0 0x556eb804b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd51fe00640 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb8042e00 0x556eb804ab00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd51000e9c0 tx=0x7fd5100053d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c14537e44c6374 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd514c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd514c00640 1 -- 172.21.15.114:0/3242902338 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51009e040 con 0x556eb8042e00 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd514c00640 1 -- 172.21.15.114:0/3242902338 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fd51009a140 con 0x556eb8042e00 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.392+0000 7fd5307d2740 1 -- 172.21.15.114:0/3242902338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd51009c540 con 0x556eb8042e00 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd514c00640 1 -- 172.21.15.114:0/3242902338 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd510082050 con 0x556eb8042e00 2024-09-19T06:51:44.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd5307d2740 10 client.15138 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd5307d2740 1 -- 172.21.15.114:0/3242902338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556eb805bff0 con 0x556eb8042e00 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd514c00640 1 -- 172.21.15.114:0/3242902338 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51009c540 con 0x556eb8042e00 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd514c00640 1 -- 172.21.15.114:0/3242902338 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd5100d9040 con 0x556eb8042e00 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd5307d2740 10 client.15138 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd5307d2740 10 client.15138 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd5307d2740 20 client.15138 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd5307d2740 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x556eb805e150 0x556eb805f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd5307d2740 4 client.15138 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:44.399 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.393+0000 7fd5307d2740 1 -- 172.21.15.114:0/3242902338 --> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556eb7fb5620 con 0x556eb805e150 2024-09-19T06:51:44.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.394+0000 7fd526a00640 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x556eb805e150 0x556eb805f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.394+0000 7fd526a00640 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x556eb805e150 0x556eb805f5c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=59869b07423f0919 server_cookie=78f25cd27b9050a7 in_seq=0 out_seq=0 2024-09-19T06:51:44.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.394+0000 7fd514c00640 10 client.15138 ms_handle_connect on v2:172.21.15.118:6834/449495659 2024-09-19T06:51:44.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd514c00640 1 -- 172.21.15.114:0/3242902338 <== mds.3 v2:172.21.15.118:6834/449495659 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556eb7fb5620 con 0x556eb805e150 2024-09-19T06:51:44.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd514c00640 10 client.15138 handle_command_reply: tid=0 2024-09-19T06:51:44.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 client.15138 shutdown 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 -- 172.21.15.114:0/3242902338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb8042e00 msgr2=0x556eb804ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb8042e00 0x556eb804ab00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd51000e9c0 tx=0x7fd5100053d0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 -- 172.21.15.114:0/3242902338 shutdown_connections 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb804d6c0 0x556eb804b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb8042e00 0x556eb804ab00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x556eb805e150 0x556eb805f5c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 --2- 172.21.15.114:0/3242902338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb804a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 -- 172.21.15.114:0/3242902338 >> 172.21.15.114:0/3242902338 conn(0x556eb803ea70 msgr2=0x556eb8044750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd514c00640 0 client.15138 ms_handle_reset on v2:172.21.15.118:6834/449495659 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.395+0000 7fd5307d2740 1 -- 172.21.15.114:0/3242902338 shutdown_connections 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.396+0000 7fd5307d2740 1 -- 172.21.15.114:0/3242902338 wait complete. 2024-09-19T06:51:44.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.396+0000 7fd5307d2740 20 client.15138 trim_cache size 0 max 16384 2024-09-19T06:51:44.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.414+0000 7fd5307d2740 1 Processor -- start 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.414+0000 7fd5307d2740 1 -- start start 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.414+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb80424e0 0x556eb805f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb804ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb804d6c0 0x556eb8047860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd5307d2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556eb805d900 con 0x556eb80424e0 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd5307d2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556eb805dfd0 con 0x556eb804d6c0 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd5307d2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556eb805e6a0 con 0x556eb8042e00 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb804ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd526a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb80424e0 0x556eb805f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb804ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:48542/0 (socket says 172.21.15.114:48542) 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd526a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb80424e0 0x556eb805f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43958/0 (socket says 172.21.15.114:43958) 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 -- 172.21.15.114:0/1422233954 learned_addr learned my addr 172.21.15.114:0/1422233954 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:44.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd527400640 1 --2- 172.21.15.114:0/1422233954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb804d6c0 0x556eb8047860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 -- 172.21.15.114:0/1422233954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb804d6c0 msgr2=0x556eb8047860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1422233954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb804d6c0 0x556eb8047860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 -- 172.21.15.114:0/1422233954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb80424e0 msgr2=0x556eb805f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1422233954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb80424e0 0x556eb805f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 -- 172.21.15.114:0/1422233954 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eb804fe10 con 0x556eb8042e00 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.415+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1422233954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb804ae90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd51009a0f0 tx=0x7fd510005440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b81671ca32ba4e0e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd517400640 1 -- 172.21.15.114:0/1422233954 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51009e070 con 0x556eb8042e00 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd517400640 1 -- 172.21.15.114:0/1422233954 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51009c540 con 0x556eb8042e00 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd517400640 1 -- 172.21.15.114:0/1422233954 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5100d31d0 con 0x556eb8042e00 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd5307d2740 1 -- 172.21.15.114:0/1422233954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 msgr2=0x556eb804ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1422233954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb804ae90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd51009a0f0 tx=0x7fd510005440 comp rx=0 tx=0).stop 2024-09-19T06:51:44.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd5307d2740 1 -- 172.21.15.114:0/1422233954 shutdown_connections 2024-09-19T06:51:44.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1422233954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb804d6c0 0x556eb8047860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1422233954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb804ae90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.416+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1422233954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb80424e0 0x556eb805f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.417+0000 7fd5307d2740 1 -- 172.21.15.114:0/1422233954 >> 172.21.15.114:0/1422233954 conn(0x556eb803ea70 msgr2=0x556eb80496d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:44.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.417+0000 7fd5307d2740 1 -- 172.21.15.114:0/1422233954 shutdown_connections 2024-09-19T06:51:44.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.417+0000 7fd5307d2740 1 -- 172.21.15.114:0/1422233954 wait complete. 2024-09-19T06:51:44.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.417+0000 7fd5307d2740 1 -- start start 2024-09-19T06:51:44.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.417+0000 7fd5307d2740 1 Processor -- start 2024-09-19T06:51:44.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb80680b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb80613e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd5307d2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb8061a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd5307d2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556eb805f440 con 0x556eb804d6c0 2024-09-19T06:51:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd5307d2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556eb805ed70 con 0x556eb80424e0 2024-09-19T06:51:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd5307d2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x556eb805fb10 con 0x556eb8042e00 2024-09-19T06:51:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd5307d2740 10 client.0 _ll_register_callbacks cb 0x7fd5100d3bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd526a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb80680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd526a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb80680b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:43488/0 (socket says 172.21.15.114:43488) 2024-09-19T06:51:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd526a00640 1 -- 172.21.15.114:0/1644814744 learned_addr learned my addr 172.21.15.114:0/1644814744 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:44.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd527400640 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb8061a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb80613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd526a00640 1 -- 172.21.15.114:0/1644814744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 msgr2=0x556eb80613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd526a00640 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb80613e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd526a00640 1 -- 172.21.15.114:0/1644814744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 msgr2=0x556eb8061a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd526a00640 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb8061a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.418+0000 7fd526a00640 1 -- 172.21.15.114:0/1644814744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eb8062210 con 0x556eb80424e0 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.419+0000 7fd526a00640 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb80680b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd518005e00 tx=0x7fd518012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc8658b83e3fe3ff server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.419+0000 7fd514c00640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.419+0000 7fd514c00640 1 -- 172.21.15.114:0/1644814744 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518004220 con 0x556eb80424e0 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.419+0000 7fd5307d2740 1 -- 172.21.15.114:0/1644814744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd51009c540 con 0x556eb80424e0 2024-09-19T06:51:44.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.419+0000 7fd514c00640 1 -- 172.21.15.114:0/1644814744 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7fd5180043c0 con 0x556eb80424e0 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.419+0000 7fd514c00640 1 -- 172.21.15.114:0/1644814744 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd518005020 con 0x556eb80424e0 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.419+0000 7fd5307d2740 10 client.25139 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.419+0000 7fd5307d2740 1 -- 172.21.15.114:0/1644814744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556eb80c89b0 con 0x556eb80424e0 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd514c00640 1 -- 172.21.15.114:0/1644814744 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51009c540 con 0x556eb80424e0 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd514c00640 1 -- 172.21.15.114:0/1644814744 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd51801e6d0 con 0x556eb80424e0 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd5307d2740 10 client.25139 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd5307d2740 10 client.25139 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd5307d2740 20 client.25139 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x556eb80cd590 0x556eb80cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd5307d2740 4 client.25139 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:44.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd5307d2740 1 -- 172.21.15.114:0/1644814744 --> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] -- command(tid 0: {"prefix": "status"}) -- 0x556eb7fafb30 con 0x556eb80cd590 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.420+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x556eb80cd590 0x556eb80cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd51fe00640 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x556eb80cd590 0x556eb80cc2a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=179886ebdf221e13 server_cookie=44a422a6552e42fd in_seq=0 out_seq=0 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd514c00640 10 client.25139 ms_handle_connect on v2:172.21.15.118:6834/449495659 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd514c00640 1 -- 172.21.15.114:0/1644814744 <== mds.3 v2:172.21.15.118:6834/449495659 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x556eb7fafb30 con 0x556eb80cd590 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd514c00640 10 client.25139 handle_command_reply: tid=0 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 client.25139 shutdown 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 -- 172.21.15.114:0/1644814744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 msgr2=0x556eb80680b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb80680b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd518005e00 tx=0x7fd518012820 comp rx=0 tx=0).stop 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 -- 172.21.15.114:0/1644814744 shutdown_connections 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556eb804d6c0 0x556eb8061a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556eb8042e00 0x556eb80613e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.118:6834/449495659,v1:172.21.15.118:6835/449495659] conn(0x556eb80cd590 0x556eb80cc2a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 --2- 172.21.15.114:0/1644814744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eb80424e0 0x556eb80680b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd5307d2740 1 -- 172.21.15.114:0/1644814744 >> 172.21.15.114:0/1644814744 conn(0x556eb803ea70 msgr2=0x556eb804aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.421+0000 7fd514c00640 0 client.25139 ms_handle_reset on v2:172.21.15.118:6834/449495659 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.422+0000 7fd5307d2740 1 -- 172.21.15.114:0/1644814744 shutdown_connections 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.422+0000 7fd5307d2740 1 -- 172.21.15.114:0/1644814744 wait complete. 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.422+0000 7fd5307d2740 20 client.25139 trim_cache size 0 max 16384 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 3, 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24508, 2024-09-19T06:51:44.430 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 107.196952733, 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 184.456817487 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:44.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.422+0000 7fd527e00640 1 -- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd50c0a3240 msgr2=0x7fd50c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.422+0000 7fd527e00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd50c0a3240 0x7fd50c0a5700 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd520031860 tx=0x7fd520005c50 comp rx=0 tx=0).stop 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.422+0000 7fd527e00640 1 -- 172.21.15.114:0/1198258364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd528102360 msgr2=0x7fd52819a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.422+0000 7fd527e00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd528102360 0x7fd52819a9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd51000e9f0 tx=0x7fd51000eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.423+0000 7fd527e00640 1 -- 172.21.15.114:0/1198258364 shutdown_connections 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.423+0000 7fd527e00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7fd50c0a3240 0x7fd50c0a5700 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.423+0000 7fd527e00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5281079c0 0x7fd52819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.423+0000 7fd527e00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd528102360 0x7fd52819a9d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.423+0000 7fd527e00640 1 --2- 172.21.15.114:0/1198258364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd5281019c0 0x7fd52819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.423+0000 7fd527e00640 1 -- 172.21.15.114:0/1198258364 >> 172.21.15.114:0/1198258364 conn(0x7fd5280fd740 msgr2=0x7fd5280fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.423+0000 7fd527e00640 1 -- 172.21.15.114:0/1198258364 shutdown_connections 2024-09-19T06:51:44.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:44.423+0000 7fd527e00640 1 -- 172.21.15.114:0/1198258364 wait complete. 2024-09-19T06:51:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:44 smithi114 ceph-mon[30820]: pgmap v381: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:44 smithi116 ceph-mon[35411]: pgmap v381: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:44.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:44 smithi118 ceph-mon[35527]: pgmap v381: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:45.243 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-19T06:51:45.611 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 -- 172.21.15.114:0/1119857074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4154103e30 msgr2=0x7f4154104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 --2- 172.21.15.114:0/1119857074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4154103e30 0x7f4154104210 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4144009900 tx=0x7f414402a410 comp rx=0 tx=0).stop 2024-09-19T06:51:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 -- 172.21.15.114:0/1119857074 shutdown_connections 2024-09-19T06:51:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 --2- 172.21.15.114:0/1119857074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4154108f60 0x7f4154110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 --2- 172.21.15.114:0/1119857074 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41541047e0 0x7f4154108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 --2- 172.21.15.114:0/1119857074 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4154103e30 0x7f4154104210 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 -- 172.21.15.114:0/1119857074 >> 172.21.15.114:0/1119857074 conn(0x7f41540ff910 msgr2=0x7f4154101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:46.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 -- 172.21.15.114:0/1119857074 shutdown_connections 2024-09-19T06:51:46.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.168+0000 7f4159a00640 1 -- 172.21.15.114:0/1119857074 wait complete. 2024-09-19T06:51:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.169+0000 7f4159a00640 1 Processor -- start 2024-09-19T06:51:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.169+0000 7f4159a00640 1 -- start start 2024-09-19T06:51:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f4159a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4154103e30 0x7f415419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f4159a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41541047e0 0x7f415419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f4159a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4154108f60 0x7f415419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f4159a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4154113fc0 con 0x7f41541047e0 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f4159a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4154113e40 con 0x7f4154103e30 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f4159a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f4154114140 con 0x7f4154108f60 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f414be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41541047e0 0x7f415419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f4153e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4154103e30 0x7f415419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f414be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41541047e0 0x7f415419a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43976/0 (socket says 172.21.15.114:43976) 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f414be00640 1 -- 172.21.15.114:0/835989577 learned_addr learned my addr 172.21.15.114:0/835989577 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.170+0000 7f4159000640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4154108f60 0x7f415419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4153e00640 1 -- 172.21.15.114:0/835989577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4154108f60 msgr2=0x7f415419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4153e00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4154108f60 0x7f415419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4153e00640 1 -- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41541047e0 msgr2=0x7f415419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4153e00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41541047e0 0x7f415419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4153e00640 1 -- 172.21.15.114:0/835989577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415419f430 con 0x7f4154103e30 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f414be00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41541047e0 0x7f415419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4153e00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4154103e30 0x7f415419a480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4144005940 tx=0x7f41440040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4151600640 1 -- 172.21.15.114:0/835989577 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414403f030 con 0x7f4154103e30 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4159a00640 1 -- 172.21.15.114:0/835989577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415419f6c0 con 0x7f4154103e30 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4151600640 1 -- 172.21.15.114:0/835989577 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41440044c0 con 0x7f4154103e30 2024-09-19T06:51:46.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.171+0000 7f4151600640 1 -- 172.21.15.114:0/835989577 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144035070 con 0x7f4154103e30 2024-09-19T06:51:46.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.172+0000 7f4159a00640 1 -- 172.21.15.114:0/835989577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41541a7000 con 0x7f4154103e30 2024-09-19T06:51:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.174+0000 7f415da41740 1 Processor -- start 2024-09-19T06:51:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f415da41740 1 -- start start 2024-09-19T06:51:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f4151600640 1 -- 172.21.15.114:0/835989577 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f4144042030 con 0x7f4154103e30 2024-09-19T06:51:46.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f415da41740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbaa78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f415da41740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa7e00 0x5563bbab2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f415da41740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 0x5563bbab4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f415da41740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5563bba0d0b0 con 0x5563bbaa74e0 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f415da41740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5563bb9b4ab0 con 0x5563bbaa7e00 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f415da41740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5563bba09070 con 0x5563bbab26c0 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f4159000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbaa78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f4153e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa7e00 0x5563bbab2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f4159000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbaa78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43988/0 (socket says 172.21.15.114:43988) 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f4159000640 1 -- 172.21.15.114:0/1516742663 learned_addr learned my addr 172.21.15.114:0/1516742663 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.175+0000 7f414be00640 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 0x5563bbab4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.176+0000 7f4159000640 1 -- 172.21.15.114:0/1516742663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 msgr2=0x5563bbab4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.176+0000 7f4159000640 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 0x5563bbab4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.176+0000 7f4159000640 1 -- 172.21.15.114:0/1516742663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa7e00 msgr2=0x5563bbab2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.176+0000 7f4159000640 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa7e00 0x5563bbab2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.176+0000 7f4159000640 1 -- 172.21.15.114:0/1516742663 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563bb9c14a0 con 0x5563bbaa74e0 2024-09-19T06:51:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.176+0000 7f4153e00640 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa7e00 0x5563bbab2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.176+0000 7f4159000640 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbaa78c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f414c009f50 tx=0x7f414c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60516702ff3ffae9 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:46.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.176+0000 7f4151600640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f41380a32c0 0x7f41380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f4151600640 1 -- 172.21.15.114:0/835989577 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f41440eff80 con 0x7f4154103e30 2024-09-19T06:51:46.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f4149600640 1 -- 172.21.15.114:0/1516742663 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c004d20 con 0x5563bbaa74e0 2024-09-19T06:51:46.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f4149600640 1 -- 172.21.15.114:0/1516742663 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f414c004050 con 0x5563bbaa74e0 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f4149600640 1 -- 172.21.15.114:0/1516742663 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c005280 con 0x5563bbaa74e0 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f4153e00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f41380a32c0 0x7f41380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f415da41740 1 -- 172.21.15.114:0/1516742663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 msgr2=0x5563bbaa78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f415da41740 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbaa78c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f414c009f50 tx=0x7f414c0118d0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f4153e00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f41380a32c0 0x7f41380a5780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f41440bc190 tx=0x7f41440f39c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f415da41740 1 -- 172.21.15.114:0/1516742663 shutdown_connections 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f415da41740 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 0x5563bbab4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f415da41740 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa7e00 0x5563bbab2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f415da41740 1 --2- 172.21.15.114:0/1516742663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbaa78c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.177+0000 7f415da41740 1 -- 172.21.15.114:0/1516742663 >> 172.21.15.114:0/1516742663 conn(0x5563bbaa3a70 msgr2=0x5563bbaa3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.178+0000 7f415da41740 1 -- 172.21.15.114:0/1516742663 shutdown_connections 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.178+0000 7f415da41740 1 -- 172.21.15.114:0/1516742663 wait complete. 2024-09-19T06:51:46.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.178+0000 7f415da41740 1 -- start start 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.178+0000 7f415da41740 1 Processor -- start 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f415da41740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbaaf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f415da41740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa7e00 0x5563bbaafae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f415da41740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 0x5563bbab0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f415da41740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5563bbab5330 con 0x5563bbaa7e00 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f415da41740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5563bb995920 con 0x5563bbaa74e0 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f415da41740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5563bbab54b0 con 0x5563bbab26c0 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f415da41740 10 client.0 _ll_register_callbacks cb 0x5563bb9b98a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbaaf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f414be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa7e00 0x5563bbaafae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbaaf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:43536/0 (socket says 172.21.15.114:43536) 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 -- 172.21.15.114:0/2258845833 learned_addr learned my addr 172.21.15.114:0/2258845833 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 0x5563bbab0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 -- 172.21.15.114:0/2258845833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 msgr2=0x5563bbab0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 0x5563bbab0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 -- 172.21.15.114:0/2258845833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa7e00 msgr2=0x5563bbaafae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa7e00 0x5563bbaafae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f4159000640 1 -- 172.21.15.114:0/2258845833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563bb77e2b0 con 0x5563bbaa74e0 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.179+0000 7f414be00640 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa7e00 0x5563bbaafae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.180+0000 7f4159000640 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbaaf5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f414c009250 tx=0x7f414c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fb1f350d8f4833b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.180+0000 7f4142000640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-09-19T06:51:46.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.180+0000 7f4142000640 1 -- 172.21.15.114:0/2258845833 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c02b070 con 0x5563bbaa74e0 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.180+0000 7f4142000640 1 -- 172.21.15.114:0/2258845833 <== mon.2 v2:172.21.15.116:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f414c025450 con 0x5563bbaa74e0 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.180+0000 7f415da41740 1 -- 172.21.15.114:0/2258845833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f414c004050 con 0x5563bbaa74e0 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.180+0000 7f4142000640 1 -- 172.21.15.114:0/2258845833 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f414c003e40 con 0x5563bbaa74e0 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.180+0000 7f415da41740 10 client.25151 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.180+0000 7f415da41740 1 -- 172.21.15.114:0/2258845833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5563bbac0fe0 con 0x5563bbaa74e0 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.181+0000 7f4142000640 1 -- 172.21.15.114:0/2258845833 <== mon.2 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f414c004050 con 0x5563bbaa74e0 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.181+0000 7f4142000640 1 -- 172.21.15.114:0/2258845833 <== mon.2 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f414c0276a0 con 0x5563bbaa74e0 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.181+0000 7f415da41740 10 client.25151 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.181+0000 7f415da41740 10 client.25151 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.181+0000 7f415da41740 20 client.25151 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.181+0000 7f415da41740 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5563bbac3420 0x5563bbac4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.181+0000 7f415da41740 4 client.25151 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:46.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.181+0000 7f415da41740 1 -- 172.21.15.114:0/2258845833 --> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563bba1a620 con 0x5563bbac3420 2024-09-19T06:51:46.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.182+0000 7f414be00640 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5563bbac3420 0x5563bbac4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.182+0000 7f414be00640 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5563bbac3420 0x5563bbac4890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36a7a58a962c74e5 server_cookie=c4dac23958a2a9e3 in_seq=0 out_seq=0 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.182+0000 7f4142000640 10 client.25151 ms_handle_connect on v2:172.21.15.118:6836/4109230511 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f4142000640 1 -- 172.21.15.114:0/2258845833 <== mds.2 v2:172.21.15.118:6836/4109230511 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5563bba1a620 con 0x5563bbac3420 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f4142000640 10 client.25151 handle_command_reply: tid=0 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 client.25151 shutdown 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 -- 172.21.15.114:0/2258845833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 msgr2=0x5563bbaaf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbaaf5a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f414c009250 tx=0x7f414c0047b0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 -- 172.21.15.114:0/2258845833 shutdown_connections 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbab26c0 0x5563bbab0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5563bbac3420 0x5563bbac4890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa7e00 0x5563bbaafae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 --2- 172.21.15.114:0/2258845833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbaaf5a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f4142000640 0 client.25151 ms_handle_reset on v2:172.21.15.118:6836/4109230511 2024-09-19T06:51:46.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.183+0000 7f415da41740 1 -- 172.21.15.114:0/2258845833 >> 172.21.15.114:0/2258845833 conn(0x5563bbaa3a70 msgr2=0x5563bbaa2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:46.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.184+0000 7f415da41740 1 -- 172.21.15.114:0/2258845833 shutdown_connections 2024-09-19T06:51:46.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.184+0000 7f415da41740 1 -- 172.21.15.114:0/2258845833 wait complete. 2024-09-19T06:51:46.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.184+0000 7f415da41740 20 client.25151 trim_cache size 0 max 16384 2024-09-19T06:51:46.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.202+0000 7f415da41740 1 Processor -- start 2024-09-19T06:51:46.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.202+0000 7f415da41740 1 -- start start 2024-09-19T06:51:46.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.202+0000 7f415da41740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbab1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.202+0000 7f415da41740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbaafe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.202+0000 7f415da41740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbab26c0 0x5563bbaccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.202+0000 7f415da41740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5563bbac2bd0 con 0x5563bbab26c0 2024-09-19T06:51:46.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.202+0000 7f415da41740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5563bbac32a0 con 0x5563bbaa74e0 2024-09-19T06:51:46.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.202+0000 7f415da41740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5563bbac3970 con 0x5563bbaa7e00 2024-09-19T06:51:46.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbab26c0 0x5563bbaccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbab26c0 0x5563bbaccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44010/0 (socket says 172.21.15.114:44010) 2024-09-19T06:51:46.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 -- 172.21.15.114:0/261704541 learned_addr learned my addr 172.21.15.114:0/261704541 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:46.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f414be00640 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbaafe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbab1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 -- 172.21.15.114:0/261704541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 msgr2=0x5563bbaafe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbaafe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 -- 172.21.15.114:0/261704541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 msgr2=0x5563bbab1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbab1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 -- 172.21.15.114:0/261704541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563bbaa4700 con 0x5563bbab26c0 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f414be00640 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbaafe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.203+0000 7f4159000640 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbab26c0 0x5563bbaccb70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f414c004910 tx=0x7f414c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786147e593eafc90 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f4148c00640 1 -- 172.21.15.114:0/261704541 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c02b030 con 0x5563bbab26c0 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f4148c00640 1 -- 172.21.15.114:0/261704541 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f414c0044d0 con 0x5563bbab26c0 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f4148c00640 1 -- 172.21.15.114:0/261704541 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c025450 con 0x5563bbab26c0 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f415da41740 1 -- 172.21.15.114:0/261704541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbab26c0 msgr2=0x5563bbaccb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f415da41740 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbab26c0 0x5563bbaccb70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f414c004910 tx=0x7f414c0118d0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f415da41740 1 -- 172.21.15.114:0/261704541 shutdown_connections 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f415da41740 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbab26c0 0x5563bbaccb70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f415da41740 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbaafe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f415da41740 1 --2- 172.21.15.114:0/261704541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbaa74e0 0x5563bbab1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.204+0000 7f415da41740 1 -- 172.21.15.114:0/261704541 >> 172.21.15.114:0/261704541 conn(0x5563bbaa3a70 msgr2=0x5563bbab0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.205+0000 7f415da41740 1 -- 172.21.15.114:0/261704541 shutdown_connections 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.205+0000 7f415da41740 1 -- 172.21.15.114:0/261704541 wait complete. 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.205+0000 7f415da41740 1 -- start start 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.205+0000 7f415da41740 1 Processor -- start 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.205+0000 7f415da41740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbac6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f415da41740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbac6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f415da41740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbab26c0 0x5563bbac7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f415da41740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5563bbac4710 con 0x5563bbaa74e0 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f415da41740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5563bbac4040 con 0x5563bbab26c0 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f415da41740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x5563bbac4de0 con 0x5563bbaa7e00 2024-09-19T06:51:46.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f415da41740 10 client.0 _ll_register_callbacks cb 0x7f414c009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbac6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbac6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44022/0 (socket says 172.21.15.114:44022) 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 -- 172.21.15.114:0/3244842727 learned_addr learned my addr 172.21.15.114:0/3244842727 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f414be00640 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbac6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbab26c0 0x5563bbac7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 -- 172.21.15.114:0/3244842727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 msgr2=0x5563bbac6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbac6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 -- 172.21.15.114:0/3244842727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbab26c0 msgr2=0x5563bbac7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbab26c0 0x5563bbac7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.206+0000 7f4159000640 1 -- 172.21.15.114:0/3244842727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563bbacb1a0 con 0x5563bbaa74e0 2024-09-19T06:51:46.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f4159000640 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbac6900 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f414c003e10 tx=0x7f414c00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d07774996346d5b server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f4142000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f4142000640 1 -- 172.21.15.114:0/3244842727 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c02b030 con 0x5563bbaa74e0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f4142000640 1 -- 172.21.15.114:0/3244842727 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f414c00f700 con 0x5563bbaa74e0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f415da41740 1 -- 172.21.15.114:0/3244842727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5563bba14b30 con 0x5563bbaa74e0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f4142000640 1 -- 172.21.15.114:0/3244842727 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f414c0046d0 con 0x5563bbaa74e0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f415da41740 10 client.15162 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f415da41740 1 -- 172.21.15.114:0/3244842727 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5563bbb2ded0 con 0x5563bbaa74e0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.207+0000 7f4142000640 1 -- 172.21.15.114:0/3244842727 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5563bba14b30 con 0x5563bbaa74e0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.208+0000 7f4142000640 1 -- 172.21.15.114:0/3244842727 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f414c030440 con 0x5563bbaa74e0 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.208+0000 7f415da41740 10 client.15162 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.208+0000 7f415da41740 10 client.15162 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.208+0000 7f415da41740 20 client.15162 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:46.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.208+0000 7f415da41740 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5563bbb32870 0x5563bbb31580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.208+0000 7f415da41740 4 client.15162 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.208+0000 7f415da41740 1 -- 172.21.15.114:0/3244842727 --> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] -- command(tid 0: {"prefix": "status"}) -- 0x5563bb990f80 con 0x5563bbb32870 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.208+0000 7f414be00640 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5563bbb32870 0x5563bbb31580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f414be00640 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5563bbb32870 0x5563bbb31580 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c047f225e5b8850f server_cookie=26afa2a38eff58be in_seq=0 out_seq=0 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f4142000640 10 client.15162 ms_handle_connect on v2:172.21.15.118:6836/4109230511 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f4142000640 1 -- 172.21.15.114:0/3244842727 <== mds.2 v2:172.21.15.118:6836/4109230511 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5563bb990f80 con 0x5563bbb32870 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f4142000640 10 client.15162 handle_command_reply: tid=0 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 client.15162 shutdown 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 -- 172.21.15.114:0/3244842727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 msgr2=0x5563bbac6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbac6900 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f414c003e10 tx=0x7f414c00fcb0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 -- 172.21.15.114:0/3244842727 shutdown_connections 2024-09-19T06:51:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5563bbab26c0 0x5563bbac7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5563bbaa7e00 0x5563bbac6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.118:6836/4109230511,v1:172.21.15.118:6837/4109230511] conn(0x5563bbb32870 0x5563bbb31580 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 --2- 172.21.15.114:0/3244842727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563bbaa74e0 0x5563bbac6900 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 -- 172.21.15.114:0/3244842727 >> 172.21.15.114:0/3244842727 conn(0x5563bbaa3a70 msgr2=0x5563bbabdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f4142000640 0 client.15162 ms_handle_reset on v2:172.21.15.118:6836/4109230511 2024-09-19T06:51:46.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.209+0000 7f415da41740 1 -- 172.21.15.114:0/3244842727 shutdown_connections 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f415da41740 1 -- 172.21.15.114:0/3244842727 wait complete. 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f415da41740 20 client.15162 trim_cache size 0 max 16384 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 2, 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24520, 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:46.217 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 109.99405064699999, 2024-09-19T06:51:46.218 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:46.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:46.218 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:46.218 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 183.639492467 2024-09-19T06:51:46.218 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:46.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 -- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f41380a32c0 msgr2=0x7f41380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f41380a32c0 0x7f41380a5780 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f41440bc190 tx=0x7f41440f39c0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 -- 172.21.15.114:0/835989577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4154103e30 msgr2=0x7f415419a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4154103e30 0x7f415419a480 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4144005940 tx=0x7f41440040c0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 -- 172.21.15.114:0/835989577 shutdown_connections 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f41380a32c0 0x7f41380a5780 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4154108f60 0x7f415419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41541047e0 0x7f415419a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 --2- 172.21.15.114:0/835989577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4154103e30 0x7f415419a480 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 -- 172.21.15.114:0/835989577 >> 172.21.15.114:0/835989577 conn(0x7f41540ff910 msgr2=0x7f4154100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 -- 172.21.15.114:0/835989577 shutdown_connections 2024-09-19T06:51:46.219 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:46.210+0000 7f4159a00640 1 -- 172.21.15.114:0/835989577 wait complete. 2024-09-19T06:51:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:46 smithi114 ceph-mon[30820]: pgmap v382: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:46 smithi116 ceph-mon[35411]: pgmap v382: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:46.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:46 smithi118 ceph-mon[35527]: pgmap v382: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:46.974 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-19T06:51:47.342 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:47.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 -- 172.21.15.114:0/2864929427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a4103e30 msgr2=0x7f91a4104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 --2- 172.21.15.114:0/2864929427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a4103e30 0x7f91a4104210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9194009be0 tx=0x7f919402a6c0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 -- 172.21.15.114:0/2864929427 shutdown_connections 2024-09-19T06:51:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 --2- 172.21.15.114:0/2864929427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a4108f60 0x7f91a4110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 --2- 172.21.15.114:0/2864929427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91a41047e0 0x7f91a4108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 --2- 172.21.15.114:0/2864929427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a4103e30 0x7f91a4104210 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 -- 172.21.15.114:0/2864929427 >> 172.21.15.114:0/2864929427 conn(0x7f91a40ff8d0 msgr2=0x7f91a4101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 -- 172.21.15.114:0/2864929427 shutdown_connections 2024-09-19T06:51:47.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.935+0000 7f91aa400640 1 -- 172.21.15.114:0/2864929427 wait complete. 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.936+0000 7f91aa400640 1 Processor -- start 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.936+0000 7f91aa400640 1 -- start start 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.936+0000 7f91aa400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91a4103e30 0x7f91a419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91aa400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a41047e0 0x7f91a419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91aa400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a4108f60 0x7f91a419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91aa400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f91a4113f70 con 0x7f91a41047e0 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91aa400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91a4113df0 con 0x7f91a4103e30 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91aa400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f91a41140f0 con 0x7f91a4108f60 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a41047e0 0x7f91a419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a41047e0 0x7f91a419a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33000/0 (socket says 172.21.15.114:33000) 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a3e00640 1 -- 172.21.15.114:0/1475542594 learned_addr learned my addr 172.21.15.114:0/1475542594 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:47.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a9000640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91a4103e30 0x7f91a419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a9a00640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a4108f60 0x7f91a419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a3e00640 1 -- 172.21.15.114:0/1475542594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a4108f60 msgr2=0x7f91a419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a3e00640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a4108f60 0x7f91a419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a3e00640 1 -- 172.21.15.114:0/1475542594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91a4103e30 msgr2=0x7f91a419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a3e00640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91a4103e30 0x7f91a419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a3e00640 1 -- 172.21.15.114:0/1475542594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a419f450 con 0x7f91a41047e0 2024-09-19T06:51:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.937+0000 7f91a9000640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91a4103e30 0x7f91a419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:47.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.938+0000 7f91a3e00640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a41047e0 0x7f91a419a9e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f919000e9f0 tx=0x7f919000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.938+0000 7f91a1600640 1 -- 172.21.15.114:0/1475542594 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919000ccc0 con 0x7f91a41047e0 2024-09-19T06:51:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.938+0000 7f91a1600640 1 -- 172.21.15.114:0/1475542594 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f919000ce60 con 0x7f91a41047e0 2024-09-19T06:51:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.938+0000 7f91a1600640 1 -- 172.21.15.114:0/1475542594 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190010690 con 0x7f91a41047e0 2024-09-19T06:51:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.938+0000 7f91aa400640 1 -- 172.21.15.114:0/1475542594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a419f740 con 0x7f91a41047e0 2024-09-19T06:51:47.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.938+0000 7f91aa400640 1 -- 172.21.15.114:0/1475542594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a41a7020 con 0x7f91a41047e0 2024-09-19T06:51:47.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.941+0000 7f91a1600640 1 -- 172.21.15.114:0/1475542594 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f91900046c0 con 0x7f91a41047e0 2024-09-19T06:51:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.941+0000 7f91ae3bd740 1 Processor -- start 2024-09-19T06:51:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.942+0000 7f91ae3bd740 1 -- start start 2024-09-19T06:51:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.942+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa264e0 0x55ae5aa268c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.942+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa31180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.942+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 0x55ae5aa33ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.942+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ae5a98c0b0 con 0x55ae5aa26e00 2024-09-19T06:51:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.942+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ae5a933ab0 con 0x55ae5aa264e0 2024-09-19T06:51:47.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.942+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ae5a988070 con 0x55ae5aa316c0 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.942+0000 7f91a9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa31180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa264e0 0x55ae5aa268c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa264e0 0x55ae5aa268c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:54070/0 (socket says 172.21.15.114:54070) 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa31180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33006/0 (socket says 172.21.15.114:33006) 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 -- 172.21.15.114:0/1402868605 learned_addr learned my addr 172.21.15.114:0/1402868605 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a3e00640 1 --2- 172.21.15.114:0/1402868605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 0x55ae5aa33ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 -- 172.21.15.114:0/1402868605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 msgr2=0x55ae5aa33ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 --2- 172.21.15.114:0/1402868605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 0x55ae5aa33ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 -- 172.21.15.114:0/1402868605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 msgr2=0x55ae5aa31180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 --2- 172.21.15.114:0/1402868605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa31180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 -- 172.21.15.114:0/1402868605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae5a9404a0 con 0x55ae5aa264e0 2024-09-19T06:51:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a1600640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91880a32c0 0x7f91880a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a9a00640 1 --2- 172.21.15.114:0/1402868605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa264e0 0x55ae5aa268c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f919c009be0 tx=0x7f919c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d49cf21bc545baac server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.943+0000 7f91a1600640 1 -- 172.21.15.114:0/1475542594 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f91900d00c0 con 0x7f91a41047e0 2024-09-19T06:51:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f919a000640 1 -- 172.21.15.114:0/1402868605 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c01e070 con 0x55ae5aa264e0 2024-09-19T06:51:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91a9000640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91880a32c0 0x7f91880a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f919a000640 1 -- 172.21.15.114:0/1402868605 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f919c009e50 con 0x55ae5aa264e0 2024-09-19T06:51:47.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f919a000640 1 -- 172.21.15.114:0/1402868605 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c00e9e0 con 0x55ae5aa264e0 2024-09-19T06:51:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91ae3bd740 1 -- 172.21.15.114:0/1402868605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa264e0 msgr2=0x55ae5aa268c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/1402868605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa264e0 0x55ae5aa268c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f919c009be0 tx=0x7f919c011810 comp rx=0 tx=0).stop 2024-09-19T06:51:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91a9000640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91880a32c0 0x7f91880a5780 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9194000f80 tx=0x7f9194005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91ae3bd740 1 -- 172.21.15.114:0/1402868605 shutdown_connections 2024-09-19T06:51:47.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/1402868605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 0x55ae5aa33ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/1402868605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa31180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/1402868605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa264e0 0x55ae5aa268c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.944+0000 7f91ae3bd740 1 -- 172.21.15.114:0/1402868605 >> 172.21.15.114:0/1402868605 conn(0x55ae5aa22a70 msgr2=0x55ae5aa22e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.945+0000 7f91ae3bd740 1 -- 172.21.15.114:0/1402868605 shutdown_connections 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.945+0000 7f91ae3bd740 1 -- 172.21.15.114:0/1402868605 wait complete. 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.945+0000 7f91ae3bd740 1 -- start start 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.945+0000 7f91ae3bd740 1 Processor -- start 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa264e0 0x55ae5aa2e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 0x55ae5aa2f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ae5aa34330 con 0x55ae5aa26e00 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ae5a914920 con 0x55ae5aa316c0 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ae5aa344b0 con 0x55ae5aa264e0 2024-09-19T06:51:47.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91ae3bd740 10 client.0 _ll_register_callbacks cb 0x55ae5a9388a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33020/0 (socket says 172.21.15.114:33020) 2024-09-19T06:51:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a3e00640 1 -- 172.21.15.114:0/3650394263 learned_addr learned my addr 172.21.15.114:0/3650394263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a9000640 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa264e0 0x55ae5aa2e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a9a00640 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 0x55ae5aa2f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a3e00640 1 -- 172.21.15.114:0/3650394263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa264e0 msgr2=0x55ae5aa2e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:47.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a3e00640 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa264e0 0x55ae5aa2e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a3e00640 1 -- 172.21.15.114:0/3650394263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 msgr2=0x55ae5aa2f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a3e00640 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 0x55ae5aa2f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.946+0000 7f91a3e00640 1 -- 172.21.15.114:0/3650394263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae5a6fd2b0 con 0x55ae5aa26e00 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f91a3e00640 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2eac0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f919009bf90 tx=0x7f919009be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b529577a05b97e48 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f918ea00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f918ea00640 1 -- 172.21.15.114:0/3650394263 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919009c290 con 0x55ae5aa26e00 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f918ea00640 1 -- 172.21.15.114:0/3650394263 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f91900d20d0 con 0x55ae5aa26e00 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f91ae3bd740 1 -- 172.21.15.114:0/3650394263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f919c009e50 con 0x55ae5aa26e00 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f918ea00640 1 -- 172.21.15.114:0/3650394263 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f91900a0640 con 0x55ae5aa26e00 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f91ae3bd740 10 client.15174 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f91ae3bd740 1 -- 172.21.15.114:0/3650394263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ae5aa3ffb0 con 0x55ae5aa26e00 2024-09-19T06:51:47.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.947+0000 7f918ea00640 1 -- 172.21.15.114:0/3650394263 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f919c009e50 con 0x55ae5aa26e00 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.948+0000 7f918ea00640 1 -- 172.21.15.114:0/3650394263 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f91900a4ae0 con 0x55ae5aa26e00 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.948+0000 7f91ae3bd740 10 client.15174 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.948+0000 7f91ae3bd740 10 client.15174 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.948+0000 7f91ae3bd740 20 client.15174 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.948+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55ae5aa425a0 0x55ae5aa43a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.948+0000 7f91ae3bd740 4 client.15174 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.948+0000 7f91ae3bd740 1 -- 172.21.15.114:0/3650394263 --> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae5a999620 con 0x55ae5aa425a0 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.948+0000 7f91a9000640 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55ae5aa425a0 0x55ae5aa43a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.949+0000 7f91a9000640 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55ae5aa425a0 0x55ae5aa43a10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf467270d6efb90e server_cookie=e56aba51ee358f2a in_seq=0 out_seq=0 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.949+0000 7f918ea00640 10 client.15174 ms_handle_connect on v2:172.21.15.116:6838/2386737584 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f918ea00640 1 -- 172.21.15.114:0/3650394263 <== mds.1 v2:172.21.15.116:6838/2386737584 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ae5a999620 con 0x55ae5aa425a0 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f918ea00640 10 client.15174 handle_command_reply: tid=0 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 client.15174 shutdown 2024-09-19T06:51:47.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 -- 172.21.15.114:0/3650394263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 msgr2=0x55ae5aa2eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2eac0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f919009bf90 tx=0x7f919009be90 comp rx=0 tx=0).stop 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 -- 172.21.15.114:0/3650394263 shutdown_connections 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55ae5aa425a0 0x55ae5aa43a10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 0x55ae5aa2f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2eac0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f918ea00640 0 client.15174 ms_handle_reset on v2:172.21.15.116:6838/2386737584 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/3650394263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa264e0 0x55ae5aa2e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 -- 172.21.15.114:0/3650394263 >> 172.21.15.114:0/3650394263 conn(0x55ae5aa22a70 msgr2=0x55ae5aa21bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 -- 172.21.15.114:0/3650394263 shutdown_connections 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 1 -- 172.21.15.114:0/3650394263 wait complete. 2024-09-19T06:51:47.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.950+0000 7f91ae3bd740 20 client.15174 trim_cache size 0 max 16384 2024-09-19T06:51:47.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91ae3bd740 1 Processor -- start 2024-09-19T06:51:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91ae3bd740 1 -- start start 2024-09-19T06:51:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa28f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 0x55ae5aa2d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ae5aa41d50 con 0x55ae5aa264e0 2024-09-19T06:51:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ae5aa42420 con 0x55ae5aa26e00 2024-09-19T06:51:47.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ae5aa42af0 con 0x55ae5aa316c0 2024-09-19T06:51:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91a9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa28f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91a9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa28f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33022/0 (socket says 172.21.15.114:33022) 2024-09-19T06:51:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91a9000640 1 -- 172.21.15.114:0/452248410 learned_addr learned my addr 172.21.15.114:0/452248410 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.969+0000 7f91a9a00640 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 0x55ae5aa2d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f91a3e00640 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f91a9000640 1 -- 172.21.15.114:0/452248410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 msgr2=0x55ae5aa2d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:47.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f91a9000640 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 0x55ae5aa2d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f91a9000640 1 -- 172.21.15.114:0/452248410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa26e00 msgr2=0x55ae5aa2ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f91a9000640 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f91a9000640 1 -- 172.21.15.114:0/452248410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae5aa2cad0 con 0x55ae5aa264e0 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f91a9000640 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa28f90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9194030630 tx=0x7f9194041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9120b4a9afe955e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f9199600640 1 -- 172.21.15.114:0/452248410 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919405b070 con 0x55ae5aa264e0 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f9199600640 1 -- 172.21.15.114:0/452248410 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9194037040 con 0x55ae5aa264e0 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.970+0000 7f9199600640 1 -- 172.21.15.114:0/452248410 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194036050 con 0x55ae5aa264e0 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 -- 172.21.15.114:0/452248410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 msgr2=0x55ae5aa28f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa28f90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9194030630 tx=0x7f9194041000 comp rx=0 tx=0).stop 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 -- 172.21.15.114:0/452248410 shutdown_connections 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa316c0 0x55ae5aa2d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa26e00 0x55ae5aa2ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/452248410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa28f90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 -- 172.21.15.114:0/452248410 >> 172.21.15.114:0/452248410 conn(0x55ae5aa22a70 msgr2=0x55ae5aa299a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 -- 172.21.15.114:0/452248410 shutdown_connections 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.971+0000 7f91ae3bd740 1 -- 172.21.15.114:0/452248410 wait complete. 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 1 -- start start 2024-09-19T06:51:47.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 1 Processor -- start 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa45b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa26e00 0x55ae5aa460c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 0x55ae5aa46600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ae5aa43890 con 0x55ae5aa264e0 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ae5aa431c0 con 0x55ae5aa316c0 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55ae5aa43f60 con 0x55ae5aa26e00 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91ae3bd740 10 client.0 _ll_register_callbacks cb 0x7f9190098530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91a9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa45b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91a9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa45b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33032/0 (socket says 172.21.15.114:33032) 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.972+0000 7f91a9000640 1 -- 172.21.15.114:0/2348857408 learned_addr learned my addr 172.21.15.114:0/2348857408 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a3e00640 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa26e00 0x55ae5aa460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a9a00640 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 0x55ae5aa46600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a3e00640 1 -- 172.21.15.114:0/2348857408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 msgr2=0x55ae5aa46600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a3e00640 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 0x55ae5aa46600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a3e00640 1 -- 172.21.15.114:0/2348857408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 msgr2=0x55ae5aa45b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a3e00640 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa45b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a3e00640 1 -- 172.21.15.114:0/2348857408 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae5aa4a480 con 0x55ae5aa26e00 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a9000640 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa45b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91a3e00640 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa26e00 0x55ae5aa460c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f919009a540 tx=0x7f91900a4ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c8bfe22e383be0d server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f918ea00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f918ea00640 1 -- 172.21.15.114:0/2348857408 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919009c290 con 0x55ae5aa26e00 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f918ea00640 1 -- 172.21.15.114:0/2348857408 <== mon.1 v2:172.21.15.118:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f91900a01f0 con 0x55ae5aa26e00 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.973+0000 7f91ae3bd740 1 -- 172.21.15.114:0/2348857408 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9194037040 con 0x55ae5aa26e00 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f918ea00640 1 -- 172.21.15.114:0/2348857408 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f91900a4f00 con 0x55ae5aa26e00 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f91ae3bd740 10 client.25219 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:47.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f91ae3bd740 1 -- 172.21.15.114:0/2348857408 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ae5aaad150 con 0x55ae5aa26e00 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f918ea00640 1 -- 172.21.15.114:0/2348857408 <== mon.1 v2:172.21.15.118:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9194037040 con 0x55ae5aa26e00 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f918ea00640 1 -- 172.21.15.114:0/2348857408 <== mon.1 v2:172.21.15.118:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f919009b6e0 con 0x55ae5aa26e00 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f91ae3bd740 10 client.25219 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f91ae3bd740 10 client.25219 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f91ae3bd740 20 client.25219 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55ae5aab1ce0 0x55ae5aab09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f91ae3bd740 4 client.25219 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.974+0000 7f91ae3bd740 1 -- 172.21.15.114:0/2348857408 --> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] -- command(tid 0: {"prefix": "status"}) -- 0x55ae5a993b30 con 0x55ae5aab1ce0 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.975+0000 7f91a9000640 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55ae5aab1ce0 0x55ae5aab09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.975+0000 7f91a9000640 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55ae5aab1ce0 0x55ae5aab09f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c36242ab42881f55 server_cookie=1e907e50466815c6 in_seq=0 out_seq=0 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.975+0000 7f918ea00640 10 client.25219 ms_handle_connect on v2:172.21.15.116:6838/2386737584 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.975+0000 7f918ea00640 1 -- 172.21.15.114:0/2348857408 <== mds.1 v2:172.21.15.116:6838/2386737584 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55ae5a993b30 con 0x55ae5aab1ce0 2024-09-19T06:51:47.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.975+0000 7f918ea00640 10 client.25219 handle_command_reply: tid=0 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.975+0000 7f91ae3bd740 1 client.25219 shutdown 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.975+0000 7f91ae3bd740 1 -- 172.21.15.114:0/2348857408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa26e00 msgr2=0x55ae5aa460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.975+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa26e00 0x55ae5aa460c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f919009a540 tx=0x7f91900a4ca0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 1 -- 172.21.15.114:0/2348857408 shutdown_connections 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.116:6838/2386737584,v1:172.21.15.116:6839/2386737584] conn(0x55ae5aab1ce0 0x55ae5aab09f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ae5aa316c0 0x55ae5aa46600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ae5aa26e00 0x55ae5aa460c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f918ea00640 0 client.25219 ms_handle_reset on v2:172.21.15.116:6838/2386737584 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 1 --2- 172.21.15.114:0/2348857408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5aa264e0 0x55ae5aa45b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 1 -- 172.21.15.114:0/2348857408 >> 172.21.15.114:0/2348857408 conn(0x55ae5aa22a70 msgr2=0x55ae5aa2ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 1 -- 172.21.15.114:0/2348857408 shutdown_connections 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 1 -- 172.21.15.114:0/2348857408 wait complete. 2024-09-19T06:51:47.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91ae3bd740 20 client.25219 trim_cache size 0 max 16384 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 1, 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24524, 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 112.765176485, 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:47.984 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:47.985 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 169.350788956 2024-09-19T06:51:47.985 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:47.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91aa400640 1 -- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91880a32c0 msgr2=0x7f91880a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91aa400640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91880a32c0 0x7f91880a5780 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9194000f80 tx=0x7f9194005e30 comp rx=0 tx=0).stop 2024-09-19T06:51:47.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91aa400640 1 -- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a41047e0 msgr2=0x7f91a419a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:47.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.976+0000 7f91aa400640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a41047e0 0x7f91a419a9e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f919000e9f0 tx=0x7f919000edf0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.977+0000 7f91aa400640 1 -- 172.21.15.114:0/1475542594 shutdown_connections 2024-09-19T06:51:47.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.977+0000 7f91aa400640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f91880a32c0 0x7f91880a5780 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.977+0000 7f91aa400640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91a4108f60 0x7f91a419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.977+0000 7f91aa400640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f91a41047e0 0x7f91a419a9e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.977+0000 7f91aa400640 1 --2- 172.21.15.114:0/1475542594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91a4103e30 0x7f91a419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:47.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.977+0000 7f91aa400640 1 -- 172.21.15.114:0/1475542594 >> 172.21.15.114:0/1475542594 conn(0x7f91a40ff8d0 msgr2=0x7f91a4100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:47.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.977+0000 7f91aa400640 1 -- 172.21.15.114:0/1475542594 shutdown_connections 2024-09-19T06:51:47.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:47.977+0000 7f91aa400640 1 -- 172.21.15.114:0/1475542594 wait complete. 2024-09-19T06:51:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:48 smithi114 ceph-mon[30820]: pgmap v383: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:51:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:48 smithi116 ceph-mon[35411]: pgmap v383: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:51:48.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-19T06:51:48.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:48 smithi118 ceph-mon[35527]: pgmap v383: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:51:49.076 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:49.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.685+0000 7f8863800640 1 -- 172.21.15.114:0/4253177273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c10e3c0 msgr2=0x7f885c110810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.685+0000 7f8863800640 1 --2- 172.21.15.114:0/4253177273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c10e3c0 0x7f885c110810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8854009ca0 tx=0x7f885402a940 comp rx=0 tx=0).stop 2024-09-19T06:51:49.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.686+0000 7f8863800640 1 -- 172.21.15.114:0/4253177273 shutdown_connections 2024-09-19T06:51:49.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.686+0000 7f8863800640 1 --2- 172.21.15.114:0/4253177273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c10e3c0 0x7f885c110810 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.686+0000 7f8863800640 1 --2- 172.21.15.114:0/4253177273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f885c074f40 0x7f885c10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.686+0000 7f8863800640 1 --2- 172.21.15.114:0/4253177273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f885c074620 0x7f885c074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.686+0000 7f8863800640 1 -- 172.21.15.114:0/4253177273 >> 172.21.15.114:0/4253177273 conn(0x7f885c0fd8c0 msgr2=0x7f885c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:49.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.686+0000 7f8863800640 1 -- 172.21.15.114:0/4253177273 shutdown_connections 2024-09-19T06:51:49.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.686+0000 7f8863800640 1 -- 172.21.15.114:0/4253177273 wait complete. 2024-09-19T06:51:49.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.686+0000 7f8863800640 1 Processor -- start 2024-09-19T06:51:49.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8863800640 1 -- start start 2024-09-19T06:51:49.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8863800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f885c074620 0x7f885c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8863800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f885c074f40 0x7f885c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8863800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c10e3c0 0x7f885c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8863800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f885c1141b0 con 0x7f885c10e3c0 2024-09-19T06:51:49.689 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8863800640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f885c114030 con 0x7f885c074620 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8863800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f885c114330 con 0x7f885c074f40 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f885be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f885c074f40 0x7f885c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8860e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f885c074620 0x7f885c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f885be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f885c074f40 0x7f885c19ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:56294/0 (socket says 172.21.15.114:56294) 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f885be00640 1 -- 172.21.15.114:0/968390205 learned_addr learned my addr 172.21.15.114:0/968390205 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.687+0000 7f8861800640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c10e3c0 0x7f885c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8860e00640 1 -- 172.21.15.114:0/968390205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f885c074f40 msgr2=0x7f885c19ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8860e00640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f885c074f40 0x7f885c19ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8860e00640 1 -- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c10e3c0 msgr2=0x7f885c19ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8860e00640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c10e3c0 0x7f885c19ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8860e00640 1 -- 172.21.15.114:0/968390205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885c19f5e0 con 0x7f885c074620 2024-09-19T06:51:49.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8860e00640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f885c074620 0x7f885c19a630 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f884c00e9f0 tx=0x7f884c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:49.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8859600640 1 -- 172.21.15.114:0/968390205 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c00ccc0 con 0x7f885c074620 2024-09-19T06:51:49.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8863800640 1 -- 172.21.15.114:0/968390205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885c19f8d0 con 0x7f885c074620 2024-09-19T06:51:49.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8859600640 1 -- 172.21.15.114:0/968390205 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f884c005020 con 0x7f885c074620 2024-09-19T06:51:49.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.688+0000 7f8859600640 1 -- 172.21.15.114:0/968390205 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c0049e0 con 0x7f885c074620 2024-09-19T06:51:49.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.689+0000 7f8863800640 1 -- 172.21.15.114:0/968390205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885c1a71b0 con 0x7f885c074620 2024-09-19T06:51:49.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.692+0000 7f8866286740 1 Processor -- start 2024-09-19T06:51:49.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.692+0000 7f8866286740 1 -- start start 2024-09-19T06:51:49.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8866286740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88e330 0x55643a88e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8866286740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88ec50 0x55643a898fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8866286740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 0x55643a89b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8866286740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55643a7f40b0 con 0x55643a899510 2024-09-19T06:51:49.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8866286740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55643a79bab0 con 0x55643a88ec50 2024-09-19T06:51:49.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8866286740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55643a7f0070 con 0x55643a88e330 2024-09-19T06:51:49.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8859600640 1 -- 172.21.15.114:0/968390205 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f884c005560 con 0x7f885c074620 2024-09-19T06:51:49.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f885be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 0x55643a89b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8861800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88e330 0x55643a88e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8861800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88e330 0x55643a88e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:56298/0 (socket says 172.21.15.114:56298) 2024-09-19T06:51:49.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.693+0000 7f8861800640 1 -- 172.21.15.114:0/1919000895 learned_addr learned my addr 172.21.15.114:0/1919000895 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:49.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8861800640 1 -- 172.21.15.114:0/1919000895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88ec50 msgr2=0x55643a898fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:49.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8861800640 1 --2- 172.21.15.114:0/1919000895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88ec50 0x55643a898fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8861800640 1 -- 172.21.15.114:0/1919000895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 msgr2=0x55643a89b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:49.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8861800640 1 --2- 172.21.15.114:0/1919000895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 0x55643a89b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8861800640 1 -- 172.21.15.114:0/1919000895 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55643a7a84a0 con 0x55643a88e330 2024-09-19T06:51:49.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8861800640 1 --2- 172.21.15.114:0/1919000895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88e330 0x55643a88e710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8854009ca0 tx=0x7f8854005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4762ab392024f94c server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:49.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8852000640 1 -- 172.21.15.114:0/1919000895 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854034040 con 0x55643a88e330 2024-09-19T06:51:49.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8852000640 1 -- 172.21.15.114:0/1919000895 <== mon.1 v2:172.21.15.118:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8854003970 con 0x55643a88e330 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.694+0000 7f8852000640 1 -- 172.21.15.114:0/1919000895 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885402fed0 con 0x55643a88e330 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8859600640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88400a32c0 0x7f88400a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8859600640 1 -- 172.21.15.114:0/968390205 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f884c0cfc30 con 0x7f885c074620 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8860e00640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88400a32c0 0x7f88400a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8866286740 1 -- 172.21.15.114:0/1919000895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88e330 msgr2=0x55643a88e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8866286740 1 --2- 172.21.15.114:0/1919000895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88e330 0x55643a88e710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8854009ca0 tx=0x7f8854005f80 comp rx=0 tx=0).stop 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8866286740 1 -- 172.21.15.114:0/1919000895 shutdown_connections 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8866286740 1 --2- 172.21.15.114:0/1919000895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 0x55643a89b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8866286740 1 --2- 172.21.15.114:0/1919000895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88ec50 0x55643a898fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8866286740 1 --2- 172.21.15.114:0/1919000895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88e330 0x55643a88e710 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.695+0000 7f8866286740 1 -- 172.21.15.114:0/1919000895 >> 172.21.15.114:0/1919000895 conn(0x55643a88a8c0 msgr2=0x55643a88aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.696+0000 7f8866286740 1 -- 172.21.15.114:0/1919000895 shutdown_connections 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.696+0000 7f8866286740 1 -- 172.21.15.114:0/1919000895 wait complete. 2024-09-19T06:51:49.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.696+0000 7f8860e00640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88400a32c0 0x7f88400a5780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f884c0a4540 tx=0x7f884c0cd1b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:49.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.696+0000 7f8866286740 1 -- start start 2024-09-19T06:51:49.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.696+0000 7f8866286740 1 Processor -- start 2024-09-19T06:51:49.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8866286740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88e330 0x55643a8964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8866286740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a896a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8866286740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 0x55643a896f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8866286740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55643a89c060 con 0x55643a899510 2024-09-19T06:51:49.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8866286740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55643a77c920 con 0x55643a88e330 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8866286740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55643a89c1e0 con 0x55643a88ec50 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8866286740 10 client.0 _ll_register_callbacks cb 0x55643a7a08a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8861800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 0x55643a896f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f885be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a896a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8861800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88e330 0x55643a8964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f885be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a896a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:56306/0 (socket says 172.21.15.114:56306) 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f8861800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88e330 0x55643a8964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:54132/0 (socket says 172.21.15.114:54132) 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.697+0000 7f885be00640 1 -- 172.21.15.114:0/2202030156 learned_addr learned my addr 172.21.15.114:0/2202030156 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f885be00640 1 -- 172.21.15.114:0/2202030156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88e330 msgr2=0x55643a8964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f885be00640 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88e330 0x55643a8964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f885be00640 1 -- 172.21.15.114:0/2202030156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 msgr2=0x55643a896f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:49.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f885be00640 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 0x55643a896f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f885be00640 1 -- 172.21.15.114:0/2202030156 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55643a5652b0 con 0x55643a88ec50 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f8861800640 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88e330 0x55643a8964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f885be00640 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a896a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8848012750 tx=0x7f8848012c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f6e886225c0430e server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f8846a00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f8846a00640 1 -- 172.21.15.114:0/2202030156 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848012e60 con 0x55643a88ec50 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f8866286740 1 -- 172.21.15.114:0/2202030156 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8854003970 con 0x55643a88ec50 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f8846a00640 1 -- 172.21.15.114:0/2202030156 <== mon.1 v2:172.21.15.118:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f884800f920 con 0x55643a88ec50 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f8846a00640 1 -- 172.21.15.114:0/2202030156 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f884800fb20 con 0x55643a88ec50 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f8866286740 10 client.25231 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.698+0000 7f8866286740 1 -- 172.21.15.114:0/2202030156 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55643a8a7ea0 con 0x55643a88ec50 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.699+0000 7f8846a00640 1 -- 172.21.15.114:0/2202030156 <== mon.1 v2:172.21.15.118:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8854003970 con 0x55643a88ec50 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.699+0000 7f8846a00640 1 -- 172.21.15.114:0/2202030156 <== mon.1 v2:172.21.15.118:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f88480049e0 con 0x55643a88ec50 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.699+0000 7f8866286740 10 client.25231 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:49.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.699+0000 7f8866286740 10 client.25231 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-19T06:51:49.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.699+0000 7f8866286740 20 client.25231 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:49.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.699+0000 7f8866286740 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x55643a8aa480 0x55643a8ab8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.699+0000 7f8866286740 4 client.25231 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-19T06:51:49.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.699+0000 7f8866286740 1 -- 172.21.15.114:0/2202030156 --> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55643a801620 con 0x55643a8aa480 2024-09-19T06:51:49.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.700+0000 7f8861800640 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x55643a8aa480 0x55643a8ab8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.700+0000 7f8861800640 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x55643a8aa480 0x55643a8ab8f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95b3129f684d65b4 server_cookie=536f139ef5276524 in_seq=0 out_seq=0 2024-09-19T06:51:49.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.700+0000 7f8846a00640 10 client.25231 ms_handle_connect on v2:172.21.15.118:6838/345383252 2024-09-19T06:51:49.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8846a00640 1 -- 172.21.15.114:0/2202030156 <== mds.0 v2:172.21.15.118:6838/345383252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55643a801620 con 0x55643a8aa480 2024-09-19T06:51:49.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8846a00640 10 client.25231 handle_command_reply: tid=0 2024-09-19T06:51:49.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 client.25231 shutdown 2024-09-19T06:51:49.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 -- 172.21.15.114:0/2202030156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 msgr2=0x55643a896a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a896a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8848012750 tx=0x7f8848012c10 comp rx=0 tx=0).stop 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 -- 172.21.15.114:0/2202030156 shutdown_connections 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x55643a8aa480 0x55643a8ab8f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a899510 0x55643a896f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a896a10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8846a00640 0 client.25231 ms_handle_reset on v2:172.21.15.118:6838/345383252 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 --2- 172.21.15.114:0/2202030156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a88e330 0x55643a8964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 -- 172.21.15.114:0/2202030156 >> 172.21.15.114:0/2202030156 conn(0x55643a88a8c0 msgr2=0x55643a8905a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 -- 172.21.15.114:0/2202030156 shutdown_connections 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 1 -- 172.21.15.114:0/2202030156 wait complete. 2024-09-19T06:51:49.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.701+0000 7f8866286740 20 client.25231 trim_cache size 0 max 16384 2024-09-19T06:51:49.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8866286740 1 Processor -- start 2024-09-19T06:51:49.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8866286740 1 -- start start 2024-09-19T06:51:49.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8866286740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8aab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8866286740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a895d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8866286740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 0x55643a894da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8866286740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55643a8a9c30 con 0x55643a88e330 2024-09-19T06:51:49.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8866286740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55643a8aa300 con 0x55643a899510 2024-09-19T06:51:49.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8866286740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55643a8aa9d0 con 0x55643a88ec50 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8861800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8861800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8aab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33084/0 (socket says 172.21.15.114:33084) 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f8861800640 1 -- 172.21.15.114:0/468842986 learned_addr learned my addr 172.21.15.114:0/468842986 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.720+0000 7f885be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a895d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8861800640 1 --2- 172.21.15.114:0/468842986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 0x55643a894da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8861800640 1 -- 172.21.15.114:0/468842986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 msgr2=0x55643a895d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8861800640 1 --2- 172.21.15.114:0/468842986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a895d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8861800640 1 -- 172.21.15.114:0/468842986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 msgr2=0x55643a894da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:49.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8861800640 1 --2- 172.21.15.114:0/468842986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 0x55643a894da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8861800640 1 -- 172.21.15.114:0/468842986 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55643a894a20 con 0x55643a88e330 2024-09-19T06:51:49.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8861800640 1 --2- 172.21.15.114:0/468842986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8aab50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8854004af0 tx=0x7f885402fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c3674a325971a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:49.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8851600640 1 -- 172.21.15.114:0/468842986 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854049040 con 0x55643a88e330 2024-09-19T06:51:49.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.721+0000 7f8851600640 1 -- 172.21.15.114:0/468842986 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8854030640 con 0x55643a88e330 2024-09-19T06:51:49.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8851600640 1 -- 172.21.15.114:0/468842986 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8854004290 con 0x55643a88e330 2024-09-19T06:51:49.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 -- 172.21.15.114:0/468842986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 msgr2=0x55643a8aab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 --2- 172.21.15.114:0/468842986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8aab50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8854004af0 tx=0x7f885402fa70 comp rx=0 tx=0).stop 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 -- 172.21.15.114:0/468842986 shutdown_connections 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 --2- 172.21.15.114:0/468842986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 0x55643a894da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 --2- 172.21.15.114:0/468842986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a895d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 --2- 172.21.15.114:0/468842986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8aab50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 -- 172.21.15.114:0/468842986 >> 172.21.15.114:0/468842986 conn(0x55643a88a8c0 msgr2=0x55643a8917f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 -- 172.21.15.114:0/468842986 shutdown_connections 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.722+0000 7f8866286740 1 -- 172.21.15.114:0/468842986 wait complete. 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 1 -- start start 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 1 Processor -- start 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a8b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 0x55643a892eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55643a8ab770 con 0x55643a88e330 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55643a8ab0a0 con 0x55643a899510 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x55643a8abe40 con 0x55643a88ec50 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8866286740 10 client.0 _ll_register_callbacks cb 0x7f8848009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.723+0000 7f8861800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f885be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a8b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f8861800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8b3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33088/0 (socket says 172.21.15.114:33088) 2024-09-19T06:51:49.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f8861800640 1 -- 172.21.15.114:0/1492422875 learned_addr learned my addr 172.21.15.114:0/1492422875 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f885be00640 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a8b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.114:56320/0 (socket says 172.21.15.114:56320) 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f8861800640 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 0x55643a892eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f885be00640 1 -- 172.21.15.114:0/1492422875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 msgr2=0x55643a892eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f885be00640 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 0x55643a892eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f885be00640 1 -- 172.21.15.114:0/1492422875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 msgr2=0x55643a8b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f885be00640 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f885be00640 1 -- 172.21.15.114:0/1492422875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55643a8936e0 con 0x55643a88ec50 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f8861800640 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f885be00640 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a8b4390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8848004e80 tx=0x7f884801c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e17970ae51cfcc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f8846a00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f8846a00640 1 -- 172.21.15.114:0/1492422875 <== mon.1 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848012e60 con 0x55643a88ec50 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f8846a00640 1 -- 172.21.15.114:0/1492422875 <== mon.1 v2:172.21.15.118:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f884800f920 con 0x55643a88ec50 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.724+0000 7f8866286740 1 -- 172.21.15.114:0/1492422875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8854030640 con 0x55643a88ec50 2024-09-19T06:51:49.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8846a00640 1 -- 172.21.15.114:0/1492422875 <== mon.1 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f884801cb20 con 0x55643a88ec50 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8866286740 10 client.25237 fetch_fsmap learned FSMap version 29 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8866286740 1 -- 172.21.15.114:0/1492422875 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55643a8b3790 con 0x55643a88ec50 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8846a00640 1 -- 172.21.15.114:0/1492422875 <== mon.1 v2:172.21.15.118:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8854030640 con 0x55643a88ec50 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8846a00640 1 -- 172.21.15.114:0/1492422875 <== mon.1 v2:172.21.15.118:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f88480035f0 con 0x55643a88ec50 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8866286740 10 client.25237 fetch_fsmap finished waiting for FSMap version 29 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8866286740 10 client.25237 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8866286740 20 client.25237 populate_metadata read hostname 'smithi114' 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8866286740 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x55643a9192f0 0x55643a918000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8866286740 4 client.25237 mds_command: new command op to 24538 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.725+0000 7f8866286740 1 -- 172.21.15.114:0/1492422875 --> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] -- command(tid 0: {"prefix": "status"}) -- 0x55643a7fbb30 con 0x55643a9192f0 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8861800640 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x55643a9192f0 0x55643a918000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8861800640 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x55643a9192f0 0x55643a918000 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=677e320dfe195473 server_cookie=b7b4794d74408287 in_seq=0 out_seq=0 2024-09-19T06:51:49.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8846a00640 10 client.25237 ms_handle_connect on v2:172.21.15.118:6838/345383252 2024-09-19T06:51:49.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8846a00640 1 -- 172.21.15.114:0/1492422875 <== mds.0 v2:172.21.15.118:6838/345383252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55643a7fbb30 con 0x55643a9192f0 2024-09-19T06:51:49.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8846a00640 10 client.25237 handle_command_reply: tid=0 2024-09-19T06:51:49.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 client.25237 shutdown 2024-09-19T06:51:49.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 -- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 msgr2=0x55643a8b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a8b4390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8848004e80 tx=0x7f884801c6a0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 -- 172.21.15.114:0/1492422875 shutdown_connections 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:6838/345383252,v1:172.21.15.118:6839/345383252] conn(0x55643a9192f0 0x55643a918000 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55643a899510 0x55643a892eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55643a88ec50 0x55643a8b4390 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 --2- 172.21.15.114:0/1492422875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55643a88e330 0x55643a8b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8846a00640 0 client.25237 ms_handle_reset on v2:172.21.15.118:6838/345383252 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.726+0000 7f8866286740 1 -- 172.21.15.114:0/1492422875 >> 172.21.15.114:0/1492422875 conn(0x55643a88a8c0 msgr2=0x55643a894c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.727+0000 7f8866286740 1 -- 172.21.15.114:0/1492422875 shutdown_connections 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.727+0000 7f8866286740 1 -- 172.21.15.114:0/1492422875 wait complete. 2024-09-19T06:51:49.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.727+0000 7f8866286740 20 client.25237 trim_cache size 0 max 16384 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "fface640-7651-11ef-bceb-c7b262605968", 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24538, 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 157.361730013, 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 29, 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 95, 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 93, 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 181.817472689 2024-09-19T06:51:49.734 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-09-19T06:51:49.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.727+0000 7f8863800640 1 -- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88400a32c0 msgr2=0x7f88400a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.727+0000 7f8863800640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88400a32c0 0x7f88400a5780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f884c0a4540 tx=0x7f884c0cd1b0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.727+0000 7f8863800640 1 -- 172.21.15.114:0/968390205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f885c074620 msgr2=0x7f885c19a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:49.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.727+0000 7f8863800640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f885c074620 0x7f885c19a630 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f884c00e9f0 tx=0x7f884c00eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.728+0000 7f8863800640 1 -- 172.21.15.114:0/968390205 shutdown_connections 2024-09-19T06:51:49.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.728+0000 7f8863800640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f88400a32c0 0x7f88400a5780 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.728+0000 7f8863800640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f885c10e3c0 0x7f885c19ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.728+0000 7f8863800640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f885c074f40 0x7f885c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.728+0000 7f8863800640 1 --2- 172.21.15.114:0/968390205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f885c074620 0x7f885c19a630 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:49.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.728+0000 7f8863800640 1 -- 172.21.15.114:0/968390205 >> 172.21.15.114:0/968390205 conn(0x7f885c0fd8c0 msgr2=0x7f885c0fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:49.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.728+0000 7f8863800640 1 -- 172.21.15.114:0/968390205 shutdown_connections 2024-09-19T06:51:49.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:49.728+0000 7f8863800640 1 -- 172.21.15.114:0/968390205 wait complete. 2024-09-19T06:51:50.521 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:50 smithi114 ceph-mon[30820]: pgmap v384: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:50.555 INFO:teuthology.run_tasks:Running task sequential... 2024-09-19T06:51:50.566 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-19T06:51:50.566 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-19T06:51:50.567 INFO:teuthology.task.print:Enabling mgr modules 2024-09-19T06:51:50.567 INFO:teuthology.run_tasks:Running task kclient... 2024-09-19T06:51:50.601 INFO:tasks.kclient:Mounting kernel clients... 2024-09-19T06:51:50.601 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-19T06:51:50.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:50.602 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-09-19T06:51:50.639 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-19T06:51:50.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:50.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-19T06:51:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:50 smithi116 ceph-mon[35411]: pgmap v384: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:50.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:50.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-09-19T06:51:50.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:50 smithi118 ceph-mon[35527]: pgmap v384: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:51:51.113 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-09-19T06:51:51.113 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-19T06:51:51.114 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-19T06:51:51.114 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f599a46ad10e37c929e5c43da2f6907ddcd1961 shell --fsid fface640-7651-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-19T06:51:51.496 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/fface640-7651-11ef-bceb-c7b262605968/mon.a/config 2024-09-19T06:51:52.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 -- 172.21.15.114:0/1306685073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63ac1019c0 msgr2=0x7f63ac101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:52.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 --2- 172.21.15.114:0/1306685073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63ac1019c0 0x7f63ac101e20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6394009930 tx=0x7f639402a420 comp rx=0 tx=0).stop 2024-09-19T06:51:52.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 -- 172.21.15.114:0/1306685073 shutdown_connections 2024-09-19T06:51:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 --2- 172.21.15.114:0/1306685073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac102360 0x7f63ac10e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 --2- 172.21.15.114:0/1306685073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63ac1019c0 0x7f63ac101e20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 --2- 172.21.15.114:0/1306685073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ac1079c0 0x7f63ac107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 -- 172.21.15.114:0/1306685073 >> 172.21.15.114:0/1306685073 conn(0x7f63ac0fd740 msgr2=0x7f63ac0ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 -- 172.21.15.114:0/1306685073 shutdown_connections 2024-09-19T06:51:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.083+0000 7f63b1600640 1 -- 172.21.15.114:0/1306685073 wait complete. 2024-09-19T06:51:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.084+0000 7f63b1600640 1 Processor -- start 2024-09-19T06:51:52.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.084+0000 7f63b1600640 1 -- start start 2024-09-19T06:51:52.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.084+0000 7f63b1600640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ac1019c0 0x7f63ac0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.084+0000 7f63b1600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac102360 0x7f63ac074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.084+0000 7f63b1600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63ac1079c0 0x7f63ac074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.084+0000 7f63b1600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111dd0 con 0x7f63ac1079c0 2024-09-19T06:51:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63b1600640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111c50 con 0x7f63ac1019c0 2024-09-19T06:51:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63b1600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111f50 con 0x7f63ac102360 2024-09-19T06:51:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ac1019c0 0x7f63ac0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63abe00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ac1019c0 0x7f63ac0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.114:54168/0 (socket says 172.21.15.114:54168) 2024-09-19T06:51:52.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63abe00640 1 -- 172.21.15.114:0/3942780396 learned_addr learned my addr 172.21.15.114:0/3942780396 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63b0c00640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63ac1079c0 0x7f63ac074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63a3e00640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac102360 0x7f63ac074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63abe00640 1 -- 172.21.15.114:0/3942780396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac102360 msgr2=0x7f63ac074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63abe00640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac102360 0x7f63ac074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63abe00640 1 -- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63ac1079c0 msgr2=0x7f63ac074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63abe00640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63ac1079c0 0x7f63ac074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.085+0000 7f63abe00640 1 -- 172.21.15.114:0/3942780396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac1a6790 con 0x7f63ac1019c0 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.086+0000 7f63abe00640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ac1019c0 0x7f63ac0744f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f639c00d950 tx=0x7f639c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:52.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.086+0000 7f63a9600640 1 -- 172.21.15.114:0/3942780396 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c017030 con 0x7f63ac1019c0 2024-09-19T06:51:52.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.086+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ac1a6a80 con 0x7f63ac1019c0 2024-09-19T06:51:52.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.086+0000 7f63a9600640 1 -- 172.21.15.114:0/3942780396 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f639c004020 con 0x7f63ac1019c0 2024-09-19T06:51:52.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.086+0000 7f63a9600640 1 -- 172.21.15.114:0/3942780396 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c0047a0 con 0x7f63ac1019c0 2024-09-19T06:51:52.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.086+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ac1a6fe0 con 0x7f63ac1019c0 2024-09-19T06:51:52.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.087+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63ac101e20 con 0x7f63ac1019c0 2024-09-19T06:51:52.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.089+0000 7f63a9600640 1 -- 172.21.15.114:0/3942780396 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f639c0041d0 con 0x7f63ac1019c0 2024-09-19T06:51:52.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.090+0000 7f63a9600640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f63900a32c0 0x7f63900a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:52.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.091+0000 7f63a9600640 1 -- 172.21.15.114:0/3942780396 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f639c0cf5b0 con 0x7f63ac1019c0 2024-09-19T06:51:52.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.091+0000 7f63a3e00640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f63900a32c0 0x7f63900a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:52.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.091+0000 7f63a3e00640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f63900a32c0 0x7f63900a5780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6394006010 tx=0x7f6394000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:52.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.095+0000 7f63a9600640 1 -- 172.21.15.114:0/3942780396 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f639c09be80 con 0x7f63ac1019c0 2024-09-19T06:51:52.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.314+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f63ac109f80 con 0x7f63ac1019c0 2024-09-19T06:51:52.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.314+0000 7f63a9600640 1 -- 172.21.15.114:0/3942780396 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v29) ==== 53+0+88 (secure 0 0 0) 0x7f639c09b780 con 0x7f63ac1019c0 2024-09-19T06:51:52.316 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-19T06:51:52.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f63900a32c0 msgr2=0x7f63900a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:52.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f63900a32c0 0x7f63900a5780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6394006010 tx=0x7f6394000bc0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ac1019c0 msgr2=0x7f63ac0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ac1019c0 0x7f63ac0744f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f639c00d950 tx=0x7f639c00de10 comp rx=0 tx=0).stop 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 shutdown_connections 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f63900a32c0 0x7f63900a5780 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63ac1079c0 0x7f63ac074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f63ac102360 0x7f63ac074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 --2- 172.21.15.114:0/3942780396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ac1019c0 0x7f63ac0744f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 >> 172.21.15.114:0/3942780396 conn(0x7f63ac0fd740 msgr2=0x7f63ac0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 shutdown_connections 2024-09-19T06:51:52.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:52.317+0000 7f63b1600640 1 -- 172.21.15.114:0/3942780396 wait complete. 2024-09-19T06:51:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:52 smithi114 ceph-mon[30820]: pgmap v385: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:52 smithi116 ceph-mon[35411]: pgmap v385: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:52.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:52 smithi118 ceph-mon[35527]: pgmap v385: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-19T06:51:53.190 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-19T06:51:53.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:53.190 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-09-19T06:51:53.227 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-19T06:51:53.227 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-19T06:51:53.227 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-19T06:51:53.228 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-09-19T06:51:53.229 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-19T06:51:53.229 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 42089sec preferred_lft 42089sec 2024-09-19T06:51:53.229 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::ec4:7aff:fe8f:cb63/64 scope link noprefixroute 2024-09-19T06:51:53.229 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-09-19T06:51:53.229 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-19T06:51:53.229 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:51:53.229 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:51:53.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-09-19T06:51:53.229 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-09-19T06:51:53.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-09-19T06:51:53.230 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-19T06:51:53.230 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:51:53.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:51:53.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:53 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3942780396' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T06:51:53.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:53.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:53.454 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-19T06:51:53.509 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-09-19T06:51:53.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:53.512 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-09-19T06:51:53.548 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-09-19T06:51:53.548 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-19T06:51:53.549 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-19T06:51:53.549 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-19T06:51:53.549 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-19T06:51:53.549 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:51:53.549 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:51:53.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-19T06:51:53.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-19T06:51:53.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-19T06:51:53.549 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:51:53.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:51:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:53 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3942780396' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T06:51:53.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:53.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:53.747 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-09-19T06:51:53.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:53 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3942780396' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-19T06:51:53.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:53.783 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-09-19T06:51:53.859 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:51:53.860 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:51:53.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-19T06:51:53.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-19T06:51:53.860 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:51:53.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:51:54.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:54.036 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-19T06:51:54.036 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:51:54.036 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:51:54.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-19T06:51:54.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-19T06:51:54.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-19T06:51:54.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-19T06:51:54.037 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-19T06:51:54.037 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:51:54.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:51:54.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:54.257 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-19T06:51:54.257 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-09-19T06:51:54.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-09-19T06:51:54.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-09-19T06:51:54.257 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-09-19T06:51:54.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:51:54.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:54.424 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T06:51:54.424 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-19T06:51:54.459 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-19T06:51:54.460 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-19T06:51:54.460 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-19T06:51:54.535 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-19T06:51:54.535 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-19T06:51:54.535 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/b40f575c-53c4-4511-9cf7-9b6a2aef6040 2024-09-19T06:51:54.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:54.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/b40f575c-53c4-4511-9cf7-9b6a2aef6040 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-09-19T06:51:54.622 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:54 smithi114 ceph-mon[30820]: pgmap v386: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:54.663 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-09-19T06:51:54.663 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-09-19T06:51:54.664 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-09-19T06:51:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:54 smithi116 ceph-mon[35411]: pgmap v386: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:54.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:54 smithi118 ceph-mon[35527]: pgmap v386: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:51:55.284 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-09-19T06:51:55.284 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-09-19T06:51:55.284 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-09-19T06:51:55.284 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:6789,172.21.15.116:6789,172.21.15.118:6789" 2024-09-19T06:51:55.284 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.114:6789,172.21.15.116:6789,172.21.15.118:6789:/volumes/qa/sv_1/b40f575c-53c4-4511-9cf7-9b6a2aef6040 2024-09-19T06:51:55.284 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=fface640-7651-11ef-bceb-c7b262605968" will pass to kernel 2024-09-19T06:51:55.286 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-19T06:51:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-09-19T06:51:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-19T06:51:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-09-19T06:51:55.286 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-19T06:51:55.289 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-19T06:51:55.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-19T06:51:55.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-19T06:51:55.349 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-19T06:51:55.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-19T06:51:55.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:55.442 INFO:teuthology.orchestra.run.smithi114.stdout:client25246 2024-09-19T06:51:55.442 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:51:55.442 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-19T06:51:55.479 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-19T06:51:55.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-19T06:51:55.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:55.608 INFO:teuthology.orchestra.run.smithi114.stdout:client25246 2024-09-19T06:51:55.609 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-19T06:51:55.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-19T06:51:55.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:55.698 INFO:teuthology.orchestra.run.smithi114.stdout:client25246 2024-09-19T06:51:55.699 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/fface640-7651-11ef-bceb-c7b262605968.client25246/status') 2024-09-19T06:51:55.776 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-19T06:51:55.798 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.25246 (3)172.21.15.114:0/1565499477 2024-09-19T06:51:55.798 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-09-19T06:51:55.798 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-09-19T06:51:55.799 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-09-19T06:51:55.799 INFO:teuthology.orchestra.run.smithi114.stderr:72 bytes copied, 7.1759e-05 s, 1.0 MB/s 2024-09-19T06:51:55.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-19T06:51:55.810 INFO:teuthology.run_tasks:Running task exec... 2024-09-19T06:51:55.820 INFO:teuthology.task.exec:Executing custom commands... 2024-09-19T06:51:55.820 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-09-19T06:51:55.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-09-19T06:51:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 -- 172.21.15.114:0/1095598323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 msgr2=0x7f3808103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 --2- 172.21.15.114:0/1095598323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 0x7f3808103450 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f37fc009900 tx=0x7f37fc02a3f0 comp rx=0 tx=0).stop 2024-09-19T06:51:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 -- 172.21.15.114:0/1095598323 shutdown_connections 2024-09-19T06:51:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 --2- 172.21.15.114:0/1095598323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3808101f40 0x7f3808109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 --2- 172.21.15.114:0/1095598323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3808101580 0x7f3808101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:55.975 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 --2- 172.21.15.114:0/1095598323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 0x7f3808103450 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 -- 172.21.15.114:0/1095598323 >> 172.21.15.114:0/1095598323 conn(0x7f38080fd0d0 msgr2=0x7f38080ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 -- 172.21.15.114:0/1095598323 shutdown_connections 2024-09-19T06:51:55.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.973+0000 7f380ec00640 1 -- 172.21.15.114:0/1095598323 wait complete. 2024-09-19T06:51:55.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.974+0000 7f380ec00640 1 Processor -- start 2024-09-19T06:51:55.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.974+0000 7f380ec00640 1 -- start start 2024-09-19T06:51:55.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3808101580 0x7f3808199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:55.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ec00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3808101f40 0x7f380819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:55.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 0x7f38081a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f380810d740 con 0x7f3808103050 2024-09-19T06:51:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ec00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f380810d5c0 con 0x7f3808101f40 2024-09-19T06:51:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ec00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 -- 0x7f380810d8c0 con 0x7f3808101580 2024-09-19T06:51:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 0x7f38081a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3808101580 0x7f3808199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3808101f40 0x7f380819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 0x7f38081a1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:33136/0 (socket says 172.21.15.114:33136) 2024-09-19T06:51:55.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380e200640 1 -- 172.21.15.114:0/3248441075 learned_addr learned my addr 172.21.15.114:0/3248441075 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ce00640 1 -- 172.21.15.114:0/3248441075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3808101580 msgr2=0x7f3808199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ce00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3808101580 0x7f3808199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ce00640 1 -- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 msgr2=0x7f38081a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ce00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 0x7f38081a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380ce00640 1 -- 172.21.15.114:0/3248441075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38081a1be0 con 0x7f3808101f40 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.975+0000 7f380e200640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 0x7f38081a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.976+0000 7f380ce00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3808101f40 0x7f380819a500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f37f400e9f0 tx=0x7f37f400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.976+0000 7f3806000640 1 -- 172.21.15.114:0/3248441075 <== mon.2 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f400cd80 con 0x7f3808101f40 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.976+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38081a1ed0 con 0x7f3808101f40 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.976+0000 7f3806000640 1 -- 172.21.15.114:0/3248441075 <== mon.2 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f40035f0 con 0x7f3808101f40 2024-09-19T06:51:55.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.976+0000 7f3806000640 1 -- 172.21.15.114:0/3248441075 <== mon.2 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4010670 con 0x7f3808101f40 2024-09-19T06:51:55.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.976+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38081a2390 con 0x7f3808101f40 2024-09-19T06:51:55.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.977+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3808101a00 con 0x7f3808101f40 2024-09-19T06:51:55.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.980+0000 7f3806000640 1 -- 172.21.15.114:0/3248441075 <== mon.2 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 138356+0+0 (secure 0 0 0) 0x7f37f40037a0 con 0x7f3808101f40 2024-09-19T06:51:55.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.981+0000 7f3806000640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37f00a3240 0x7f37f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-19T06:51:55.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.981+0000 7f3806000640 1 -- 172.21.15.114:0/3248441075 <== mon.2 v2:172.21.15.116:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9237+0+0 (secure 0 0 0) 0x7f37f40cfff0 con 0x7f3808101f40 2024-09-19T06:51:55.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.981+0000 7f380d800640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37f00a3240 0x7f37f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-19T06:51:55.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.982+0000 7f380d800640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37f00a3240 0x7f37f00a5700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f37fc02a9e0 tx=0x7f37fc0315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-19T06:51:55.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:55.982+0000 7f3806000640 1 -- 172.21.15.114:0/3248441075 <== mon.2 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188913 (secure 0 0 0) 0x7f37f409c990 con 0x7f3808101f40 2024-09-19T06:51:56.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.196+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f38081a2680 con 0x7f3808101f40 2024-09-19T06:51:56.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.286+0000 7f3806000640 1 -- 172.21.15.114:0/3248441075 <== mon.2 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v30) ==== 117+0+0 (secure 0 0 0) 0x7f37f409c290 con 0x7f3808101f40 2024-09-19T06:51:56.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37f00a3240 msgr2=0x7f37f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:56.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37f00a3240 0x7f37f00a5700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f37fc02a9e0 tx=0x7f37fc0315b0 comp rx=0 tx=0).stop 2024-09-19T06:51:56.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3808101f40 msgr2=0x7f380819a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-19T06:51:56.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3808101f40 0x7f380819a500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f37f400e9f0 tx=0x7f37f400eeb0 comp rx=0 tx=0).stop 2024-09-19T06:51:56.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 shutdown_connections 2024-09-19T06:51:56.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:6800/3586930729,v1:172.21.15.114:6801/3586930729] conn(0x7f37f00a3240 0x7f37f00a5700 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:56.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3808103050 0x7f38081a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3808101f40 0x7f380819a500 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 --2- 172.21.15.114:0/3248441075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3808101580 0x7f3808199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-19T06:51:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 >> 172.21.15.114:0/3248441075 conn(0x7f38080fd0d0 msgr2=0x7f3808074420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-19T06:51:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 shutdown_connections 2024-09-19T06:51:56.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-09-19T06:51:56.289+0000 7f380ec00640 1 -- 172.21.15.114:0/3248441075 wait complete. 2024-09-19T06:51:56.307 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-19T06:51:56.319 INFO:teuthology.run_tasks:Running task workunit... 2024-09-19T06:51:56.367 INFO:tasks.workunit:Pulling workunits from ref 6fad5dd73c93545e52f57f681e140a9c3529c11e 2024-09-19T06:51:56.367 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-19T06:51:56.367 INFO:tasks.workunit:timeout=3h 2024-09-19T06:51:56.367 INFO:tasks.workunit:cleanup=True 2024-09-19T06:51:56.368 DEBUG:teuthology.orchestra.run.smithi114:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-19T06:51:56.403 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-19T06:51:56.404 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-19T06:51:56.404 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 78h/120d Inode: 1099511628283 Links: 2 2024-09-19T06:51:56.404 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-19T06:51:56.404 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-19T06:51:56.404 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-09-19 06:50:08.616333779 +0000 2024-09-19T06:51:56.404 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-09-19 06:50:08.616333779 +0000 2024-09-19T06:51:56.404 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-09-19 06:51:55.342125858 +0000 2024-09-19T06:51:56.404 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-09-19 06:50:08.616333779 +0000 2024-09-19T06:51:56.404 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-19T06:51:56.405 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-19T06:51:56.508 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6fad5dd73c93545e52f57f681e140a9c3529c11e 2024-09-19T06:51:56.546 INFO:tasks.workunit.client.0.smithi114.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-19T06:51:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:56 smithi114 ceph-mon[30820]: pgmap v387: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:56 smithi114 ceph-mon[30820]: from='client.? 172.21.15.114:0/3248441075' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-19T06:51:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:56 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-19T06:51:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:56 smithi116 ceph-mon[35411]: pgmap v387: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:56 smithi116 ceph-mon[35411]: from='client.? 172.21.15.114:0/3248441075' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-19T06:51:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:56 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-19T06:51:56.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:56 smithi118 ceph-mon[35527]: pgmap v387: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:51:56.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:56 smithi118 ceph-mon[35527]: from='client.? 172.21.15.114:0/3248441075' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-19T06:51:56.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:56 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-19T06:51:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:57 smithi114 ceph-mon[30820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-19T06:51:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:57 smithi114 ceph-mon[30820]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:51:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:57 smithi116 ceph-mon[35411]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-19T06:51:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:57 smithi116 ceph-mon[35411]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:51:57.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:57 smithi118 ceph-mon[35527]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-19T06:51:57.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:57 smithi118 ceph-mon[35527]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:51:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:51:58 smithi114 ceph-mon[30820]: pgmap v388: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-19T06:51:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:51:58 smithi116 ceph-mon[35411]: pgmap v388: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-19T06:51:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:51:58 smithi118 ceph-mon[35527]: pgmap v388: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-19T06:52:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:00 smithi114 ceph-mon[30820]: pgmap v389: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T06:52:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:00 smithi116 ceph-mon[35411]: pgmap v389: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T06:52:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:00 smithi118 ceph-mon[35527]: pgmap v389: 129 pgs: 129 active+clean; 645 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-19T06:52:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:02 smithi114 ceph-mon[30820]: pgmap v390: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-19T06:52:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:02 smithi116 ceph-mon[35411]: pgmap v390: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-19T06:52:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:02 smithi118 ceph-mon[35527]: pgmap v390: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-19T06:52:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:04 smithi114 ceph-mon[30820]: pgmap v391: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-09-19T06:52:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:04 smithi116 ceph-mon[35411]: pgmap v391: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-09-19T06:52:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:04 smithi118 ceph-mon[35527]: pgmap v391: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-09-19T06:52:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:06 smithi114 ceph-mon[30820]: pgmap v392: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-19T06:52:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:06 smithi116 ceph-mon[35411]: pgmap v392: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-19T06:52:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:06 smithi118 ceph-mon[35527]: pgmap v392: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-19T06:52:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:08 smithi114 ceph-mon[30820]: pgmap v393: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-19T06:52:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:08 smithi116 ceph-mon[35411]: pgmap v393: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-19T06:52:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:08 smithi118 ceph-mon[35527]: pgmap v393: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-19T06:52:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:10 smithi114 ceph-mon[30820]: pgmap v394: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-09-19T06:52:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:10 smithi116 ceph-mon[35411]: pgmap v394: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-09-19T06:52:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:10 smithi118 ceph-mon[35527]: pgmap v394: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-09-19T06:52:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:12 smithi114 ceph-mon[30820]: pgmap v395: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-19T06:52:13.193 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:12 smithi118 ceph-mon[35527]: pgmap v395: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-19T06:52:13.193 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:12 smithi116 ceph-mon[35411]: pgmap v395: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-19T06:52:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:14 smithi114 ceph-mon[30820]: pgmap v396: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:14 smithi116 ceph-mon[35411]: pgmap v396: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:14 smithi118 ceph-mon[35527]: pgmap v396: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:16 smithi114 ceph-mon[30820]: pgmap v397: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:52:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:16 smithi116 ceph-mon[35411]: pgmap v397: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:52:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:16 smithi118 ceph-mon[35527]: pgmap v397: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:52:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:16 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:52:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:52:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:52:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:52:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:17 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:52:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:17 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:52:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:17 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:52:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:18 smithi114 ceph-mon[30820]: pgmap v398: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:18 smithi116 ceph-mon[35411]: pgmap v398: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:18 smithi118 ceph-mon[35527]: pgmap v398: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:20 smithi114 ceph-mon[30820]: pgmap v399: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:20 smithi116 ceph-mon[35411]: pgmap v399: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:20 smithi118 ceph-mon[35527]: pgmap v399: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:21 smithi114 ceph-mon[30820]: pgmap v400: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:52:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:21 smithi116 ceph-mon[35411]: pgmap v400: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:52:21.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:21 smithi118 ceph-mon[35527]: pgmap v400: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:52:24.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:24 smithi118 ceph-mon[35527]: pgmap v401: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:24 smithi114 ceph-mon[30820]: pgmap v401: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:24 smithi116 ceph-mon[35411]: pgmap v401: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:26.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:26 smithi118 ceph-mon[35527]: pgmap v402: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:26 smithi114 ceph-mon[30820]: pgmap v402: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:26 smithi116 ceph-mon[35411]: pgmap v402: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:28.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:28 smithi118 ceph-mon[35527]: pgmap v403: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:28 smithi114 ceph-mon[30820]: pgmap v403: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:28 smithi116 ceph-mon[35411]: pgmap v403: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:30.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:30 smithi118 ceph-mon[35527]: pgmap v404: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:30 smithi114 ceph-mon[30820]: pgmap v404: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:30 smithi116 ceph-mon[35411]: pgmap v404: 129 pgs: 129 active+clean; 656 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:32 smithi118 ceph-mon[35527]: pgmap v405: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:52:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:32 smithi114 ceph-mon[30820]: pgmap v405: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:52:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:32 smithi116 ceph-mon[35411]: pgmap v405: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:52:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:34 smithi118 ceph-mon[35527]: pgmap v406: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-09-19T06:52:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:34 smithi114 ceph-mon[30820]: pgmap v406: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-09-19T06:52:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:34 smithi116 ceph-mon[35411]: pgmap v406: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-09-19T06:52:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:36 smithi118 ceph-mon[35527]: pgmap v407: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-19T06:52:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:36 smithi114 ceph-mon[30820]: pgmap v407: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-19T06:52:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:36 smithi116 ceph-mon[35411]: pgmap v407: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-19T06:52:38.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:38 smithi116 ceph-mon[35411]: pgmap v408: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-19T06:52:38.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:38 smithi118 ceph-mon[35527]: pgmap v408: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-19T06:52:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:38 smithi114 ceph-mon[30820]: pgmap v408: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 10 op/s 2024-09-19T06:52:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:40 smithi114 ceph-mon[30820]: pgmap v409: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-19T06:52:40.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:40 smithi118 ceph-mon[35527]: pgmap v409: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-19T06:52:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:40 smithi116 ceph-mon[35411]: pgmap v409: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-09-19T06:52:42.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:42 smithi118 ceph-mon[35527]: pgmap v410: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:52:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:42 smithi114 ceph-mon[30820]: pgmap v410: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:52:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:42 smithi116 ceph-mon[35411]: pgmap v410: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-09-19T06:52:44.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:44 smithi118 ceph-mon[35527]: pgmap v411: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:44 smithi114 ceph-mon[30820]: pgmap v411: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:44 smithi116 ceph-mon[35411]: pgmap v411: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:46.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:46 smithi118 ceph-mon[35527]: pgmap v412: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:46 smithi114 ceph-mon[30820]: pgmap v412: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:46 smithi116 ceph-mon[35411]: pgmap v412: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:48.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:48 smithi118 ceph-mon[35527]: pgmap v413: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:48 smithi114 ceph-mon[30820]: pgmap v413: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:48 smithi116 ceph-mon[35411]: pgmap v413: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:50.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:50 smithi118 ceph-mon[35527]: pgmap v414: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:50 smithi114 ceph-mon[30820]: pgmap v414: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:50 smithi116 ceph-mon[35411]: pgmap v414: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:52:52.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:52 smithi118 ceph-mon[35527]: pgmap v415: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:52:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:52 smithi114 ceph-mon[30820]: pgmap v415: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:52:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:52 smithi116 ceph-mon[35411]: pgmap v415: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:52:54.508 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:54 smithi118 ceph-mon[35527]: pgmap v416: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:54 smithi114 ceph-mon[30820]: pgmap v416: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:54 smithi116 ceph-mon[35411]: pgmap v416: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:52:56.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:56 smithi118 ceph-mon[35527]: pgmap v417: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:56 smithi114 ceph-mon[30820]: pgmap v417: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:56 smithi116 ceph-mon[35411]: pgmap v417: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:52:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:52:58 smithi114 ceph-mon[30820]: pgmap v418: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:52:58 smithi116 ceph-mon[35411]: pgmap v418: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:52:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:52:58 smithi118 ceph-mon[35527]: pgmap v418: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-09-19T06:53:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:00 smithi114 ceph-mon[30820]: pgmap v419: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:53:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:00 smithi116 ceph-mon[35411]: pgmap v419: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:53:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:00 smithi118 ceph-mon[35527]: pgmap v419: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-19T06:53:01.939 INFO:tasks.workunit.client.0.smithi114.stderr:Updating files: 66% (8531/12896) Updating files: 67% (8641/12896) Updating files: 68% (8770/12896) Updating files: 69% (8899/12896) Updating files: 70% (9028/12896) Updating files: 71% (9157/12896) Updating files: 72% (9286/12896) Updating files: 73% (9415/12896) Updating files: 74% (9544/12896) Updating files: 75% (9672/12896) Updating files: 76% (9801/12896) Updating files: 77% (9930/12896) Updating files: 78% (10059/12896) Updating files: 79% (10188/12896) Updating files: 80% (10317/12896) Updating files: 81% (10446/12896) Updating files: 82% (10575/12896) Updating files: 83% (10704/12896) Updating files: 84% (10833/12896) Updating files: 85% (10962/12896) Updating files: 86% (11091/12896) Updating files: 87% (11220/12896) Updating files: 88% (11349/12896) Updating files: 89% (11478/12896) Updating files: 90% (11607/12896) Updating files: 91% (11736/12896) Updating files: 92% (11865/12896) Updating files: 93% (11994/12896) Updating files: 94% (12123/12896) Updating files: 95% (12252/12896) Updating files: 96% (12381/12896) Updating files: 97% (12510/12896) Updating files: 98% (12639/12896) Updating files: 99% (12768/12896) Updating files: 100% (12896/12896) Updating files: 100% (12896/12896), done. 2024-09-19T06:53:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:02 smithi114 ceph-mon[30820]: pgmap v420: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:53:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:02 smithi116 ceph-mon[35411]: pgmap v420: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:53:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:02 smithi118 ceph-mon[35527]: pgmap v420: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-09-19T06:53:03.027 INFO:tasks.workunit.client.0.smithi114.stderr:Note: switching to '6fad5dd73c93545e52f57f681e140a9c3529c11e'. 2024-09-19T06:53:03.027 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:03.027 INFO:tasks.workunit.client.0.smithi114.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-19T06:53:03.027 INFO:tasks.workunit.client.0.smithi114.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-19T06:53:03.027 INFO:tasks.workunit.client.0.smithi114.stderr:state without impacting any branches by switching back to a branch. 2024-09-19T06:53:03.027 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:03.027 INFO:tasks.workunit.client.0.smithi114.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr: git switch -c 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr:Or undo this operation with: 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr: git switch - 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:03.028 INFO:tasks.workunit.client.0.smithi114.stderr:HEAD is now at 6fad5dd73c9 Merge pull request #59812 from neha-ojha/wip-rtd-squid 2024-09-19T06:53:03.042 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-19T06:53:03.130 INFO:tasks.workunit.client.0.smithi114.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-19T06:53:03.135 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-19T06:53:03.135 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-19T06:53:03.853 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-19T06:53:03.960 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-19T06:53:04.026 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-19T06:53:04.029 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-19T06:53:04.029 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-19T06:53:04.093 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-19T06:53:04.100 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-09-19T06:53:04.101 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-19T06:53:04.136 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-09-19T06:53:04.137 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-09-19T06:53:04.138 DEBUG:teuthology.orchestra.run.smithi114:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fad5dd73c93545e52f57f681e140a9c3529c11e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-09-19T06:53:04.227 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'getting iogen' 2024-09-19T06:53:04.228 INFO:tasks.workunit.client.0.smithi114.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-09-19T06:53:04.228 INFO:tasks.workunit.client.0.smithi114.stdout:getting iogen 2024-09-19T06:53:04.255 INFO:tasks.workunit.client.0.smithi114.stderr:--2024-09-19 06:53:04-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-09-19T06:53:04.256 INFO:tasks.workunit.client.0.smithi114.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-09-19T06:53:04.323 INFO:tasks.workunit.client.0.smithi114.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-09-19T06:53:04.431 INFO:tasks.workunit.client.0.smithi114.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-19T06:53:04.432 INFO:tasks.workunit.client.0.smithi114.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-09-19T06:53:04.433 INFO:tasks.workunit.client.0.smithi114.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-09-19T06:53:04.437 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:04.437 INFO:tasks.workunit.client.0.smithi114.stderr: 0K ....... 100% 661M=0s 2024-09-19T06:53:04.438 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:04.438 INFO:tasks.workunit.client.0.smithi114.stderr:2024-09-19 06:53:04 (661 MB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-09-19T06:53:04.438 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-09-19T06:53:04.438 INFO:tasks.workunit.client.0.smithi114.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-09-19T06:53:04.454 INFO:tasks.workunit.client.0.smithi114.stdout:iogen_3.1p0 2024-09-19T06:53:04.455 INFO:tasks.workunit.client.0.smithi114.stdout:iogen_3.1p0/src 2024-09-19T06:53:04.456 INFO:tasks.workunit.client.0.smithi114.stdout:iogen_3.1p0/src/iogen.8 2024-09-19T06:53:04.458 INFO:tasks.workunit.client.0.smithi114.stdout:iogen_3.1p0/src/iogen.c 2024-09-19T06:53:04.459 INFO:tasks.workunit.client.0.smithi114.stdout:iogen_3.1p0/README 2024-09-19T06:53:04.460 INFO:tasks.workunit.client.0.smithi114.stdout:iogen_3.1p0/Makefile.bsd 2024-09-19T06:53:04.461 INFO:tasks.workunit.client.0.smithi114.stdout:iogen_3.1p0/Makefile 2024-09-19T06:53:04.463 INFO:tasks.workunit.client.0.smithi114.stderr:+ cd iogen_3.1p0 2024-09-19T06:53:04.463 INFO:tasks.workunit.client.0.smithi114.stdout:making iogen 2024-09-19T06:53:04.464 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'making iogen' 2024-09-19T06:53:04.464 INFO:tasks.workunit.client.0.smithi114.stderr:+ make 2024-09-19T06:53:04.473 INFO:tasks.workunit.client.0.smithi114.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-09-19T06:53:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:04 smithi114 ceph-mon[30820]: pgmap v421: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:53:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:04 smithi116 ceph-mon[35411]: pgmap v421: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:53:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:04 smithi118 ceph-mon[35527]: pgmap v421: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-19T06:53:04.779 INFO:tasks.workunit.client.0.smithi114.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-09-19T06:53:04.896 INFO:tasks.workunit.client.0.smithi114.stdout:running iogen 2024-09-19T06:53:04.897 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'running iogen' 2024-09-19T06:53:04.897 INFO:tasks.workunit.client.0.smithi114.stderr:+ ./iogen -n 5 -s 2g 2024-09-19T06:53:04.898 INFO:tasks.workunit.client.0.smithi114.stderr:child spawned 85483 2024-09-19T06:53:04.898 INFO:tasks.workunit.client.0.smithi114.stderr:child spawned 85484 2024-09-19T06:53:04.898 INFO:tasks.workunit.client.0.smithi114.stderr:child spawned 85485 2024-09-19T06:53:04.899 INFO:tasks.workunit.client.0.smithi114.stderr:child spawned 85486 2024-09-19T06:53:04.899 INFO:tasks.workunit.client.0.smithi114.stderr:child spawned 85487 2024-09-19T06:53:04.899 INFO:tasks.workunit.client.0.smithi114.stdout:sleep for 10 min 2024-09-19T06:53:04.900 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'sleep for 10 min' 2024-09-19T06:53:04.900 INFO:tasks.workunit.client.0.smithi114.stderr:+ sleep 600 2024-09-19T06:53:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:06 smithi116 ceph-mon[35411]: pgmap v422: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:53:06.697 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:06 smithi114 ceph-mon[30820]: pgmap v422: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:53:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:06 smithi118 ceph-mon[35527]: pgmap v422: 129 pgs: 129 active+clean; 661 KiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-09-19T06:53:08.558 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:08 smithi118 ceph-mon[35527]: pgmap v423: 129 pgs: 129 active+clean; 25 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 14 op/s 2024-09-19T06:53:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:08 smithi114 ceph-mon[30820]: pgmap v423: 129 pgs: 129 active+clean; 25 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 14 op/s 2024-09-19T06:53:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:08 smithi116 ceph-mon[35411]: pgmap v423: 129 pgs: 129 active+clean; 25 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 MiB/s wr, 14 op/s 2024-09-19T06:53:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:10 smithi114 ceph-mon[30820]: pgmap v424: 129 pgs: 129 active+clean; 269 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 22 MiB/s wr, 19 op/s 2024-09-19T06:53:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:10 smithi116 ceph-mon[35411]: pgmap v424: 129 pgs: 129 active+clean; 269 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 22 MiB/s wr, 19 op/s 2024-09-19T06:53:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:10 smithi118 ceph-mon[35527]: pgmap v424: 129 pgs: 129 active+clean; 269 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 22 MiB/s wr, 19 op/s 2024-09-19T06:53:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:12 smithi114 ceph-mon[30820]: pgmap v425: 129 pgs: 129 active+clean; 685 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 57 MiB/s wr, 34 op/s 2024-09-19T06:53:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:12 smithi116 ceph-mon[35411]: pgmap v425: 129 pgs: 129 active+clean; 685 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 57 MiB/s wr, 34 op/s 2024-09-19T06:53:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:12 smithi118 ceph-mon[35527]: pgmap v425: 129 pgs: 129 active+clean; 685 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 57 MiB/s wr, 34 op/s 2024-09-19T06:53:14.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:14 smithi114 ceph-mon[30820]: pgmap v426: 129 pgs: 129 active+clean; 873 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 73 MiB/s wr, 35 op/s 2024-09-19T06:53:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:14 smithi116 ceph-mon[35411]: pgmap v426: 129 pgs: 129 active+clean; 873 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 73 MiB/s wr, 35 op/s 2024-09-19T06:53:14.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:14 smithi118 ceph-mon[35527]: pgmap v426: 129 pgs: 129 active+clean; 873 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 73 MiB/s wr, 35 op/s 2024-09-19T06:53:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:16 smithi114 ceph-mon[30820]: pgmap v427: 129 pgs: 129 active+clean; 2.0 GiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 167 MiB/s wr, 60 op/s 2024-09-19T06:53:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:16 smithi116 ceph-mon[35411]: pgmap v427: 129 pgs: 129 active+clean; 2.0 GiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 167 MiB/s wr, 60 op/s 2024-09-19T06:53:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:16 smithi118 ceph-mon[35527]: pgmap v427: 129 pgs: 129 active+clean; 2.0 GiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 167 MiB/s wr, 60 op/s 2024-09-19T06:53:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:53:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:53:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:17 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:53:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:17 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:17.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:17 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:18.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:18 smithi114 ceph-mon[30820]: pgmap v428: 129 pgs: 129 active+clean; 2.3 GiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 198 MiB/s wr, 70 op/s 2024-09-19T06:53:18.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:53:18.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:53:18.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:18 smithi116 ceph-mon[35411]: pgmap v428: 129 pgs: 129 active+clean; 2.3 GiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 198 MiB/s wr, 70 op/s 2024-09-19T06:53:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:53:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:53:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:18 smithi118 ceph-mon[35527]: pgmap v428: 129 pgs: 129 active+clean; 2.3 GiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 198 MiB/s wr, 70 op/s 2024-09-19T06:53:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:53:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:53:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:53:20.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:20 smithi114 ceph-mon[30820]: pgmap v429: 129 pgs: 129 active+clean; 2.8 GiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 236 MiB/s wr, 74 op/s 2024-09-19T06:53:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:20 smithi116 ceph-mon[35411]: pgmap v429: 129 pgs: 129 active+clean; 2.8 GiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 236 MiB/s wr, 74 op/s 2024-09-19T06:53:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:20 smithi118 ceph-mon[35527]: pgmap v429: 129 pgs: 129 active+clean; 2.8 GiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 236 MiB/s wr, 74 op/s 2024-09-19T06:53:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:22 smithi114 ceph-mon[30820]: pgmap v430: 129 pgs: 129 active+clean; 3.5 GiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 273 MiB/s wr, 87 op/s 2024-09-19T06:53:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:22 smithi116 ceph-mon[35411]: pgmap v430: 129 pgs: 129 active+clean; 3.5 GiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 273 MiB/s wr, 87 op/s 2024-09-19T06:53:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:22 smithi118 ceph-mon[35527]: pgmap v430: 129 pgs: 129 active+clean; 3.5 GiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 273 MiB/s wr, 87 op/s 2024-09-19T06:53:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:24 smithi114 ceph-mon[30820]: pgmap v431: 129 pgs: 129 active+clean; 3.8 GiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 268 MiB/s wr, 78 op/s 2024-09-19T06:53:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:24 smithi116 ceph-mon[35411]: pgmap v431: 129 pgs: 129 active+clean; 3.8 GiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 268 MiB/s wr, 78 op/s 2024-09-19T06:53:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:24 smithi118 ceph-mon[35527]: pgmap v431: 129 pgs: 129 active+clean; 3.8 GiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 268 MiB/s wr, 78 op/s 2024-09-19T06:53:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:25 smithi114 ceph-mon[30820]: pgmap v432: 129 pgs: 129 active+clean; 4.9 GiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 344 MiB/s wr, 99 op/s 2024-09-19T06:53:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:25 smithi116 ceph-mon[35411]: pgmap v432: 129 pgs: 129 active+clean; 4.9 GiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 344 MiB/s wr, 99 op/s 2024-09-19T06:53:25.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:25 smithi118 ceph-mon[35527]: pgmap v432: 129 pgs: 129 active+clean; 4.9 GiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 344 MiB/s wr, 99 op/s 2024-09-19T06:53:28.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:28 smithi118 ceph-mon[35527]: pgmap v433: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 280 MiB/s wr, 84 op/s 2024-09-19T06:53:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:28 smithi114 ceph-mon[30820]: pgmap v433: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 280 MiB/s wr, 84 op/s 2024-09-19T06:53:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:28 smithi116 ceph-mon[35411]: pgmap v433: 129 pgs: 129 active+clean; 5.2 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 280 MiB/s wr, 84 op/s 2024-09-19T06:53:30.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:30 smithi118 ceph-mon[35527]: pgmap v434: 129 pgs: 129 active+clean; 5.9 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 303 MiB/s wr, 87 op/s 2024-09-19T06:53:30.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:30 smithi114 ceph-mon[30820]: pgmap v434: 129 pgs: 129 active+clean; 5.9 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 303 MiB/s wr, 87 op/s 2024-09-19T06:53:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:30 smithi116 ceph-mon[35411]: pgmap v434: 129 pgs: 129 active+clean; 5.9 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 303 MiB/s wr, 87 op/s 2024-09-19T06:53:32.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:32 smithi118 ceph-mon[35527]: pgmap v435: 129 pgs: 129 active+clean; 6.4 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 311 MiB/s wr, 92 op/s 2024-09-19T06:53:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:32 smithi114 ceph-mon[30820]: pgmap v435: 129 pgs: 129 active+clean; 6.4 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 311 MiB/s wr, 92 op/s 2024-09-19T06:53:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:32 smithi116 ceph-mon[35411]: pgmap v435: 129 pgs: 129 active+clean; 6.4 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 311 MiB/s wr, 92 op/s 2024-09-19T06:53:34.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:34 smithi118 ceph-mon[35527]: pgmap v436: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 273 MiB/s wr, 78 op/s 2024-09-19T06:53:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:34 smithi114 ceph-mon[30820]: pgmap v436: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 273 MiB/s wr, 78 op/s 2024-09-19T06:53:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:34 smithi116 ceph-mon[35411]: pgmap v436: 129 pgs: 129 active+clean; 6.7 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 273 MiB/s wr, 78 op/s 2024-09-19T06:53:36.506 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:36 smithi118 ceph-mon[35527]: pgmap v437: 129 pgs: 129 active+clean; 7.8 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 337 MiB/s wr, 98 op/s 2024-09-19T06:53:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:36 smithi114 ceph-mon[30820]: pgmap v437: 129 pgs: 129 active+clean; 7.8 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 337 MiB/s wr, 98 op/s 2024-09-19T06:53:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:36 smithi116 ceph-mon[35411]: pgmap v437: 129 pgs: 129 active+clean; 7.8 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 337 MiB/s wr, 98 op/s 2024-09-19T06:53:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:38 smithi114 ceph-mon[30820]: pgmap v438: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 275 MiB/s wr, 81 op/s 2024-09-19T06:53:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:38 smithi116 ceph-mon[35411]: pgmap v438: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 275 MiB/s wr, 81 op/s 2024-09-19T06:53:38.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:38 smithi118 ceph-mon[35527]: pgmap v438: 129 pgs: 129 active+clean; 8.1 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 275 MiB/s wr, 81 op/s 2024-09-19T06:53:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:40 smithi114 ceph-mon[30820]: pgmap v439: 129 pgs: 129 active+clean; 8.7 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 297 MiB/s wr, 84 op/s 2024-09-19T06:53:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:40 smithi116 ceph-mon[35411]: pgmap v439: 129 pgs: 129 active+clean; 8.7 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 297 MiB/s wr, 84 op/s 2024-09-19T06:53:40.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:40 smithi118 ceph-mon[35527]: pgmap v439: 129 pgs: 129 active+clean; 8.7 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 297 MiB/s wr, 84 op/s 2024-09-19T06:53:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:42 smithi114 ceph-mon[30820]: pgmap v440: 129 pgs: 129 active+clean; 9.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 291 MiB/s wr, 86 op/s 2024-09-19T06:53:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:42 smithi116 ceph-mon[35411]: pgmap v440: 129 pgs: 129 active+clean; 9.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 291 MiB/s wr, 86 op/s 2024-09-19T06:53:42.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:42 smithi118 ceph-mon[35527]: pgmap v440: 129 pgs: 129 active+clean; 9.3 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 291 MiB/s wr, 86 op/s 2024-09-19T06:53:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:44 smithi114 ceph-mon[30820]: pgmap v441: 129 pgs: 129 active+clean; 9.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 262 MiB/s wr, 88 op/s 2024-09-19T06:53:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:44 smithi116 ceph-mon[35411]: pgmap v441: 129 pgs: 129 active+clean; 9.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 262 MiB/s wr, 88 op/s 2024-09-19T06:53:44.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:44 smithi118 ceph-mon[35527]: pgmap v441: 129 pgs: 129 active+clean; 9.5 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 262 MiB/s wr, 88 op/s 2024-09-19T06:53:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:46 smithi114 ceph-mon[30820]: pgmap v442: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 324 MiB/s wr, 316 op/s 2024-09-19T06:53:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:46 smithi116 ceph-mon[35411]: pgmap v442: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 324 MiB/s wr, 316 op/s 2024-09-19T06:53:46.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:46 smithi118 ceph-mon[35527]: pgmap v442: 129 pgs: 129 active+clean; 9.9 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 324 MiB/s wr, 316 op/s 2024-09-19T06:53:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:48 smithi114 ceph-mon[30820]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 256 MiB/s wr, 388 op/s 2024-09-19T06:53:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:48 smithi116 ceph-mon[35411]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 256 MiB/s wr, 388 op/s 2024-09-19T06:53:48.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:48 smithi118 ceph-mon[35527]: pgmap v443: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 256 MiB/s wr, 388 op/s 2024-09-19T06:53:50.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:50 smithi114 ceph-mon[30820]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 269 MiB/s wr, 590 op/s 2024-09-19T06:53:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:50 smithi116 ceph-mon[35411]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 269 MiB/s wr, 590 op/s 2024-09-19T06:53:50.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:50 smithi118 ceph-mon[35527]: pgmap v444: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 269 MiB/s wr, 590 op/s 2024-09-19T06:53:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:52 smithi114 ceph-mon[30820]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 264 MiB/s wr, 767 op/s 2024-09-19T06:53:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:52 smithi116 ceph-mon[35411]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 264 MiB/s wr, 767 op/s 2024-09-19T06:53:52.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:52 smithi118 ceph-mon[35527]: pgmap v445: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 264 MiB/s wr, 767 op/s 2024-09-19T06:53:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:54 smithi114 ceph-mon[30820]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 816 op/s 2024-09-19T06:53:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:54 smithi116 ceph-mon[35411]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 816 op/s 2024-09-19T06:53:54.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:54 smithi118 ceph-mon[35527]: pgmap v446: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 816 op/s 2024-09-19T06:53:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:56 smithi114 ceph-mon[30820]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 293 MiB/s wr, 1.09k op/s 2024-09-19T06:53:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:56 smithi116 ceph-mon[35411]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 293 MiB/s wr, 1.09k op/s 2024-09-19T06:53:56.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:56 smithi118 ceph-mon[35527]: pgmap v447: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 293 MiB/s wr, 1.09k op/s 2024-09-19T06:53:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:53:58 smithi114 ceph-mon[30820]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 240 MiB/s wr, 948 op/s 2024-09-19T06:53:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:53:58 smithi116 ceph-mon[35411]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 240 MiB/s wr, 948 op/s 2024-09-19T06:53:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:53:58 smithi118 ceph-mon[35527]: pgmap v448: 129 pgs: 129 active+clean; 10 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 240 MiB/s wr, 948 op/s 2024-09-19T06:54:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:00 smithi114 ceph-mon[30820]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 262 MiB/s wr, 1.01k op/s 2024-09-19T06:54:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:00 smithi116 ceph-mon[35411]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 262 MiB/s wr, 1.01k op/s 2024-09-19T06:54:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:00 smithi118 ceph-mon[35527]: pgmap v449: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 262 MiB/s wr, 1.01k op/s 2024-09-19T06:54:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:01 smithi114 ceph-mon[30820]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 265 MiB/s wr, 951 op/s 2024-09-19T06:54:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:01 smithi116 ceph-mon[35411]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 265 MiB/s wr, 951 op/s 2024-09-19T06:54:01.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:01 smithi118 ceph-mon[35527]: pgmap v450: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 265 MiB/s wr, 951 op/s 2024-09-19T06:54:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:04 smithi114 ceph-mon[30820]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 234 MiB/s wr, 817 op/s 2024-09-19T06:54:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:04 smithi116 ceph-mon[35411]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 234 MiB/s wr, 817 op/s 2024-09-19T06:54:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:04 smithi118 ceph-mon[35527]: pgmap v451: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 234 MiB/s wr, 817 op/s 2024-09-19T06:54:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:06 smithi114 ceph-mon[30820]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 299 MiB/s wr, 1.04k op/s 2024-09-19T06:54:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:06 smithi116 ceph-mon[35411]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 299 MiB/s wr, 1.04k op/s 2024-09-19T06:54:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:06 smithi118 ceph-mon[35527]: pgmap v452: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 299 MiB/s wr, 1.04k op/s 2024-09-19T06:54:08.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:08 smithi114 ceph-mon[30820]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 244 MiB/s wr, 850 op/s 2024-09-19T06:54:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:08 smithi116 ceph-mon[35411]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 244 MiB/s wr, 850 op/s 2024-09-19T06:54:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:08 smithi118 ceph-mon[35527]: pgmap v453: 129 pgs: 129 active+clean; 10 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 244 MiB/s wr, 850 op/s 2024-09-19T06:54:10.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:10 smithi114 ceph-mon[30820]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 256 MiB/s wr, 905 op/s 2024-09-19T06:54:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:10 smithi116 ceph-mon[35411]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 256 MiB/s wr, 905 op/s 2024-09-19T06:54:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:10 smithi118 ceph-mon[35527]: pgmap v454: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 256 MiB/s wr, 905 op/s 2024-09-19T06:54:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:12 smithi114 ceph-mon[30820]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 255 MiB/s wr, 921 op/s 2024-09-19T06:54:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:12 smithi116 ceph-mon[35411]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 255 MiB/s wr, 921 op/s 2024-09-19T06:54:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:12 smithi118 ceph-mon[35527]: pgmap v455: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 255 MiB/s wr, 921 op/s 2024-09-19T06:54:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:14 smithi114 ceph-mon[30820]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 235 MiB/s wr, 863 op/s 2024-09-19T06:54:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:14 smithi116 ceph-mon[35411]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 235 MiB/s wr, 863 op/s 2024-09-19T06:54:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:14 smithi118 ceph-mon[35527]: pgmap v456: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 235 MiB/s wr, 863 op/s 2024-09-19T06:54:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:16 smithi114 ceph-mon[30820]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 289 MiB/s wr, 1.09k op/s 2024-09-19T06:54:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:16 smithi116 ceph-mon[35411]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 289 MiB/s wr, 1.09k op/s 2024-09-19T06:54:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:16 smithi118 ceph-mon[35527]: pgmap v457: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 289 MiB/s wr, 1.09k op/s 2024-09-19T06:54:18.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:18 smithi114 ceph-mon[30820]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 237 MiB/s wr, 916 op/s 2024-09-19T06:54:18.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:54:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:18 smithi116 ceph-mon[35411]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 237 MiB/s wr, 916 op/s 2024-09-19T06:54:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:54:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:18 smithi118 ceph-mon[35527]: pgmap v458: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 237 MiB/s wr, 916 op/s 2024-09-19T06:54:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:18 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:54:20.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:20 smithi114 ceph-mon[30820]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 255 MiB/s wr, 993 op/s 2024-09-19T06:54:20.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:54:20.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:54:20.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:54:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:20 smithi116 ceph-mon[35411]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 255 MiB/s wr, 993 op/s 2024-09-19T06:54:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:54:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:54:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:54:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:20 smithi118 ceph-mon[35527]: pgmap v459: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 255 MiB/s wr, 993 op/s 2024-09-19T06:54:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:54:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:54:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:54:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:22 smithi114 ceph-mon[30820]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 259 MiB/s wr, 1.03k op/s 2024-09-19T06:54:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:22 smithi116 ceph-mon[35411]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 259 MiB/s wr, 1.03k op/s 2024-09-19T06:54:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:22 smithi118 ceph-mon[35527]: pgmap v460: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 259 MiB/s wr, 1.03k op/s 2024-09-19T06:54:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:23 smithi114 ceph-mon[30820]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 231 MiB/s wr, 940 op/s 2024-09-19T06:54:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:23 smithi116 ceph-mon[35411]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 231 MiB/s wr, 940 op/s 2024-09-19T06:54:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:23 smithi118 ceph-mon[35527]: pgmap v461: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 231 MiB/s wr, 940 op/s 2024-09-19T06:54:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:26 smithi114 ceph-mon[30820]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 278 MiB/s wr, 1.12k op/s 2024-09-19T06:54:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:26 smithi116 ceph-mon[35411]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 278 MiB/s wr, 1.12k op/s 2024-09-19T06:54:26.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:26 smithi118 ceph-mon[35527]: pgmap v462: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 278 MiB/s wr, 1.12k op/s 2024-09-19T06:54:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:28 smithi114 ceph-mon[30820]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 926 op/s 2024-09-19T06:54:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:28 smithi116 ceph-mon[35411]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 926 op/s 2024-09-19T06:54:28.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:28 smithi118 ceph-mon[35527]: pgmap v463: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 926 op/s 2024-09-19T06:54:30.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:30 smithi114 ceph-mon[30820]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 253 MiB/s wr, 1.01k op/s 2024-09-19T06:54:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:30 smithi116 ceph-mon[35411]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 253 MiB/s wr, 1.01k op/s 2024-09-19T06:54:30.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:30 smithi118 ceph-mon[35527]: pgmap v464: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 253 MiB/s wr, 1.01k op/s 2024-09-19T06:54:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:32 smithi114 ceph-mon[30820]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 249 MiB/s wr, 996 op/s 2024-09-19T06:54:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:32 smithi116 ceph-mon[35411]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 249 MiB/s wr, 996 op/s 2024-09-19T06:54:32.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:32 smithi118 ceph-mon[35527]: pgmap v465: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 249 MiB/s wr, 996 op/s 2024-09-19T06:54:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:34 smithi114 ceph-mon[30820]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 226 MiB/s wr, 889 op/s 2024-09-19T06:54:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:34 smithi116 ceph-mon[35411]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 226 MiB/s wr, 889 op/s 2024-09-19T06:54:34.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:34 smithi118 ceph-mon[35527]: pgmap v466: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 226 MiB/s wr, 889 op/s 2024-09-19T06:54:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:36 smithi114 ceph-mon[30820]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 282 MiB/s wr, 1.11k op/s 2024-09-19T06:54:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:36 smithi116 ceph-mon[35411]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 282 MiB/s wr, 1.11k op/s 2024-09-19T06:54:36.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:36 smithi118 ceph-mon[35527]: pgmap v467: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 282 MiB/s wr, 1.11k op/s 2024-09-19T06:54:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:38 smithi114 ceph-mon[30820]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 236 MiB/s wr, 937 op/s 2024-09-19T06:54:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:38 smithi116 ceph-mon[35411]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 236 MiB/s wr, 937 op/s 2024-09-19T06:54:38.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:38 smithi118 ceph-mon[35527]: pgmap v468: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 236 MiB/s wr, 937 op/s 2024-09-19T06:54:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:40 smithi114 ceph-mon[30820]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 260 MiB/s wr, 1.04k op/s 2024-09-19T06:54:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:40 smithi116 ceph-mon[35411]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 260 MiB/s wr, 1.04k op/s 2024-09-19T06:54:40.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:40 smithi118 ceph-mon[35527]: pgmap v469: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 260 MiB/s wr, 1.04k op/s 2024-09-19T06:54:41.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:41 smithi118 ceph-mon[35527]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 254 MiB/s wr, 1.03k op/s 2024-09-19T06:54:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:41 smithi114 ceph-mon[30820]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 254 MiB/s wr, 1.03k op/s 2024-09-19T06:54:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:41 smithi116 ceph-mon[35411]: pgmap v470: 129 pgs: 129 active+clean; 10 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 254 MiB/s wr, 1.03k op/s 2024-09-19T06:54:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:44 smithi114 ceph-mon[30820]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 236 MiB/s wr, 957 op/s 2024-09-19T06:54:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:44 smithi116 ceph-mon[35411]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 236 MiB/s wr, 957 op/s 2024-09-19T06:54:44.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:44 smithi118 ceph-mon[35527]: pgmap v471: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 236 MiB/s wr, 957 op/s 2024-09-19T06:54:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:46 smithi114 ceph-mon[30820]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 290 MiB/s wr, 1.17k op/s 2024-09-19T06:54:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:46 smithi116 ceph-mon[35411]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 290 MiB/s wr, 1.17k op/s 2024-09-19T06:54:46.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:46 smithi118 ceph-mon[35527]: pgmap v472: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 290 MiB/s wr, 1.17k op/s 2024-09-19T06:54:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:48 smithi114 ceph-mon[30820]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 246 MiB/s wr, 996 op/s 2024-09-19T06:54:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:48 smithi116 ceph-mon[35411]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 246 MiB/s wr, 996 op/s 2024-09-19T06:54:48.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:48 smithi118 ceph-mon[35527]: pgmap v473: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 246 MiB/s wr, 996 op/s 2024-09-19T06:54:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:50 smithi114 ceph-mon[30820]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 273 MiB/s wr, 1.10k op/s 2024-09-19T06:54:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:50 smithi116 ceph-mon[35411]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 273 MiB/s wr, 1.10k op/s 2024-09-19T06:54:50.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:50 smithi118 ceph-mon[35527]: pgmap v474: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 273 MiB/s wr, 1.10k op/s 2024-09-19T06:54:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:52 smithi114 ceph-mon[30820]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 267 MiB/s wr, 1.08k op/s 2024-09-19T06:54:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:52 smithi116 ceph-mon[35411]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 267 MiB/s wr, 1.08k op/s 2024-09-19T06:54:52.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:52 smithi118 ceph-mon[35527]: pgmap v475: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 267 MiB/s wr, 1.08k op/s 2024-09-19T06:54:54.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:54 smithi114 ceph-mon[30820]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 248 MiB/s wr, 1.00k op/s 2024-09-19T06:54:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:54 smithi116 ceph-mon[35411]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 248 MiB/s wr, 1.00k op/s 2024-09-19T06:54:54.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:54 smithi118 ceph-mon[35527]: pgmap v476: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 248 MiB/s wr, 1.00k op/s 2024-09-19T06:54:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:56 smithi114 ceph-mon[30820]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 300 MiB/s wr, 1.22k op/s 2024-09-19T06:54:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:56 smithi116 ceph-mon[35411]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 300 MiB/s wr, 1.22k op/s 2024-09-19T06:54:56.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:56 smithi118 ceph-mon[35527]: pgmap v477: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 300 MiB/s wr, 1.22k op/s 2024-09-19T06:54:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:54:58 smithi114 ceph-mon[30820]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 250 MiB/s wr, 1.03k op/s 2024-09-19T06:54:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:54:58 smithi116 ceph-mon[35411]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 250 MiB/s wr, 1.03k op/s 2024-09-19T06:54:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:54:58 smithi118 ceph-mon[35527]: pgmap v478: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 250 MiB/s wr, 1.03k op/s 2024-09-19T06:55:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:00 smithi114 ceph-mon[30820]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 267 MiB/s wr, 1.11k op/s 2024-09-19T06:55:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:00 smithi116 ceph-mon[35411]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 267 MiB/s wr, 1.11k op/s 2024-09-19T06:55:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:00 smithi118 ceph-mon[35527]: pgmap v479: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 267 MiB/s wr, 1.11k op/s 2024-09-19T06:55:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:02 smithi114 ceph-mon[30820]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 251 MiB/s wr, 1.06k op/s 2024-09-19T06:55:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:02 smithi116 ceph-mon[35411]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 251 MiB/s wr, 1.06k op/s 2024-09-19T06:55:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:02 smithi118 ceph-mon[35527]: pgmap v480: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 251 MiB/s wr, 1.06k op/s 2024-09-19T06:55:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:04 smithi114 ceph-mon[30820]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 986 op/s 2024-09-19T06:55:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:04 smithi116 ceph-mon[35411]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 986 op/s 2024-09-19T06:55:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:04 smithi118 ceph-mon[35527]: pgmap v481: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 986 op/s 2024-09-19T06:55:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:06 smithi114 ceph-mon[30820]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 283 MiB/s wr, 1.18k op/s 2024-09-19T06:55:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:06 smithi116 ceph-mon[35411]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 283 MiB/s wr, 1.18k op/s 2024-09-19T06:55:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:06 smithi118 ceph-mon[35527]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 283 MiB/s wr, 1.18k op/s 2024-09-19T06:55:07.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:07 smithi118 ceph-mon[35527]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 986 op/s 2024-09-19T06:55:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:07 smithi114 ceph-mon[30820]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 986 op/s 2024-09-19T06:55:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:07 smithi116 ceph-mon[35411]: pgmap v483: 129 pgs: 129 active+clean; 10 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 986 op/s 2024-09-19T06:55:10.624 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:10 smithi114 ceph-mon[30820]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 262 MiB/s wr, 1.11k op/s 2024-09-19T06:55:10.631 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:10 smithi116 ceph-mon[35411]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 262 MiB/s wr, 1.11k op/s 2024-09-19T06:55:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:10 smithi118 ceph-mon[35527]: pgmap v484: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 262 MiB/s wr, 1.11k op/s 2024-09-19T06:55:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:12 smithi114 ceph-mon[30820]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 254 MiB/s wr, 1.09k op/s 2024-09-19T06:55:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:12 smithi116 ceph-mon[35411]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 254 MiB/s wr, 1.09k op/s 2024-09-19T06:55:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:12 smithi118 ceph-mon[35527]: pgmap v485: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 254 MiB/s wr, 1.09k op/s 2024-09-19T06:55:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:14 smithi114 ceph-mon[30820]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 247 MiB/s wr, 1.05k op/s 2024-09-19T06:55:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:14 smithi116 ceph-mon[35411]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 247 MiB/s wr, 1.05k op/s 2024-09-19T06:55:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:14 smithi118 ceph-mon[35527]: pgmap v486: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 247 MiB/s wr, 1.05k op/s 2024-09-19T06:55:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:16 smithi114 ceph-mon[30820]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 287 MiB/s wr, 1.22k op/s 2024-09-19T06:55:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:16 smithi116 ceph-mon[35411]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 287 MiB/s wr, 1.22k op/s 2024-09-19T06:55:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:16 smithi118 ceph-mon[35527]: pgmap v487: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 287 MiB/s wr, 1.22k op/s 2024-09-19T06:55:18.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:18 smithi114 ceph-mon[30820]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 246 MiB/s wr, 1.04k op/s 2024-09-19T06:55:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:18 smithi116 ceph-mon[35411]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 246 MiB/s wr, 1.04k op/s 2024-09-19T06:55:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:18 smithi118 ceph-mon[35527]: pgmap v488: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 246 MiB/s wr, 1.04k op/s 2024-09-19T06:55:20.672 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:20 smithi114 ceph-mon[30820]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 273 MiB/s wr, 1.16k op/s 2024-09-19T06:55:20.672 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:55:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:20 smithi116 ceph-mon[35411]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 273 MiB/s wr, 1.16k op/s 2024-09-19T06:55:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:55:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:20 smithi118 ceph-mon[35527]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 273 MiB/s wr, 1.16k op/s 2024-09-19T06:55:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:20 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:55:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:22 smithi114 ceph-mon[30820]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 258 MiB/s wr, 1.08k op/s 2024-09-19T06:55:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:55:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:55:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:55:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:22 smithi116 ceph-mon[35411]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 258 MiB/s wr, 1.08k op/s 2024-09-19T06:55:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:55:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:55:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:55:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:22 smithi118 ceph-mon[35527]: pgmap v490: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 258 MiB/s wr, 1.08k op/s 2024-09-19T06:55:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:55:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:55:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:55:24.672 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:24 smithi114 ceph-mon[30820]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 244 MiB/s wr, 1.01k op/s 2024-09-19T06:55:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:24 smithi116 ceph-mon[35411]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 244 MiB/s wr, 1.01k op/s 2024-09-19T06:55:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:24 smithi118 ceph-mon[35527]: pgmap v491: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 244 MiB/s wr, 1.01k op/s 2024-09-19T06:55:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:26 smithi114 ceph-mon[30820]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.17k op/s 2024-09-19T06:55:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:26 smithi116 ceph-mon[35411]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.17k op/s 2024-09-19T06:55:26.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:26 smithi118 ceph-mon[35527]: pgmap v492: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.17k op/s 2024-09-19T06:55:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:28 smithi116 ceph-mon[35411]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 980 op/s 2024-09-19T06:55:28.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:28 smithi118 ceph-mon[35527]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 980 op/s 2024-09-19T06:55:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:28 smithi114 ceph-mon[30820]: pgmap v493: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 980 op/s 2024-09-19T06:55:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:30 smithi116 ceph-mon[35411]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 253 MiB/s wr, 1.05k op/s 2024-09-19T06:55:30.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:30 smithi118 ceph-mon[35527]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 253 MiB/s wr, 1.05k op/s 2024-09-19T06:55:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:30 smithi114 ceph-mon[30820]: pgmap v494: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 253 MiB/s wr, 1.05k op/s 2024-09-19T06:55:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:31 smithi114 ceph-mon[30820]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 MiB/s wr, 1.01k op/s 2024-09-19T06:55:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:31 smithi116 ceph-mon[35411]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 MiB/s wr, 1.01k op/s 2024-09-19T06:55:32.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:31 smithi118 ceph-mon[35527]: pgmap v495: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 248 MiB/s wr, 1.01k op/s 2024-09-19T06:55:34.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:34 smithi114 ceph-mon[30820]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 235 MiB/s wr, 973 op/s 2024-09-19T06:55:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:34 smithi116 ceph-mon[35411]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 235 MiB/s wr, 973 op/s 2024-09-19T06:55:34.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:34 smithi118 ceph-mon[35527]: pgmap v496: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 235 MiB/s wr, 973 op/s 2024-09-19T06:55:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:36 smithi114 ceph-mon[30820]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.18k op/s 2024-09-19T06:55:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:36 smithi116 ceph-mon[35411]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.18k op/s 2024-09-19T06:55:36.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:36 smithi118 ceph-mon[35527]: pgmap v497: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.18k op/s 2024-09-19T06:55:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:38 smithi114 ceph-mon[30820]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 244 MiB/s wr, 1.02k op/s 2024-09-19T06:55:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:38 smithi116 ceph-mon[35411]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 244 MiB/s wr, 1.02k op/s 2024-09-19T06:55:38.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:38 smithi118 ceph-mon[35527]: pgmap v498: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 244 MiB/s wr, 1.02k op/s 2024-09-19T06:55:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:40 smithi114 ceph-mon[30820]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 272 MiB/s wr, 1.13k op/s 2024-09-19T06:55:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:40 smithi116 ceph-mon[35411]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 272 MiB/s wr, 1.13k op/s 2024-09-19T06:55:40.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:40 smithi118 ceph-mon[35527]: pgmap v499: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 272 MiB/s wr, 1.13k op/s 2024-09-19T06:55:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:42 smithi114 ceph-mon[30820]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 271 MiB/s wr, 1.13k op/s 2024-09-19T06:55:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:42 smithi116 ceph-mon[35411]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 271 MiB/s wr, 1.13k op/s 2024-09-19T06:55:42.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:42 smithi118 ceph-mon[35527]: pgmap v500: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 271 MiB/s wr, 1.13k op/s 2024-09-19T06:55:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:44 smithi114 ceph-mon[30820]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 253 MiB/s wr, 1.06k op/s 2024-09-19T06:55:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:44 smithi116 ceph-mon[35411]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 253 MiB/s wr, 1.06k op/s 2024-09-19T06:55:44.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:44 smithi118 ceph-mon[35527]: pgmap v501: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 253 MiB/s wr, 1.06k op/s 2024-09-19T06:55:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:46 smithi116 ceph-mon[35411]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 297 MiB/s wr, 1.24k op/s 2024-09-19T06:55:46.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:46 smithi118 ceph-mon[35527]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 297 MiB/s wr, 1.24k op/s 2024-09-19T06:55:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:46 smithi114 ceph-mon[30820]: pgmap v502: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 297 MiB/s wr, 1.24k op/s 2024-09-19T06:55:48.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:48 smithi118 ceph-mon[35527]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 250 MiB/s wr, 1.03k op/s 2024-09-19T06:55:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:48 smithi114 ceph-mon[30820]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 250 MiB/s wr, 1.03k op/s 2024-09-19T06:55:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:48 smithi116 ceph-mon[35411]: pgmap v503: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 250 MiB/s wr, 1.03k op/s 2024-09-19T06:55:49.757 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:49 smithi118 ceph-mon[35527]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 268 MiB/s wr, 1.11k op/s 2024-09-19T06:55:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:49 smithi114 ceph-mon[30820]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 268 MiB/s wr, 1.11k op/s 2024-09-19T06:55:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:49 smithi116 ceph-mon[35411]: pgmap v504: 129 pgs: 129 active+clean; 10 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 268 MiB/s wr, 1.11k op/s 2024-09-19T06:55:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:52 smithi114 ceph-mon[30820]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 261 MiB/s wr, 1.09k op/s 2024-09-19T06:55:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:52 smithi116 ceph-mon[35411]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 261 MiB/s wr, 1.09k op/s 2024-09-19T06:55:52.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:52 smithi118 ceph-mon[35527]: pgmap v505: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 261 MiB/s wr, 1.09k op/s 2024-09-19T06:55:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:54 smithi114 ceph-mon[30820]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 246 MiB/s wr, 1.02k op/s 2024-09-19T06:55:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:54 smithi116 ceph-mon[35411]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 246 MiB/s wr, 1.02k op/s 2024-09-19T06:55:54.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:54 smithi118 ceph-mon[35527]: pgmap v506: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 246 MiB/s wr, 1.02k op/s 2024-09-19T06:55:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:56 smithi114 ceph-mon[30820]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 290 MiB/s wr, 1.21k op/s 2024-09-19T06:55:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:56 smithi116 ceph-mon[35411]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 290 MiB/s wr, 1.21k op/s 2024-09-19T06:55:56.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:56 smithi118 ceph-mon[35527]: pgmap v507: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 290 MiB/s wr, 1.21k op/s 2024-09-19T06:55:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:55:58 smithi114 ceph-mon[30820]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 247 MiB/s wr, 1.03k op/s 2024-09-19T06:55:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:55:58 smithi116 ceph-mon[35411]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 247 MiB/s wr, 1.03k op/s 2024-09-19T06:55:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:55:58 smithi118 ceph-mon[35527]: pgmap v508: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 247 MiB/s wr, 1.03k op/s 2024-09-19T06:56:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:00 smithi114 ceph-mon[30820]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 260 MiB/s wr, 1.09k op/s 2024-09-19T06:56:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:00 smithi116 ceph-mon[35411]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 260 MiB/s wr, 1.09k op/s 2024-09-19T06:56:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:00 smithi118 ceph-mon[35527]: pgmap v509: 129 pgs: 129 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 260 MiB/s wr, 1.09k op/s 2024-09-19T06:56:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:02 smithi114 ceph-mon[30820]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 253 MiB/s wr, 1.05k op/s 2024-09-19T06:56:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:02 smithi116 ceph-mon[35411]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 253 MiB/s wr, 1.05k op/s 2024-09-19T06:56:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:02 smithi118 ceph-mon[35527]: pgmap v510: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 253 MiB/s wr, 1.05k op/s 2024-09-19T06:56:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:04 smithi114 ceph-mon[30820]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 227 MiB/s wr, 945 op/s 2024-09-19T06:56:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:04 smithi116 ceph-mon[35411]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 227 MiB/s wr, 945 op/s 2024-09-19T06:56:04.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:04 smithi118 ceph-mon[35527]: pgmap v511: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 227 MiB/s wr, 945 op/s 2024-09-19T06:56:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:06 smithi114 ceph-mon[30820]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 273 MiB/s wr, 1.15k op/s 2024-09-19T06:56:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:06 smithi116 ceph-mon[35411]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 273 MiB/s wr, 1.15k op/s 2024-09-19T06:56:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:06 smithi118 ceph-mon[35527]: pgmap v512: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 273 MiB/s wr, 1.15k op/s 2024-09-19T06:56:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:08 smithi114 ceph-mon[30820]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 230 MiB/s wr, 965 op/s 2024-09-19T06:56:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:08 smithi116 ceph-mon[35411]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 230 MiB/s wr, 965 op/s 2024-09-19T06:56:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:08 smithi118 ceph-mon[35527]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 230 MiB/s wr, 965 op/s 2024-09-19T06:56:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:10 smithi114 ceph-mon[30820]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 248 MiB/s wr, 1.04k op/s 2024-09-19T06:56:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:10 smithi116 ceph-mon[35411]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 248 MiB/s wr, 1.04k op/s 2024-09-19T06:56:10.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:10 smithi118 ceph-mon[35527]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 248 MiB/s wr, 1.04k op/s 2024-09-19T06:56:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:12 smithi116 ceph-mon[35411]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 248 MiB/s wr, 1.04k op/s 2024-09-19T06:56:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:12 smithi118 ceph-mon[35527]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 248 MiB/s wr, 1.04k op/s 2024-09-19T06:56:12.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:12 smithi114 ceph-mon[30820]: pgmap v515: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 248 MiB/s wr, 1.04k op/s 2024-09-19T06:56:13.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:13 smithi118 ceph-mon[35527]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 228 MiB/s wr, 953 op/s 2024-09-19T06:56:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:13 smithi114 ceph-mon[30820]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 228 MiB/s wr, 953 op/s 2024-09-19T06:56:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:13 smithi116 ceph-mon[35411]: pgmap v516: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 228 MiB/s wr, 953 op/s 2024-09-19T06:56:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:16 smithi114 ceph-mon[30820]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 286 MiB/s wr, 1.19k op/s 2024-09-19T06:56:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:16 smithi116 ceph-mon[35411]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 286 MiB/s wr, 1.19k op/s 2024-09-19T06:56:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:16 smithi118 ceph-mon[35527]: pgmap v517: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 286 MiB/s wr, 1.19k op/s 2024-09-19T06:56:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:18 smithi114 ceph-mon[30820]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 236 MiB/s wr, 979 op/s 2024-09-19T06:56:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:18 smithi116 ceph-mon[35411]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 236 MiB/s wr, 979 op/s 2024-09-19T06:56:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:18 smithi118 ceph-mon[35527]: pgmap v518: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 236 MiB/s wr, 979 op/s 2024-09-19T06:56:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:20 smithi114 ceph-mon[30820]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 254 MiB/s wr, 1.07k op/s 2024-09-19T06:56:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:20 smithi116 ceph-mon[35411]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 254 MiB/s wr, 1.07k op/s 2024-09-19T06:56:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:20 smithi118 ceph-mon[35527]: pgmap v519: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 254 MiB/s wr, 1.07k op/s 2024-09-19T06:56:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:22 smithi114 ceph-mon[30820]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 252 MiB/s wr, 1.06k op/s 2024-09-19T06:56:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:56:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:22 smithi116 ceph-mon[35411]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 252 MiB/s wr, 1.06k op/s 2024-09-19T06:56:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:56:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:22 smithi118 ceph-mon[35527]: pgmap v520: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 252 MiB/s wr, 1.06k op/s 2024-09-19T06:56:22.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:22 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:56:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:56:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:56:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:56:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:56:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:56:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:56:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:56:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:56:23.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:56:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:24 smithi114 ceph-mon[30820]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 978 op/s 2024-09-19T06:56:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:24 smithi116 ceph-mon[35411]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 978 op/s 2024-09-19T06:56:24.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:24 smithi118 ceph-mon[35527]: pgmap v521: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 978 op/s 2024-09-19T06:56:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:26 smithi114 ceph-mon[30820]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 283 MiB/s wr, 1.20k op/s 2024-09-19T06:56:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:26 smithi116 ceph-mon[35411]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 283 MiB/s wr, 1.20k op/s 2024-09-19T06:56:26.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:26 smithi118 ceph-mon[35527]: pgmap v522: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 283 MiB/s wr, 1.20k op/s 2024-09-19T06:56:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:28 smithi114 ceph-mon[30820]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 991 op/s 2024-09-19T06:56:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:28 smithi116 ceph-mon[35411]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 991 op/s 2024-09-19T06:56:28.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:28 smithi118 ceph-mon[35527]: pgmap v523: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 234 MiB/s wr, 991 op/s 2024-09-19T06:56:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:30 smithi116 ceph-mon[35411]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 256 MiB/s wr, 1.08k op/s 2024-09-19T06:56:30.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:30 smithi118 ceph-mon[35527]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 256 MiB/s wr, 1.08k op/s 2024-09-19T06:56:30.925 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:30 smithi114 ceph-mon[30820]: pgmap v524: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 256 MiB/s wr, 1.08k op/s 2024-09-19T06:56:32.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:32 smithi118 ceph-mon[35527]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 249 MiB/s wr, 1.04k op/s 2024-09-19T06:56:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:32 smithi114 ceph-mon[30820]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 249 MiB/s wr, 1.04k op/s 2024-09-19T06:56:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:32 smithi116 ceph-mon[35411]: pgmap v525: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 249 MiB/s wr, 1.04k op/s 2024-09-19T06:56:34.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:34 smithi118 ceph-mon[35527]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 224 MiB/s wr, 939 op/s 2024-09-19T06:56:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:34 smithi114 ceph-mon[30820]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 224 MiB/s wr, 939 op/s 2024-09-19T06:56:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:34 smithi116 ceph-mon[35411]: pgmap v526: 129 pgs: 129 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 224 MiB/s wr, 939 op/s 2024-09-19T06:56:36.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:36 smithi118 ceph-mon[35527]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 276 MiB/s wr, 1.16k op/s 2024-09-19T06:56:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:36 smithi114 ceph-mon[30820]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 276 MiB/s wr, 1.16k op/s 2024-09-19T06:56:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:36 smithi116 ceph-mon[35411]: pgmap v527: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 276 MiB/s wr, 1.16k op/s 2024-09-19T06:56:37.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:37 smithi114 ceph-mon[30820]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 231 MiB/s wr, 965 op/s 2024-09-19T06:56:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:37 smithi116 ceph-mon[35411]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 231 MiB/s wr, 965 op/s 2024-09-19T06:56:38.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:37 smithi118 ceph-mon[35527]: pgmap v528: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 231 MiB/s wr, 965 op/s 2024-09-19T06:56:40.627 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:40 smithi114 ceph-mon[30820]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 254 MiB/s wr, 1.06k op/s 2024-09-19T06:56:40.630 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:40 smithi116 ceph-mon[35411]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 254 MiB/s wr, 1.06k op/s 2024-09-19T06:56:40.709 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:40 smithi118 ceph-mon[35527]: pgmap v529: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 254 MiB/s wr, 1.06k op/s 2024-09-19T06:56:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:42 smithi114 ceph-mon[30820]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 247 MiB/s wr, 1.04k op/s 2024-09-19T06:56:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:42 smithi116 ceph-mon[35411]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 247 MiB/s wr, 1.04k op/s 2024-09-19T06:56:42.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:42 smithi118 ceph-mon[35527]: pgmap v530: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 247 MiB/s wr, 1.04k op/s 2024-09-19T06:56:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:44 smithi114 ceph-mon[30820]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 226 MiB/s wr, 948 op/s 2024-09-19T06:56:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:44 smithi116 ceph-mon[35411]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 226 MiB/s wr, 948 op/s 2024-09-19T06:56:44.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:44 smithi118 ceph-mon[35527]: pgmap v531: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 226 MiB/s wr, 948 op/s 2024-09-19T06:56:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:46 smithi116 ceph-mon[35411]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 279 MiB/s wr, 1.17k op/s 2024-09-19T06:56:46.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:46 smithi118 ceph-mon[35527]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 279 MiB/s wr, 1.17k op/s 2024-09-19T06:56:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:46 smithi114 ceph-mon[30820]: pgmap v532: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 279 MiB/s wr, 1.17k op/s 2024-09-19T06:56:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:48 smithi116 ceph-mon[35411]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 228 MiB/s wr, 953 op/s 2024-09-19T06:56:48.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:48 smithi118 ceph-mon[35527]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 228 MiB/s wr, 953 op/s 2024-09-19T06:56:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:48 smithi114 ceph-mon[30820]: pgmap v533: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 228 MiB/s wr, 953 op/s 2024-09-19T06:56:50.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:50 smithi118 ceph-mon[35527]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 255 MiB/s wr, 1.06k op/s 2024-09-19T06:56:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:50 smithi114 ceph-mon[30820]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 255 MiB/s wr, 1.06k op/s 2024-09-19T06:56:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:50 smithi116 ceph-mon[35411]: pgmap v534: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 255 MiB/s wr, 1.06k op/s 2024-09-19T06:56:52.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:52 smithi118 ceph-mon[35527]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 247 MiB/s wr, 1.03k op/s 2024-09-19T06:56:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:52 smithi114 ceph-mon[30820]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 247 MiB/s wr, 1.03k op/s 2024-09-19T06:56:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:52 smithi116 ceph-mon[35411]: pgmap v535: 129 pgs: 129 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 247 MiB/s wr, 1.03k op/s 2024-09-19T06:56:54.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:54 smithi118 ceph-mon[35527]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 968 op/s 2024-09-19T06:56:54.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:54 smithi114 ceph-mon[30820]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 968 op/s 2024-09-19T06:56:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:54 smithi116 ceph-mon[35411]: pgmap v536: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 233 MiB/s wr, 968 op/s 2024-09-19T06:56:56.553 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:56 smithi118 ceph-mon[35527]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.19k op/s 2024-09-19T06:56:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:56 smithi114 ceph-mon[30820]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.19k op/s 2024-09-19T06:56:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:56 smithi116 ceph-mon[35411]: pgmap v537: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 284 MiB/s wr, 1.19k op/s 2024-09-19T06:56:58.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:56:58 smithi118 ceph-mon[35527]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 MiB/s wr, 1.01k op/s 2024-09-19T06:56:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:56:58 smithi114 ceph-mon[30820]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 MiB/s wr, 1.01k op/s 2024-09-19T06:56:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:56:58 smithi116 ceph-mon[35411]: pgmap v538: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 MiB/s wr, 1.01k op/s 2024-09-19T06:57:00.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:00 smithi118 ceph-mon[35527]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 265 MiB/s wr, 1.10k op/s 2024-09-19T06:57:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:00 smithi114 ceph-mon[30820]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 265 MiB/s wr, 1.10k op/s 2024-09-19T06:57:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:00 smithi116 ceph-mon[35411]: pgmap v539: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 265 MiB/s wr, 1.10k op/s 2024-09-19T06:57:02.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:02 smithi118 ceph-mon[35527]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 254 MiB/s wr, 1.07k op/s 2024-09-19T06:57:02.924 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:02 smithi114 ceph-mon[30820]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 254 MiB/s wr, 1.07k op/s 2024-09-19T06:57:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:02 smithi116 ceph-mon[35411]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 254 MiB/s wr, 1.07k op/s 2024-09-19T06:57:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:03 smithi114 ceph-mon[30820]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 1.00k op/s 2024-09-19T06:57:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:03 smithi116 ceph-mon[35411]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 1.00k op/s 2024-09-19T06:57:04.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:03 smithi118 ceph-mon[35527]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 1.00k op/s 2024-09-19T06:57:06.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:06 smithi114 ceph-mon[30820]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 286 MiB/s wr, 1.20k op/s 2024-09-19T06:57:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:06 smithi116 ceph-mon[35411]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 286 MiB/s wr, 1.20k op/s 2024-09-19T06:57:06.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:06 smithi118 ceph-mon[35527]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 286 MiB/s wr, 1.20k op/s 2024-09-19T06:57:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:08 smithi114 ceph-mon[30820]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 242 MiB/s wr, 1.01k op/s 2024-09-19T06:57:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:08 smithi116 ceph-mon[35411]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 242 MiB/s wr, 1.01k op/s 2024-09-19T06:57:08.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:08 smithi118 ceph-mon[35527]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 242 MiB/s wr, 1.01k op/s 2024-09-19T06:57:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:10 smithi114 ceph-mon[30820]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 257 MiB/s wr, 1.09k op/s 2024-09-19T06:57:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:10 smithi116 ceph-mon[35411]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 257 MiB/s wr, 1.09k op/s 2024-09-19T06:57:10.726 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:10 smithi118 ceph-mon[35527]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 257 MiB/s wr, 1.09k op/s 2024-09-19T06:57:12.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:12 smithi118 ceph-mon[35527]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 252 MiB/s wr, 1.08k op/s 2024-09-19T06:57:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:12 smithi114 ceph-mon[30820]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 252 MiB/s wr, 1.08k op/s 2024-09-19T06:57:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:12 smithi116 ceph-mon[35411]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 252 MiB/s wr, 1.08k op/s 2024-09-19T06:57:14.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:14 smithi118 ceph-mon[35527]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 236 MiB/s wr, 999 op/s 2024-09-19T06:57:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:14 smithi114 ceph-mon[30820]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 236 MiB/s wr, 999 op/s 2024-09-19T06:57:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:14 smithi116 ceph-mon[35411]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 236 MiB/s wr, 999 op/s 2024-09-19T06:57:16.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:16 smithi118 ceph-mon[35527]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 287 MiB/s wr, 1.23k op/s 2024-09-19T06:57:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:16 smithi114 ceph-mon[30820]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 287 MiB/s wr, 1.23k op/s 2024-09-19T06:57:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:16 smithi116 ceph-mon[35411]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 287 MiB/s wr, 1.23k op/s 2024-09-19T06:57:18.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:18 smithi118 ceph-mon[35527]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 240 MiB/s wr, 1.04k op/s 2024-09-19T06:57:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:18 smithi114 ceph-mon[30820]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 240 MiB/s wr, 1.04k op/s 2024-09-19T06:57:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:18 smithi116 ceph-mon[35411]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 240 MiB/s wr, 1.04k op/s 2024-09-19T06:57:20.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:20 smithi118 ceph-mon[35527]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 264 MiB/s wr, 1.14k op/s 2024-09-19T06:57:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:20 smithi114 ceph-mon[30820]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 264 MiB/s wr, 1.14k op/s 2024-09-19T06:57:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:20 smithi116 ceph-mon[35411]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 264 MiB/s wr, 1.14k op/s 2024-09-19T06:57:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:22 smithi114 ceph-mon[30820]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 257 MiB/s wr, 1.11k op/s 2024-09-19T06:57:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:22 smithi116 ceph-mon[35411]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 257 MiB/s wr, 1.11k op/s 2024-09-19T06:57:23.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:22 smithi118 ceph-mon[35527]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 257 MiB/s wr, 1.11k op/s 2024-09-19T06:57:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:57:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:23 smithi114 ceph-mon[30820]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 237 MiB/s wr, 1.02k op/s 2024-09-19T06:57:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:57:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:23 smithi116 ceph-mon[35411]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 237 MiB/s wr, 1.02k op/s 2024-09-19T06:57:24.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:23 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:57:24.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:23 smithi118 ceph-mon[35527]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 237 MiB/s wr, 1.02k op/s 2024-09-19T06:57:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:57:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:57:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:57:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:57:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:57:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:57:25.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:57:25.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:57:25.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:24 smithi118 ceph-mon[35527]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:57:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:25 smithi114 ceph-mon[30820]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 289 MiB/s wr, 1.25k op/s 2024-09-19T06:57:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:25 smithi116 ceph-mon[35411]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 289 MiB/s wr, 1.25k op/s 2024-09-19T06:57:26.256 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:25 smithi118 ceph-mon[35527]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 289 MiB/s wr, 1.25k op/s 2024-09-19T06:57:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:28 smithi114 ceph-mon[30820]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 MiB/s wr, 1.04k op/s 2024-09-19T06:57:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:28 smithi116 ceph-mon[35411]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 MiB/s wr, 1.04k op/s 2024-09-19T06:57:28.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:28 smithi118 ceph-mon[35527]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 243 MiB/s wr, 1.04k op/s 2024-09-19T06:57:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:30 smithi116 ceph-mon[35411]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 264 MiB/s wr, 1.13k op/s 2024-09-19T06:57:30.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:30 smithi118 ceph-mon[35527]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 264 MiB/s wr, 1.13k op/s 2024-09-19T06:57:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:30 smithi114 ceph-mon[30820]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 264 MiB/s wr, 1.13k op/s 2024-09-19T06:57:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:32 smithi116 ceph-mon[35411]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 254 MiB/s wr, 1.10k op/s 2024-09-19T06:57:32.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:32 smithi118 ceph-mon[35527]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 254 MiB/s wr, 1.10k op/s 2024-09-19T06:57:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:32 smithi114 ceph-mon[30820]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 254 MiB/s wr, 1.10k op/s 2024-09-19T06:57:34.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:34 smithi118 ceph-mon[35527]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 249 MiB/s wr, 1.08k op/s 2024-09-19T06:57:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:34 smithi114 ceph-mon[30820]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 249 MiB/s wr, 1.08k op/s 2024-09-19T06:57:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:34 smithi116 ceph-mon[35411]: pgmap v556: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 249 MiB/s wr, 1.08k op/s 2024-09-19T06:57:36.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:36 smithi118 ceph-mon[35527]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 291 MiB/s wr, 1.26k op/s 2024-09-19T06:57:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:36 smithi114 ceph-mon[30820]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 291 MiB/s wr, 1.26k op/s 2024-09-19T06:57:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:36 smithi116 ceph-mon[35411]: pgmap v557: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 291 MiB/s wr, 1.26k op/s 2024-09-19T06:57:38.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:38 smithi118 ceph-mon[35527]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 1.03k op/s 2024-09-19T06:57:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:38 smithi114 ceph-mon[30820]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 1.03k op/s 2024-09-19T06:57:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:38 smithi116 ceph-mon[35411]: pgmap v558: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 239 MiB/s wr, 1.03k op/s 2024-09-19T06:57:40.756 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:40 smithi118 ceph-mon[35527]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 258 MiB/s wr, 1.12k op/s 2024-09-19T06:57:40.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:40 smithi114 ceph-mon[30820]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 258 MiB/s wr, 1.12k op/s 2024-09-19T06:57:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:40 smithi116 ceph-mon[35411]: pgmap v559: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 258 MiB/s wr, 1.12k op/s 2024-09-19T06:57:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:41 smithi114 ceph-mon[30820]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 249 MiB/s wr, 1.08k op/s 2024-09-19T06:57:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:41 smithi116 ceph-mon[35411]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 249 MiB/s wr, 1.08k op/s 2024-09-19T06:57:42.006 INFO:journalctl@ceph.mon.c.smithi118.stdout:Sep 19 06:57:41 smithi118 ceph-mon[35527]: pgmap v560: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 249 MiB/s wr, 1.08k op/s 2024-09-19T06:57:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:44 smithi114 ceph-mon[30820]: pgmap v561: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 241 MiB/s wr, 1.04k op/s 2024-09-19T06:57:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:44 smithi116 ceph-mon[35411]: pgmap v561: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 241 MiB/s wr, 1.04k op/s 2024-09-19T06:57:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:57:46 smithi116 ceph-mon[35411]: pgmap v562: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 280 MiB/s wr, 1.20k op/s 2024-09-19T06:57:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:57:46 smithi114 ceph-mon[30820]: pgmap v562: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 280 MiB/s wr, 1.20k op/s 2024-09-19T06:58:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: pgmap v564: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 252 MiB/s wr, 1.08k op/s 2024-09-19T06:58:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: pgmap v565: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 211 MiB/s wr, 901 op/s 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: pgmap v566: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 172 MiB/s wr, 739 op/s 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: pgmap v567: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 139 MiB/s wr, 597 op/s 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: mon.b calling monitor election 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: mon.a calling monitor election 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: pgmap v568: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 68 MiB/s wr, 292 op/s 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: pgmap v569: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 39 MiB/s wr, 171 op/s 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: pgmap v570: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 8.8 MiB/s wr, 40 op/s 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: mon.a is new leader, mons a,b in quorum (ranks 0,2) 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: monmap epoch 3 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: last_changed 2024-09-19T06:42:36.321900+0000 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: min_mon_release 19 (squid) 2024-09-19T06:58:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: election_strategy: 1 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: osdmap e95: 12 total, 12 up, 12 in 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: mgrmap e18: x(active, since 16m), standbys: z, y 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-19T06:58:02.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:02 smithi114 ceph-mon[30820]: mon.c (rank 1) addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] is down (out of quorum) 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: pgmap v564: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 252 MiB/s wr, 1.08k op/s 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: pgmap v565: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 211 MiB/s wr, 901 op/s 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: pgmap v566: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 172 MiB/s wr, 739 op/s 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: pgmap v567: 129 pgs: 129 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 139 MiB/s wr, 597 op/s 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: mon.b calling monitor election 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: mon.a calling monitor election 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: pgmap v568: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 68 MiB/s wr, 292 op/s 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: pgmap v569: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 39 MiB/s wr, 171 op/s 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: pgmap v570: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 8.8 MiB/s wr, 40 op/s 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: mon.a is new leader, mons a,b in quorum (ranks 0,2) 2024-09-19T06:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: monmap epoch 3 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: fsid fface640-7651-11ef-bceb-c7b262605968 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: last_changed 2024-09-19T06:42:36.321900+0000 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: created 2024-09-19T06:40:54.002091+0000 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: min_mon_release 19 (squid) 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: election_strategy: 1 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: 1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.c 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: 2: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: osdmap e95: 12 total, 12 up, 12 in 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: mgrmap e18: x(active, since 16m), standbys: z, y 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-19T06:58:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:02 smithi116 ceph-mon[35411]: mon.c (rank 1) addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] is down (out of quorum) 2024-09-19T06:58:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:04 smithi114 ceph-mon[30820]: pgmap v571: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 634 KiB/s wr, 5 op/s 2024-09-19T06:58:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:04 smithi116 ceph-mon[35411]: pgmap v571: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 634 KiB/s wr, 5 op/s 2024-09-19T06:58:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:06 smithi114 ceph-mon[30820]: pgmap v572: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 203 KiB/s wr, 2 op/s 2024-09-19T06:58:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:06 smithi116 ceph-mon[35411]: pgmap v572: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 203 KiB/s wr, 2 op/s 2024-09-19T06:58:07.679 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:58:07 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:58:07.384+0000 7fac1865f640 -1 osd.1 95 heartbeat_check: no reply from 172.21.15.118:6804 osd.2 since back 2024-09-19T06:57:46.278707+0000 front 2024-09-19T06:57:46.278696+0000 (oldest deadline 2024-09-19T06:58:07.378382+0000) 2024-09-19T06:58:07.679 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:58:07 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:58:07.384+0000 7fac1865f640 -1 osd.1 95 heartbeat_check: no reply from 172.21.15.118:6812 osd.5 since back 2024-09-19T06:57:46.278824+0000 front 2024-09-19T06:57:46.278662+0000 (oldest deadline 2024-09-19T06:58:07.378382+0000) 2024-09-19T06:58:07.679 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:58:07 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:58:07.384+0000 7fac1865f640 -1 osd.1 95 heartbeat_check: no reply from 172.21.15.118:6820 osd.8 since back 2024-09-19T06:57:46.279040+0000 front 2024-09-19T06:57:46.278913+0000 (oldest deadline 2024-09-19T06:58:07.378382+0000) 2024-09-19T06:58:07.679 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:58:07 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:58:07.384+0000 7fac1865f640 -1 osd.1 95 heartbeat_check: no reply from 172.21.15.118:6828 osd.11 since back 2024-09-19T06:57:46.278970+0000 front 2024-09-19T06:57:46.280332+0000 (oldest deadline 2024-09-19T06:58:07.378382+0000) 2024-09-19T06:58:08.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-0[44399]: 2024-09-19T06:58:08.270+0000 7f3bc9e00640 -1 osd.0 95 heartbeat_check: no reply from 172.21.15.118:6804 osd.2 since back 2024-09-19T06:57:44.206548+0000 front 2024-09-19T06:57:44.207002+0000 (oldest deadline 2024-09-19T06:58:07.706156+0000) 2024-09-19T06:58:08.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-0[44399]: 2024-09-19T06:58:08.270+0000 7f3bc9e00640 -1 osd.0 95 heartbeat_check: no reply from 172.21.15.118:6812 osd.5 since back 2024-09-19T06:57:44.206804+0000 front 2024-09-19T06:57:44.207053+0000 (oldest deadline 2024-09-19T06:58:07.706156+0000) 2024-09-19T06:58:08.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-0[44399]: 2024-09-19T06:58:08.270+0000 7f3bc9e00640 -1 osd.0 95 heartbeat_check: no reply from 172.21.15.118:6820 osd.8 since back 2024-09-19T06:57:44.206871+0000 front 2024-09-19T06:57:44.207089+0000 (oldest deadline 2024-09-19T06:58:07.706156+0000) 2024-09-19T06:58:08.667 INFO:journalctl@ceph.osd.0.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-0[44399]: 2024-09-19T06:58:08.270+0000 7f3bc9e00640 -1 osd.0 95 heartbeat_check: no reply from 172.21.15.118:6828 osd.11 since back 2024-09-19T06:57:44.206935+0000 front 2024-09-19T06:57:44.207475+0000 (oldest deadline 2024-09-19T06:58:07.706156+0000) 2024-09-19T06:58:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: pgmap v573: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 203 KiB/s wr, 1 op/s 2024-09-19T06:58:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.2 reported failed by osd.0 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.5 reported failed by osd.0 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.8 reported failed by osd.0 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.11 reported failed by osd.0 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.2 reported failed by osd.1 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.2 failed (root=default,host=smithi118) (2 reporters from different osd after 22.000170 >= grace 20.000000) 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.5 reported failed by osd.1 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.5 failed (root=default,host=smithi118) (2 reporters from different osd after 22.000403 >= grace 20.000000) 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.8 reported failed by osd.1 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.8 failed (root=default,host=smithi118) (2 reporters from different osd after 22.000598 >= grace 20.000000) 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.11 reported failed by osd.1 2024-09-19T06:58:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:08 smithi114 ceph-mon[30820]: osd.11 failed (root=default,host=smithi118) (2 reporters from different osd after 22.000756 >= grace 20.000000) 2024-09-19T06:58:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: pgmap v573: 129 pgs: 1 active+clean+laggy, 128 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 203 KiB/s wr, 1 op/s 2024-09-19T06:58:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.2 reported failed by osd.0 2024-09-19T06:58:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.5 reported failed by osd.0 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.8 reported failed by osd.0 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.11 reported failed by osd.0 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.2 reported failed by osd.1 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.2 failed (root=default,host=smithi118) (2 reporters from different osd after 22.000170 >= grace 20.000000) 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.5 reported failed by osd.1 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.5 failed (root=default,host=smithi118) (2 reporters from different osd after 22.000403 >= grace 20.000000) 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.8 reported failed by osd.1 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.8 failed (root=default,host=smithi118) (2 reporters from different osd after 22.000598 >= grace 20.000000) 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.11 reported failed by osd.1 2024-09-19T06:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-mon[35411]: osd.11 failed (root=default,host=smithi118) (2 reporters from different osd after 22.000756 >= grace 20.000000) 2024-09-19T06:58:08.679 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:58:08.379+0000 7fac1865f640 -1 osd.1 95 heartbeat_check: no reply from 172.21.15.118:6804 osd.2 since back 2024-09-19T06:57:46.278707+0000 front 2024-09-19T06:57:46.278696+0000 (oldest deadline 2024-09-19T06:58:07.378382+0000) 2024-09-19T06:58:08.679 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:58:08.379+0000 7fac1865f640 -1 osd.1 95 heartbeat_check: no reply from 172.21.15.118:6812 osd.5 since back 2024-09-19T06:57:46.278824+0000 front 2024-09-19T06:57:46.278662+0000 (oldest deadline 2024-09-19T06:58:07.378382+0000) 2024-09-19T06:58:08.679 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:58:08.379+0000 7fac1865f640 -1 osd.1 95 heartbeat_check: no reply from 172.21.15.118:6820 osd.8 since back 2024-09-19T06:57:46.279040+0000 front 2024-09-19T06:57:46.278913+0000 (oldest deadline 2024-09-19T06:58:07.378382+0000) 2024-09-19T06:58:08.680 INFO:journalctl@ceph.osd.1.smithi116.stdout:Sep 19 06:58:08 smithi116 ceph-fface640-7651-11ef-bceb-c7b262605968-osd-1[39847]: 2024-09-19T06:58:08.379+0000 7fac1865f640 -1 osd.1 95 heartbeat_check: no reply from 172.21.15.118:6828 osd.11 since back 2024-09-19T06:57:46.278970+0000 front 2024-09-19T06:57:46.280332+0000 (oldest deadline 2024-09-19T06:58:07.378382+0000) 2024-09-19T06:58:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:09 smithi114 ceph-mon[30820]: Health check failed: 4 osds down (OSD_DOWN) 2024-09-19T06:58:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:09 smithi114 ceph-mon[30820]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2024-09-19T06:58:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:09 smithi114 ceph-mon[30820]: osdmap e96: 12 total, 8 up, 12 in 2024-09-19T06:58:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:09 smithi114 ceph-mon[30820]: osdmap e97: 12 total, 8 up, 12 in 2024-09-19T06:58:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:09 smithi116 ceph-mon[35411]: Health check failed: 4 osds down (OSD_DOWN) 2024-09-19T06:58:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:09 smithi116 ceph-mon[35411]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2024-09-19T06:58:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:09 smithi116 ceph-mon[35411]: osdmap e96: 12 total, 8 up, 12 in 2024-09-19T06:58:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:09 smithi116 ceph-mon[35411]: osdmap e97: 12 total, 8 up, 12 in 2024-09-19T06:58:10.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:10 smithi114 ceph-mon[30820]: pgmap v575: 129 pgs: 1 down, 16 peering, 39 stale+active+clean, 1 active+clean+laggy, 72 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 7.9 MiB/s wr, 33 op/s 2024-09-19T06:58:10.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:10 smithi114 ceph-mon[30820]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg down, 12 pgs peering (PG_AVAILABILITY) 2024-09-19T06:58:10.829 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:10 smithi116 ceph-mon[35411]: pgmap v575: 129 pgs: 1 down, 16 peering, 39 stale+active+clean, 1 active+clean+laggy, 72 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 7.9 MiB/s wr, 33 op/s 2024-09-19T06:58:10.829 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:10 smithi116 ceph-mon[35411]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg down, 12 pgs peering (PG_AVAILABILITY) 2024-09-19T06:58:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:11 smithi114 ceph-mon[30820]: pgmap v577: 129 pgs: 1 undersized+peered, 5 active+undersized, 2 down, 23 peering, 31 stale+active+clean, 1 active+clean+laggy, 2 undersized+degraded+peered, 8 active+undersized+degraded, 56 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 92 op/s; 615/10620 objects degraded (5.791%) 2024-09-19T06:58:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:11 smithi114 ceph-mon[30820]: Health check failed: Degraded data redundancy: 615/10620 objects degraded (5.791%), 10 pgs degraded (PG_DEGRADED) 2024-09-19T06:58:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:11 smithi116 ceph-mon[35411]: pgmap v577: 129 pgs: 1 undersized+peered, 5 active+undersized, 2 down, 23 peering, 31 stale+active+clean, 1 active+clean+laggy, 2 undersized+degraded+peered, 8 active+undersized+degraded, 56 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 92 op/s; 615/10620 objects degraded (5.791%) 2024-09-19T06:58:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:11 smithi116 ceph-mon[35411]: Health check failed: Degraded data redundancy: 615/10620 objects degraded (5.791%), 10 pgs degraded (PG_DEGRADED) 2024-09-19T06:58:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:14 smithi114 ceph-mon[30820]: pgmap v578: 129 pgs: 4 undersized+peered, 17 active+undersized, 3 down, 23 peering, 17 stale+active+clean, 5 undersized+degraded+peered, 15 active+undersized+degraded, 45 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 165 op/s; 1094/10620 objects degraded (10.301%) 2024-09-19T06:58:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:14 smithi116 ceph-mon[35411]: pgmap v578: 129 pgs: 4 undersized+peered, 17 active+undersized, 3 down, 23 peering, 17 stale+active+clean, 5 undersized+degraded+peered, 15 active+undersized+degraded, 45 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 39 MiB/s wr, 165 op/s; 1094/10620 objects degraded (10.301%) 2024-09-19T06:58:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:15 smithi116 ceph-mon[35411]: Health check update: Reduced data availability: 4 pgs inactive, 3 pgs down, 13 pgs peering (PG_AVAILABILITY) 2024-09-19T06:58:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:15 smithi114 ceph-mon[30820]: Health check update: Reduced data availability: 4 pgs inactive, 3 pgs down, 13 pgs peering (PG_AVAILABILITY) 2024-09-19T06:58:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:16 smithi116 ceph-mon[35411]: pgmap v579: 129 pgs: 13 undersized+peered, 29 active+undersized, 3 down, 7 peering, 2 stale+active+clean, 14 undersized+degraded+peered, 30 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 241 op/s; 2465/10620 objects degraded (23.211%) 2024-09-19T06:58:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:16 smithi116 ceph-mon[35411]: osdmap e98: 12 total, 8 up, 12 in 2024-09-19T06:58:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:16 smithi114 ceph-mon[30820]: pgmap v579: 129 pgs: 13 undersized+peered, 29 active+undersized, 3 down, 7 peering, 2 stale+active+clean, 14 undersized+degraded+peered, 30 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 241 op/s; 2465/10620 objects degraded (23.211%) 2024-09-19T06:58:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:16 smithi114 ceph-mon[30820]: osdmap e98: 12 total, 8 up, 12 in 2024-09-19T06:58:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:17 smithi114 ceph-mon[30820]: Health check update: Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded (PG_DEGRADED) 2024-09-19T06:58:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:17 smithi116 ceph-mon[35411]: Health check update: Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded (PG_DEGRADED) 2024-09-19T06:58:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:18 smithi114 ceph-mon[30820]: pgmap v581: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 218 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:18 smithi116 ceph-mon[35411]: pgmap v581: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 218 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:20 smithi114 ceph-mon[30820]: pgmap v582: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 188 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:20 smithi114 ceph-mon[30820]: Health check update: Reduced data availability: 9 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-19T06:58:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:20 smithi116 ceph-mon[35411]: pgmap v582: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 188 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:20 smithi116 ceph-mon[35411]: Health check update: Reduced data availability: 9 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2024-09-19T06:58:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:22 smithi114 ceph-mon[30820]: pgmap v583: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 149 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:22 smithi116 ceph-mon[35411]: pgmap v583: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s wr, 149 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:24 smithi114 ceph-mon[30820]: pgmap v584: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 91 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:24 smithi116 ceph-mon[35411]: pgmap v584: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 91 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:58:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:58:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: pgmap v585: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 7.7 MiB/s wr, 36 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T06:58:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T06:58:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:58:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:58:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:58:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: pgmap v585: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 7.7 MiB/s wr, 36 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T06:58:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T06:58:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T06:58:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:58:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:58:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:58:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:27 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T06:58:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:27 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T06:58:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:27 smithi114 ceph-mon[30820]: pgmap v586: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7.4 MiB/s wr, 35 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:27 smithi114 ceph-mon[30820]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-09-19T06:58:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:27 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T06:58:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:27 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T06:58:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:27 smithi116 ceph-mon[35411]: pgmap v586: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7.4 MiB/s wr, 35 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:27 smithi116 ceph-mon[35411]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-09-19T06:58:29.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:29 smithi114 ceph-mon[30820]: pgmap v587: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4.3 MiB/s wr, 21 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:29.933 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:29 smithi116 ceph-mon[35411]: pgmap v587: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4.3 MiB/s wr, 21 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:31 smithi114 ceph-mon[30820]: pgmap v588: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 11 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:31 smithi116 ceph-mon[35411]: pgmap v588: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 11 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:33 smithi114 ceph-mon[30820]: pgmap v589: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:33 smithi116 ceph-mon[35411]: pgmap v589: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:35 smithi114 ceph-mon[30820]: pgmap v590: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:35 smithi114 ceph-mon[30820]: mgrmap e19: x(active, since 16m), standbys: y 2024-09-19T06:58:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:35 smithi116 ceph-mon[35411]: pgmap v590: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:35 smithi116 ceph-mon[35411]: mgrmap e19: x(active, since 16m), standbys: y 2024-09-19T06:58:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:36 smithi114 ceph-mon[30820]: pgmap v591: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:36 smithi116 ceph-mon[35411]: pgmap v591: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:38 smithi114 ceph-mon[30820]: pgmap v592: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:38 smithi116 ceph-mon[35411]: pgmap v592: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:40 smithi114 ceph-mon[30820]: pgmap v593: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:40 smithi116 ceph-mon[35411]: pgmap v593: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:42 smithi114 ceph-mon[30820]: pgmap v594: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:42 smithi116 ceph-mon[35411]: pgmap v594: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:44 smithi114 ceph-mon[30820]: pgmap v595: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:44 smithi116 ceph-mon[35411]: pgmap v595: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:46 smithi114 ceph-mon[30820]: pgmap v596: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:46 smithi116 ceph-mon[35411]: pgmap v596: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:48 smithi114 ceph-mon[30820]: pgmap v597: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:48 smithi116 ceph-mon[35411]: pgmap v597: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:50 smithi114 ceph-mon[30820]: pgmap v598: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:50 smithi116 ceph-mon[35411]: pgmap v598: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:52 smithi114 ceph-mon[30820]: pgmap v599: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:52 smithi116 ceph-mon[35411]: pgmap v599: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:54 smithi114 ceph-mon[30820]: pgmap v600: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:54 smithi116 ceph-mon[35411]: pgmap v600: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:56 smithi114 ceph-mon[30820]: pgmap v601: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:56 smithi116 ceph-mon[35411]: pgmap v601: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:58:58 smithi116 ceph-mon[35411]: pgmap v602: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:58:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:58:58 smithi114 ceph-mon[30820]: pgmap v602: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:00 smithi116 ceph-mon[35411]: pgmap v603: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:00 smithi114 ceph-mon[30820]: pgmap v603: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:02 smithi116 ceph-mon[35411]: pgmap v604: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:02 smithi114 ceph-mon[30820]: pgmap v604: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: pgmap v605: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: Replacing daemon mds.f as rank 3 with standby daemon mds.k 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: Replacing daemon mds.c as rank 2 with standby daemon mds.j 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: Replacing daemon mds.i as rank 0 with standby daemon mds.g 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: osdmap e99: 12 total, 8 up, 12 in 2024-09-19T06:59:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:04 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:replay 2 up:active 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: pgmap v605: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: Replacing daemon mds.f as rank 3 with standby daemon mds.k 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: Replacing daemon mds.c as rank 2 with standby daemon mds.j 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: Replacing daemon mds.i as rank 0 with standby daemon mds.g 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: osdmap e99: 12 total, 8 up, 12 in 2024-09-19T06:59:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:04 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:replay 2 up:active 2024-09-19T06:59:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:06 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T06:59:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:06 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T06:59:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:06 smithi114 ceph-mon[30820]: pgmap v607: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:06 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T06:59:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:06 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T06:59:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:06 smithi116 ceph-mon[35411]: pgmap v607: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:08 smithi114 ceph-mon[30820]: pgmap v608: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:08 smithi116 ceph-mon[35411]: pgmap v608: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:09 smithi114 ceph-mon[30820]: Health check update: Reduced data availability: 32 pgs inactive, 3 pgs down, 2 pgs stale (PG_AVAILABILITY) 2024-09-19T06:59:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:09 smithi114 ceph-mon[30820]: Health check update: Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded, 86 pgs undersized (PG_DEGRADED) 2024-09-19T06:59:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:09 smithi116 ceph-mon[35411]: Health check update: Reduced data availability: 32 pgs inactive, 3 pgs down, 2 pgs stale (PG_AVAILABILITY) 2024-09-19T06:59:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:09 smithi116 ceph-mon[35411]: Health check update: Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded, 86 pgs undersized (PG_DEGRADED) 2024-09-19T06:59:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:10 smithi114 ceph-mon[30820]: pgmap v609: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:10 smithi116 ceph-mon[35411]: pgmap v609: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:12 smithi114 ceph-mon[30820]: pgmap v610: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:12 smithi116 ceph-mon[35411]: pgmap v610: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:14 smithi114 ceph-mon[30820]: pgmap v611: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:14 smithi116 ceph-mon[35411]: pgmap v611: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:16 smithi114 ceph-mon[30820]: Health check update: Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded, 93 pgs undersized (PG_DEGRADED) 2024-09-19T06:59:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:16 smithi116 ceph-mon[35411]: Health check update: Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded, 93 pgs undersized (PG_DEGRADED) 2024-09-19T06:59:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:17 smithi114 ceph-mon[30820]: pgmap v612: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:17 smithi116 ceph-mon[35411]: pgmap v612: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 5 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:19 smithi114 ceph-mon[30820]: pgmap v613: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:19 smithi116 ceph-mon[35411]: pgmap v613: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:21 smithi114 ceph-mon[30820]: pgmap v614: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:21 smithi116 ceph-mon[35411]: pgmap v614: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:23 smithi114 ceph-mon[30820]: pgmap v615: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:23 smithi116 ceph-mon[35411]: pgmap v615: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:25 smithi114 ceph-mon[30820]: pgmap v616: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:25 smithi116 ceph-mon[35411]: pgmap v616: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:59:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T06:59:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:27 smithi114 ceph-mon[30820]: pgmap v617: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:59:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:59:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:59:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:27 smithi116 ceph-mon[35411]: pgmap v617: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T06:59:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T06:59:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T06:59:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:28 smithi114 ceph-mon[30820]: pgmap v618: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:28 smithi114 ceph-mon[30820]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-09-19T06:59:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:28 smithi116 ceph-mon[35411]: pgmap v618: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:28 smithi116 ceph-mon[35411]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-09-19T06:59:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:30 smithi114 ceph-mon[30820]: pgmap v619: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:30 smithi116 ceph-mon[35411]: pgmap v619: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:32 smithi114 ceph-mon[30820]: pgmap v620: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:32 smithi116 ceph-mon[35411]: pgmap v620: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:34 smithi114 ceph-mon[30820]: pgmap v621: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:34 smithi116 ceph-mon[35411]: pgmap v621: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:36 smithi114 ceph-mon[30820]: pgmap v622: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:36 smithi116 ceph-mon[35411]: pgmap v622: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:38 smithi114 ceph-mon[30820]: pgmap v623: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:38 smithi116 ceph-mon[35411]: pgmap v623: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:40 smithi114 ceph-mon[30820]: pgmap v624: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:40 smithi116 ceph-mon[35411]: pgmap v624: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:42 smithi114 ceph-mon[30820]: pgmap v625: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:42 smithi116 ceph-mon[35411]: pgmap v625: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:44 smithi114 ceph-mon[30820]: pgmap v626: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:44 smithi116 ceph-mon[35411]: pgmap v626: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:46 smithi114 ceph-mon[30820]: pgmap v627: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:46 smithi116 ceph-mon[35411]: pgmap v627: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:48 smithi114 ceph-mon[30820]: pgmap v628: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:48.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:48 smithi116 ceph-mon[35411]: pgmap v628: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:50 smithi114 ceph-mon[30820]: pgmap v629: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:50 smithi116 ceph-mon[35411]: pgmap v629: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:52 smithi114 ceph-mon[30820]: pgmap v630: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:52 smithi116 ceph-mon[35411]: pgmap v630: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:54 smithi114 ceph-mon[30820]: pgmap v631: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:54 smithi116 ceph-mon[35411]: pgmap v631: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:56 smithi114 ceph-mon[30820]: pgmap v632: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:56 smithi116 ceph-mon[35411]: pgmap v632: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 06:59:58 smithi114 ceph-mon[30820]: pgmap v633: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T06:59:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 06:59:58 smithi116 ceph-mon[35411]: pgmap v633: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pgmap v634: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: Health detail: HEALTH_WARN 1 filesystem is degraded; 1/3 mons down, quorum a,b; 4 osds down; 1 host (4 osds) down; Reduced data availability: 32 pgs inactive, 3 pgs down, 2 pgs stale; Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded, 93 pgs undersized 2024-09-19T07:00:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-09-19T07:00:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: fs cephfs is degraded 2024-09-19T07:00:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-19T07:00:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: mon.c (rank 1) addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] is down (out of quorum) 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: [WRN] OSD_DOWN: 4 osds down 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: osd.2 (root=default,host=smithi118) is down 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: osd.5 (root=default,host=smithi118) is down 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: osd.8 (root=default,host=smithi118) is down 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: osd.11 (root=default,host=smithi118) is down 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: host smithi118 (root=default) (4 osds) is down 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: [WRN] PG_AVAILABILITY: Reduced data availability: 32 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.0 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.a is stuck inactive for 110s, current state undersized+degraded+peered, last acting [3] 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.b is stuck inactive for 110s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.c is stuck inactive for 110s, current state undersized+degraded+peered, last acting [0] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.f is stuck inactive for 110s, current state undersized+degraded+peered, last acting [4] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.1c is stuck inactive for 110s, current state undersized+degraded+peered, last acting [1] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.1d is stuck inactive for 110s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.0 is stuck inactive for 10m, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.8 is stuck inactive for 10m, current state undersized+peered, last acting [0] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.9 is stuck inactive for 10m, current state undersized+peered, last acting [10] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.e is stuck inactive for 10m, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.f is stuck inactive for 110s, current state undersized+peered, last acting [7] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.15 is stuck inactive for 10m, current state undersized+peered, last acting [7] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.1a is stuck inactive for 10m, current state undersized+peered, last acting [9] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.1e is stuck inactive for 110s, current state undersized+peered, last acting [9] 2024-09-19T07:00:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.20 is stuck stale for 110s, current state stale+active+clean, last acting [11,2,5] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.21 is stuck inactive for 10m, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.25 is stuck stale for 110s, current state stale+active+clean, last acting [11,8,5] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.26 is stuck inactive for 10m, current state undersized+peered, last acting [1] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.29 is stuck inactive for 110s, current state undersized+peered, last acting [3] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.2c is stuck inactive for 10m, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.31 is stuck inactive for 110s, current state undersized+peered, last acting [0] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.37 is stuck inactive for 110s, current state undersized+peered, last acting [1] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.3c is stuck inactive for 110s, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.2 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,10,NONE,9] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.3 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,NONE,7,6] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.5 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [7,NONE,NONE,9] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.8 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,NONE,7,1] 2024-09-19T07:00:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.f is down, acting [NONE,3,NONE,NONE] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.12 is down, acting [9,NONE,NONE,NONE] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.14 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,NONE,7,6] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.15 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,NONE,3,6] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.1c is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,6,9,NONE] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.1d is down, acting [NONE,NONE,7,NONE] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: [WRN] PG_DEGRADED: Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded, 93 pgs undersized 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 1.0 is stuck undersized for 109s, current state active+undersized+degraded, last acting [7,10] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.0 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.4 is stuck undersized for 103s, current state active+undersized, last acting [9,0] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.5 is stuck undersized for 109s, current state active+undersized+degraded, last acting [0,9] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.6 is stuck undersized for 103s, current state active+undersized+degraded, last acting [9,3] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.7 is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,6] 2024-09-19T07:00:00.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.8 is stuck undersized for 109s, current state active+undersized+degraded, last acting [1,0] 2024-09-19T07:00:00.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.a is stuck undersized for 109s, current state undersized+degraded+peered, last acting [3] 2024-09-19T07:00:00.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.b is stuck undersized for 109s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.c is stuck undersized for 109s, current state undersized+degraded+peered, last acting [0] 2024-09-19T07:00:00.425 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.d is stuck undersized for 109s, current state active+undersized+degraded, last acting [10,3] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.e is stuck undersized for 109s, current state active+undersized+degraded, last acting [6,3] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.f is stuck undersized for 109s, current state undersized+degraded+peered, last acting [4] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.10 is stuck undersized for 109s, current state active+undersized+degraded, last acting [0,4] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.1c is stuck undersized for 109s, current state undersized+degraded+peered, last acting [1] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.1d is stuck undersized for 109s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.1e is stuck undersized for 109s, current state active+undersized, last acting [3,0] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 2.1f is stuck undersized for 103s, current state active+undersized+degraded, last acting [9,7] 2024-09-19T07:00:00.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.0 is stuck undersized for 109s, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.1 is stuck undersized for 109s, current state active+undersized, last acting [6,4] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.2 is stuck undersized for 109s, current state active+undersized, last acting [3,1] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.4 is stuck undersized for 109s, current state active+undersized, last acting [6,9] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.7 is stuck undersized for 109s, current state active+undersized, last acting [3,7] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.8 is stuck undersized for 109s, current state undersized+peered, last acting [0] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.9 is stuck undersized for 109s, current state undersized+peered, last acting [10] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.c is stuck undersized for 109s, current state active+undersized, last acting [7,3] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.d is stuck undersized for 109s, current state active+undersized, last acting [7,1] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.e is stuck undersized for 109s, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.f is stuck undersized for 109s, current state undersized+peered, last acting [7] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.10 is stuck undersized for 109s, current state active+undersized, last acting [7,10] 2024-09-19T07:00:00.427 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.11 is stuck undersized for 109s, current state active+undersized, last acting [4,1] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.1d is stuck undersized for 109s, current state active+undersized, last acting [6,1] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 3.1e is stuck undersized for 103s, current state undersized+peered, last acting [9] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.1 is stuck undersized for 109s, current state active+undersized+degraded, last acting [NONE,1,4,7] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.2 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [NONE,10,NONE,9] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.3 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [NONE,NONE,7,6] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.4 is stuck undersized for 109s, current state active+undersized+degraded, last acting [6,0,1,NONE] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.5 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [7,NONE,NONE,9] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.6 is stuck undersized for 103s, current state active+undersized+degraded, last acting [NONE,9,6,10] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.7 is stuck undersized for 109s, current state active+undersized+degraded, last acting [6,NONE,10,0] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.8 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [NONE,NONE,7,1] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.9 is stuck undersized for 109s, current state active+undersized+degraded, last acting [NONE,0,3,1] 2024-09-19T07:00:00.428 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.a is stuck undersized for 109s, current state active+undersized+degraded, last acting [1,NONE,6,7] 2024-09-19T07:00:00.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.c is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,9,7,NONE] 2024-09-19T07:00:00.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.d is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,NONE,10,3] 2024-09-19T07:00:00.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.e is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,NONE,10,9] 2024-09-19T07:00:00.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.16 is stuck undersized for 109s, current state active+undersized+degraded, last acting [6,3,9,NONE] 2024-09-19T07:00:00.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.19 is stuck undersized for 109s, current state active+undersized+degraded, last acting [NONE,3,9,0] 2024-09-19T07:00:00.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.1a is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,6,NONE,3] 2024-09-19T07:00:00.429 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:00 smithi114 ceph-mon[30820]: pg 4.1b is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,3,9,NONE] 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pgmap v634: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: Health detail: HEALTH_WARN 1 filesystem is degraded; 1/3 mons down, quorum a,b; 4 osds down; 1 host (4 osds) down; Reduced data availability: 32 pgs inactive, 3 pgs down, 2 pgs stale; Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded, 93 pgs undersized 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: fs cephfs is degraded 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: mon.c (rank 1) addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] is down (out of quorum) 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: [WRN] OSD_DOWN: 4 osds down 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: osd.2 (root=default,host=smithi118) is down 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: osd.5 (root=default,host=smithi118) is down 2024-09-19T07:00:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: osd.8 (root=default,host=smithi118) is down 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: osd.11 (root=default,host=smithi118) is down 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: host smithi118 (root=default) (4 osds) is down 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: [WRN] PG_AVAILABILITY: Reduced data availability: 32 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.0 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.a is stuck inactive for 110s, current state undersized+degraded+peered, last acting [3] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.b is stuck inactive for 110s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.c is stuck inactive for 110s, current state undersized+degraded+peered, last acting [0] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.f is stuck inactive for 110s, current state undersized+degraded+peered, last acting [4] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.1c is stuck inactive for 110s, current state undersized+degraded+peered, last acting [1] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.1d is stuck inactive for 110s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.0 is stuck inactive for 10m, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.8 is stuck inactive for 10m, current state undersized+peered, last acting [0] 2024-09-19T07:00:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.9 is stuck inactive for 10m, current state undersized+peered, last acting [10] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.e is stuck inactive for 10m, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.f is stuck inactive for 110s, current state undersized+peered, last acting [7] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.15 is stuck inactive for 10m, current state undersized+peered, last acting [7] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.1a is stuck inactive for 10m, current state undersized+peered, last acting [9] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.1e is stuck inactive for 110s, current state undersized+peered, last acting [9] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.20 is stuck stale for 110s, current state stale+active+clean, last acting [11,2,5] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.21 is stuck inactive for 10m, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.25 is stuck stale for 110s, current state stale+active+clean, last acting [11,8,5] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.26 is stuck inactive for 10m, current state undersized+peered, last acting [1] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.29 is stuck inactive for 110s, current state undersized+peered, last acting [3] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.2c is stuck inactive for 10m, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.31 is stuck inactive for 110s, current state undersized+peered, last acting [0] 2024-09-19T07:00:00.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.37 is stuck inactive for 110s, current state undersized+peered, last acting [1] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.3c is stuck inactive for 110s, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.2 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,10,NONE,9] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.3 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,NONE,7,6] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.5 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [7,NONE,NONE,9] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.8 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,NONE,7,1] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.f is down, acting [NONE,3,NONE,NONE] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.12 is down, acting [9,NONE,NONE,NONE] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.14 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,NONE,7,6] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.15 is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,NONE,3,6] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.1c is stuck inactive for 110s, current state undersized+degraded+peered, last acting [NONE,6,9,NONE] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.1d is down, acting [NONE,NONE,7,NONE] 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: [WRN] PG_DEGRADED: Degraded data redundancy: 2627/10620 objects degraded (24.736%), 48 pgs degraded, 93 pgs undersized 2024-09-19T07:00:00.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 1.0 is stuck undersized for 109s, current state active+undersized+degraded, last acting [7,10] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.0 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.4 is stuck undersized for 103s, current state active+undersized, last acting [9,0] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.5 is stuck undersized for 109s, current state active+undersized+degraded, last acting [0,9] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.6 is stuck undersized for 103s, current state active+undersized+degraded, last acting [9,3] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.7 is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,6] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.8 is stuck undersized for 109s, current state active+undersized+degraded, last acting [1,0] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.a is stuck undersized for 109s, current state undersized+degraded+peered, last acting [3] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.b is stuck undersized for 109s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.c is stuck undersized for 109s, current state undersized+degraded+peered, last acting [0] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.d is stuck undersized for 109s, current state active+undersized+degraded, last acting [10,3] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.e is stuck undersized for 109s, current state active+undersized+degraded, last acting [6,3] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.f is stuck undersized for 109s, current state undersized+degraded+peered, last acting [4] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.10 is stuck undersized for 109s, current state active+undersized+degraded, last acting [0,4] 2024-09-19T07:00:00.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.1c is stuck undersized for 109s, current state undersized+degraded+peered, last acting [1] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.1d is stuck undersized for 109s, current state undersized+degraded+peered, last acting [7] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.1e is stuck undersized for 109s, current state active+undersized, last acting [3,0] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 2.1f is stuck undersized for 103s, current state active+undersized+degraded, last acting [9,7] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.0 is stuck undersized for 109s, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.1 is stuck undersized for 109s, current state active+undersized, last acting [6,4] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.2 is stuck undersized for 109s, current state active+undersized, last acting [3,1] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.4 is stuck undersized for 109s, current state active+undersized, last acting [6,9] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.7 is stuck undersized for 109s, current state active+undersized, last acting [3,7] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.8 is stuck undersized for 109s, current state undersized+peered, last acting [0] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.9 is stuck undersized for 109s, current state undersized+peered, last acting [10] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.c is stuck undersized for 109s, current state active+undersized, last acting [7,3] 2024-09-19T07:00:00.435 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.d is stuck undersized for 109s, current state active+undersized, last acting [7,1] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.e is stuck undersized for 109s, current state undersized+peered, last acting [4] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.f is stuck undersized for 109s, current state undersized+peered, last acting [7] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.10 is stuck undersized for 109s, current state active+undersized, last acting [7,10] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.11 is stuck undersized for 109s, current state active+undersized, last acting [4,1] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.1d is stuck undersized for 109s, current state active+undersized, last acting [6,1] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 3.1e is stuck undersized for 103s, current state undersized+peered, last acting [9] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.1 is stuck undersized for 109s, current state active+undersized+degraded, last acting [NONE,1,4,7] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.2 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [NONE,10,NONE,9] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.3 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [NONE,NONE,7,6] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.4 is stuck undersized for 109s, current state active+undersized+degraded, last acting [6,0,1,NONE] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.5 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [7,NONE,NONE,9] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.6 is stuck undersized for 103s, current state active+undersized+degraded, last acting [NONE,9,6,10] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.7 is stuck undersized for 109s, current state active+undersized+degraded, last acting [6,NONE,10,0] 2024-09-19T07:00:00.436 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.8 is stuck undersized for 109s, current state undersized+degraded+peered, last acting [NONE,NONE,7,1] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.9 is stuck undersized for 109s, current state active+undersized+degraded, last acting [NONE,0,3,1] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.a is stuck undersized for 109s, current state active+undersized+degraded, last acting [1,NONE,6,7] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.c is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,9,7,NONE] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.d is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,NONE,10,3] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.e is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,NONE,10,9] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.16 is stuck undersized for 109s, current state active+undersized+degraded, last acting [6,3,9,NONE] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.19 is stuck undersized for 109s, current state active+undersized+degraded, last acting [NONE,3,9,0] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.1a is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,6,NONE,3] 2024-09-19T07:00:00.437 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:00 smithi116 ceph-mon[35411]: pg 4.1b is stuck undersized for 109s, current state active+undersized+degraded, last acting [4,3,9,NONE] 2024-09-19T07:00:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:02 smithi114 ceph-mon[30820]: pgmap v635: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:02 smithi116 ceph-mon[35411]: pgmap v635: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:04 smithi114 ceph-mon[30820]: pgmap v636: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:04 smithi116 ceph-mon[35411]: pgmap v636: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:06 smithi114 ceph-mon[30820]: pgmap v637: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:06 smithi116 ceph-mon[35411]: pgmap v637: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:08 smithi114 ceph-mon[30820]: pgmap v638: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:08 smithi116 ceph-mon[35411]: pgmap v638: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:10 smithi114 ceph-mon[30820]: pgmap v639: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:10 smithi116 ceph-mon[35411]: pgmap v639: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:12 smithi114 ceph-mon[30820]: pgmap v640: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:12 smithi116 ceph-mon[35411]: pgmap v640: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:14 smithi114 ceph-mon[30820]: pgmap v641: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:14 smithi116 ceph-mon[35411]: pgmap v641: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:16 smithi114 ceph-mon[30820]: pgmap v642: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:16 smithi116 ceph-mon[35411]: pgmap v642: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:18 smithi114 ceph-mon[30820]: pgmap v643: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:18 smithi116 ceph-mon[35411]: pgmap v643: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:20 smithi114 ceph-mon[30820]: pgmap v644: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:20 smithi116 ceph-mon[35411]: pgmap v644: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:22 smithi114 ceph-mon[30820]: pgmap v645: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:22 smithi116 ceph-mon[35411]: pgmap v645: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:24 smithi114 ceph-mon[30820]: pgmap v646: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:24 smithi116 ceph-mon[35411]: pgmap v646: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:26 smithi114 ceph-mon[30820]: pgmap v647: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:26 smithi116 ceph-mon[35411]: pgmap v647: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:00:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:00:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:28 smithi114 ceph-mon[30820]: pgmap v648: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:00:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:00:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:00:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:28 smithi116 ceph-mon[35411]: pgmap v648: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:00:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:00:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:00:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:30 smithi114 ceph-mon[30820]: pgmap v649: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:30 smithi116 ceph-mon[35411]: pgmap v649: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:32 smithi116 ceph-mon[35411]: pgmap v650: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:32 smithi114 ceph-mon[30820]: pgmap v650: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:34 smithi116 ceph-mon[35411]: pgmap v651: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:34 smithi114 ceph-mon[30820]: pgmap v651: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:36 smithi114 ceph-mon[30820]: pgmap v652: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:36.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:36 smithi116 ceph-mon[35411]: pgmap v652: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:38 smithi114 ceph-mon[30820]: pgmap v653: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:38 smithi116 ceph-mon[35411]: pgmap v653: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:40.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:40 smithi114 ceph-mon[30820]: pgmap v654: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:40.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:40 smithi116 ceph-mon[35411]: pgmap v654: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:42 smithi114 ceph-mon[30820]: pgmap v655: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:42 smithi116 ceph-mon[35411]: pgmap v655: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:44 smithi114 ceph-mon[30820]: pgmap v656: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:44 smithi116 ceph-mon[35411]: pgmap v656: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:46 smithi114 ceph-mon[30820]: pgmap v657: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:46 smithi116 ceph-mon[35411]: pgmap v657: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:48 smithi114 ceph-mon[30820]: pgmap v658: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:48 smithi116 ceph-mon[35411]: pgmap v658: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:50 smithi114 ceph-mon[30820]: pgmap v659: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:50 smithi116 ceph-mon[35411]: pgmap v659: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:52 smithi114 ceph-mon[30820]: pgmap v660: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:52 smithi116 ceph-mon[35411]: pgmap v660: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:54 smithi114 ceph-mon[30820]: pgmap v661: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:54 smithi116 ceph-mon[35411]: pgmap v661: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:55 smithi114 ceph-mon[30820]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-09-19T07:00:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:55 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:00:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:55 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:00:55.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:55 smithi116 ceph-mon[35411]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-09-19T07:00:55.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:55 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:00:55.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:55 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:00:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:56 smithi114 ceph-mon[30820]: pgmap v662: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:56 smithi116 ceph-mon[35411]: pgmap v662: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:00:58 smithi114 ceph-mon[30820]: pgmap v663: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:00:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:00:58 smithi116 ceph-mon[35411]: pgmap v663: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:00 smithi114 ceph-mon[30820]: pgmap v664: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:00 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:00 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:00 smithi116 ceph-mon[35411]: pgmap v664: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:00 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:00 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:02 smithi114 ceph-mon[30820]: pgmap v665: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:02 smithi116 ceph-mon[35411]: pgmap v665: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:04 smithi114 ceph-mon[30820]: pgmap v666: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:04 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:04 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:04 smithi116 ceph-mon[35411]: pgmap v666: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:04 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:04 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:06 smithi114 ceph-mon[30820]: pgmap v667: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:06 smithi116 ceph-mon[35411]: pgmap v667: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:08 smithi114 ceph-mon[30820]: pgmap v668: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:08 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:08 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:08 smithi116 ceph-mon[35411]: pgmap v668: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:08 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:08 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:10 smithi114 ceph-mon[30820]: pgmap v669: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:10 smithi116 ceph-mon[35411]: pgmap v669: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:12 smithi114 ceph-mon[30820]: pgmap v670: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:12 smithi116 ceph-mon[35411]: pgmap v670: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:14 smithi114 ceph-mon[30820]: pgmap v671: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:14 smithi116 ceph-mon[35411]: pgmap v671: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:16 smithi114 ceph-mon[30820]: pgmap v672: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:16 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:16 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:16 smithi116 ceph-mon[35411]: pgmap v672: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:16 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:16 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:18 smithi114 ceph-mon[30820]: pgmap v673: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:18 smithi116 ceph-mon[35411]: pgmap v673: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:20 smithi114 ceph-mon[30820]: pgmap v674: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:20 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:20 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:20 smithi116 ceph-mon[35411]: pgmap v674: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:20 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:20 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:22 smithi114 ceph-mon[30820]: pgmap v675: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:22 smithi116 ceph-mon[35411]: pgmap v675: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:24 smithi114 ceph-mon[30820]: pgmap v676: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:24 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:24 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:24 smithi116 ceph-mon[35411]: pgmap v676: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:24 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:24 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:26 smithi114 ceph-mon[30820]: pgmap v677: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:26 smithi116 ceph-mon[35411]: pgmap v677: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:28 smithi114 ceph-mon[30820]: pgmap v678: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:28 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:01:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:28 smithi116 ceph-mon[35411]: pgmap v678: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:28 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:01:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:30 smithi114 ceph-mon[30820]: pgmap v679: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:30 smithi116 ceph-mon[35411]: pgmap v679: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:01:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:01:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:01:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:01:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:01:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:32 smithi114 ceph-mon[30820]: pgmap v680: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:32 smithi116 ceph-mon[35411]: pgmap v680: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:34 smithi114 ceph-mon[30820]: pgmap v681: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:34 smithi116 ceph-mon[35411]: pgmap v681: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:36 smithi114 ceph-mon[30820]: pgmap v682: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:36 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:36 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:36 smithi116 ceph-mon[35411]: pgmap v682: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:36 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:36 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:38 smithi114 ceph-mon[30820]: pgmap v683: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:38 smithi116 ceph-mon[35411]: pgmap v683: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:40 smithi116 ceph-mon[35411]: pgmap v684: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:40 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:40 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:40 smithi114 ceph-mon[30820]: pgmap v684: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:40 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:40 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:42 smithi116 ceph-mon[35411]: pgmap v685: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:42 smithi114 ceph-mon[30820]: pgmap v685: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:44 smithi114 ceph-mon[30820]: pgmap v686: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:44 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:44 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:44 smithi116 ceph-mon[35411]: pgmap v686: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:44 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:44 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:46 smithi114 ceph-mon[30820]: pgmap v687: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:46 smithi116 ceph-mon[35411]: pgmap v687: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:48 smithi114 ceph-mon[30820]: pgmap v688: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:48 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:48 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:48 smithi116 ceph-mon[35411]: pgmap v688: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:48 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:48 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:50 smithi114 ceph-mon[30820]: pgmap v689: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:50 smithi116 ceph-mon[35411]: pgmap v689: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:52 smithi114 ceph-mon[30820]: pgmap v690: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:52 smithi116 ceph-mon[35411]: pgmap v690: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:54 smithi114 ceph-mon[30820]: pgmap v691: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:54 smithi116 ceph-mon[35411]: pgmap v691: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:56 smithi114 ceph-mon[30820]: pgmap v692: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:56 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:56 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:56 smithi116 ceph-mon[35411]: pgmap v692: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:56 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:01:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:56 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:01:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:01:58 smithi114 ceph-mon[30820]: pgmap v693: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:01:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:01:58 smithi116 ceph-mon[35411]: pgmap v693: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:00 smithi114 ceph-mon[30820]: pgmap v694: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:00 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:00 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:00 smithi116 ceph-mon[35411]: pgmap v694: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:00 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:00 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:02 smithi114 ceph-mon[30820]: pgmap v695: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:02 smithi116 ceph-mon[35411]: pgmap v695: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:04 smithi114 ceph-mon[30820]: pgmap v696: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:04 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:04 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:04 smithi116 ceph-mon[35411]: pgmap v696: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:04 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:04 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:06 smithi114 ceph-mon[30820]: pgmap v697: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:06 smithi116 ceph-mon[35411]: pgmap v697: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:08 smithi114 ceph-mon[30820]: pgmap v698: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:08 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:08 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:08 smithi116 ceph-mon[35411]: pgmap v698: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:08 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:08 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:09 smithi114 ceph-mon[30820]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-09-19T07:02:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:09 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:09 smithi116 ceph-mon[35411]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-09-19T07:02:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:09 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:10.817 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:10 smithi114 ceph-mon[30820]: pgmap v699: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:10.818 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:10 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:10.818 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:10 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:10.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:10 smithi116 ceph-mon[35411]: pgmap v699: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:10.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:10 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:10.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:10 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:12 smithi114 ceph-mon[30820]: pgmap v700: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:12 smithi116 ceph-mon[35411]: pgmap v700: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:13 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:13 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:13 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:13 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:14 smithi114 ceph-mon[30820]: pgmap v701: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:14 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:14 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:14 smithi116 ceph-mon[35411]: pgmap v701: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:14 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:14 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:15 smithi114 ceph-mon[30820]: Health check update: 3 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-09-19T07:02:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:15 smithi116 ceph-mon[35411]: Health check update: 3 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-09-19T07:02:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:16 smithi114 ceph-mon[30820]: pgmap v702: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:16 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:16 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:16 smithi116 ceph-mon[35411]: pgmap v702: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:16 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:16 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:18 smithi114 ceph-mon[30820]: pgmap v703: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:18 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:18 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:18 smithi116 ceph-mon[35411]: pgmap v703: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:18 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:18 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:19 smithi114 ceph-mon[30820]: pgmap v704: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:19 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:19 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:19 smithi116 ceph-mon[35411]: pgmap v704: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:19 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:19 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:21 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:21 smithi114 ceph-mon[30820]: pgmap v705: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:21 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:21 smithi116 ceph-mon[35411]: pgmap v705: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:23 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:23 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:23 smithi114 ceph-mon[30820]: pgmap v706: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:23 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:23 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:23 smithi116 ceph-mon[35411]: pgmap v706: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:25 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:25 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:25 smithi114 ceph-mon[30820]: pgmap v707: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:25 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:25 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:25 smithi116 ceph-mon[35411]: pgmap v707: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:27 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:27 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:27 smithi114 ceph-mon[30820]: pgmap v708: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:27 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:27 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:27 smithi116 ceph-mon[35411]: pgmap v708: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:29.174 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:29.174 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:28 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:28 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:29 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:29 smithi114 ceph-mon[30820]: pgmap v709: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:29 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:29 smithi116 ceph-mon[35411]: pgmap v709: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:02:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:02:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:31 smithi114 ceph-mon[30820]: pgmap v710: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:02:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:02:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:31 smithi116 ceph-mon[35411]: pgmap v710: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:02:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:02:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:02:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:33 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:33 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:33 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:33 smithi116 ceph-mon[35411]: pgmap v711: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:33 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:33 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:33 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:33 smithi114 ceph-mon[30820]: pgmap v711: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:35 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:35 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:35 smithi114 ceph-mon[30820]: pgmap v712: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:35 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:35 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:35 smithi116 ceph-mon[35411]: pgmap v712: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:37 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:37 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:37 smithi114 ceph-mon[30820]: pgmap v713: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:37 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:37 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:37 smithi116 ceph-mon[35411]: pgmap v713: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:39 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:39 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:39 smithi114 ceph-mon[30820]: pgmap v714: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:39 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:39 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:39 smithi116 ceph-mon[35411]: pgmap v714: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:41.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:41 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:41.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:41 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:41.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:41 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:41.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:41 smithi114 ceph-mon[30820]: pgmap v715: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:41.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:41 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:41 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:41 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:41.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:41 smithi116 ceph-mon[35411]: pgmap v715: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:43 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:43 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:43 smithi114 ceph-mon[30820]: pgmap v716: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:43 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:43 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:43 smithi116 ceph-mon[35411]: pgmap v716: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:45 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:45 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:45 smithi114 ceph-mon[30820]: pgmap v717: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:45 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:45 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:45 smithi116 ceph-mon[35411]: pgmap v717: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:47 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:47 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:47 smithi114 ceph-mon[30820]: pgmap v718: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:47 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:47 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:47 smithi116 ceph-mon[35411]: pgmap v718: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:48 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:48 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:48 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:48 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:49 smithi114 ceph-mon[30820]: pgmap v719: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:49 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:49 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:49 smithi116 ceph-mon[35411]: pgmap v719: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:49 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:49 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:51 smithi114 ceph-mon[30820]: pgmap v720: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:51 smithi116 ceph-mon[35411]: pgmap v720: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:52 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:52 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:52 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:02:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:52 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:53 smithi114 ceph-mon[30820]: pgmap v721: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:53 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:53 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:53 smithi116 ceph-mon[35411]: pgmap v721: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:53 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:53 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:55 smithi114 ceph-mon[30820]: pgmap v722: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:55 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:55 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:55 smithi116 ceph-mon[35411]: pgmap v722: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:55 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:02:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:55 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:57 smithi114 ceph-mon[30820]: pgmap v723: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:57 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:57 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:02:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:57 smithi116 ceph-mon[35411]: pgmap v723: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:02:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:57 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:02:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:57 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:59 smithi114 ceph-mon[30820]: pgmap v724: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:59 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:02:59 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:59 smithi116 ceph-mon[35411]: pgmap v724: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:59 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:02:59 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:01 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:01 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:01 smithi114 ceph-mon[30820]: pgmap v725: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:01 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:01 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:01 smithi116 ceph-mon[35411]: pgmap v725: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:02 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:02 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:02 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:02 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:03 smithi114 ceph-mon[30820]: pgmap v726: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:03 smithi116 ceph-mon[35411]: pgmap v726: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:04.902 INFO:tasks.workunit.client.0.smithi114.stdout:stopping iogen 2024-09-19T07:03:04.902 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo 'stopping iogen' 2024-09-19T07:03:04.902 INFO:tasks.workunit.client.0.smithi114.stderr:+ ./iogen -k 2024-09-19T07:03:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:04 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:04 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:04 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:04 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:05 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:05 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:05 smithi114 ceph-mon[30820]: pgmap v727: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:05 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:05 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:05 smithi116 ceph-mon[35411]: pgmap v727: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:07 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:07 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:07 smithi114 ceph-mon[30820]: pgmap v728: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:07 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:07 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:07 smithi116 ceph-mon[35411]: pgmap v728: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:09 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:09 smithi114 ceph-mon[30820]: pgmap v729: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:09 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:09 smithi116 ceph-mon[35411]: pgmap v729: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:11 smithi114 ceph-mon[30820]: pgmap v730: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:11 smithi116 ceph-mon[35411]: pgmap v730: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:12 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:12 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:12 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:12 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:13 smithi114 ceph-mon[30820]: pgmap v731: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:13 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:13 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:13 smithi116 ceph-mon[35411]: pgmap v731: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:13 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:13 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:15 smithi114 ceph-mon[30820]: pgmap v732: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:15 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:15 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:15 smithi116 ceph-mon[35411]: pgmap v732: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:15 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:15 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:17 smithi114 ceph-mon[30820]: pgmap v733: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:17 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:17 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:17 smithi116 ceph-mon[35411]: pgmap v733: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:17 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:17 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:19 smithi114 ceph-mon[30820]: pgmap v734: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:19 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:19 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:19 smithi116 ceph-mon[35411]: pgmap v734: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:19 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:19 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:20 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:20 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:20 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:20 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:21 smithi114 ceph-mon[30820]: pgmap v735: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:21 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:21 smithi116 ceph-mon[35411]: pgmap v735: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:21 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:23 smithi114 ceph-mon[30820]: pgmap v736: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:23 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:23 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:23 smithi116 ceph-mon[35411]: pgmap v736: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:23 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:23 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:24 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:24 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:24 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:24 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:25 smithi116 ceph-mon[35411]: pgmap v737: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:25 smithi114 ceph-mon[30820]: pgmap v737: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:27 smithi114 ceph-mon[30820]: pgmap v738: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:27 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:27 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:27 smithi116 ceph-mon[35411]: pgmap v738: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:27 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:27 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:28 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:28 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:29 smithi114 ceph-mon[30820]: pgmap v739: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:29 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:29 smithi116 ceph-mon[35411]: pgmap v739: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:29 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:31 smithi114 ceph-mon[30820]: pgmap v740: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:03:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:31 smithi116 ceph-mon[35411]: pgmap v740: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:03:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:32 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:32 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:32 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:32 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:33 smithi114 ceph-mon[30820]: pgmap v741: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:33 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:33 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:33 smithi116 ceph-mon[35411]: pgmap v741: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:33 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:33 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:03:34.999+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:03:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:03:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:03:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:03:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:03:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:03:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:03:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:03:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:03:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:03:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:03:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-mon[30820]: pgmap v742: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:03:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:03:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:03:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-mon[30820]: pgmap v743: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:36.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:35 smithi116 ceph-mon[35411]: pgmap v742: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:35 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:35 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:35 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:03:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:03:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:03:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:03:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:03:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:03:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:03:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:35 smithi116 ceph-mon[35411]: pgmap v743: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:03:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:36 smithi114 ceph-mon[30820]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2024-09-19T07:03:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:36 smithi116 ceph-mon[35411]: Health check failed: 1 hosts fail cephadm check (CEPHADM_HOST_CHECK_FAILED) 2024-09-19T07:03:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:37 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:37 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:37 smithi114 ceph-mon[30820]: pgmap v744: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:37 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:37 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:37 smithi116 ceph-mon[35411]: pgmap v744: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:39 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:39 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:39 smithi114 ceph-mon[30820]: pgmap v745: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:39 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:39 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:39 smithi116 ceph-mon[35411]: pgmap v745: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:41.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:40 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:41.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:40 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:41.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:40 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:41.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:40 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:42.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:42 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:42.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:42 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:42.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:42 smithi114 ceph-mon[30820]: pgmap v746: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:42 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:42 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:42 smithi116 ceph-mon[35411]: pgmap v746: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:44 smithi114 ceph-mon[30820]: pgmap v747: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:44 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:44 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:44 smithi116 ceph-mon[35411]: pgmap v747: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:44 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:44 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:45 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:45 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:45 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:45 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:46 smithi114 ceph-mon[30820]: pgmap v748: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:46 smithi116 ceph-mon[35411]: pgmap v748: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:48 smithi114 ceph-mon[30820]: pgmap v749: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:48 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:48 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:48 smithi116 ceph-mon[35411]: pgmap v749: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:48 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:48 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:49.229 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:49 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:49.229 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:49 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:49 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:49 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:50 smithi114 ceph-mon[30820]: pgmap v750: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:50 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:50 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:50 smithi116 ceph-mon[35411]: pgmap v750: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:50 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:50 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:52 smithi114 ceph-mon[30820]: pgmap v751: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:52 smithi116 ceph-mon[35411]: pgmap v751: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:53 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:53 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:53 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:03:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:53 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:54 smithi114 ceph-mon[30820]: pgmap v752: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:54 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:54 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:54 smithi116 ceph-mon[35411]: pgmap v752: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:54 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:54 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:56 smithi114 ceph-mon[30820]: pgmap v753: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:56 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:56 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:56 smithi116 ceph-mon[35411]: pgmap v753: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:56 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:03:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:56 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:58 smithi114 ceph-mon[30820]: pgmap v754: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:58 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:03:58 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:03:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:58 smithi116 ceph-mon[35411]: pgmap v754: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:03:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:58 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:03:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:03:58 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:00 smithi114 ceph-mon[30820]: pgmap v755: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:00 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:00 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:00 smithi116 ceph-mon[35411]: pgmap v755: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:00 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:00 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:01 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:01 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:01 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:01 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:02 smithi114 ceph-mon[30820]: pgmap v756: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:02 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:02 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:02 smithi116 ceph-mon[35411]: pgmap v756: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:02 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:02 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:04 smithi114 ceph-mon[30820]: pgmap v757: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:04 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:04 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:04 smithi116 ceph-mon[35411]: pgmap v757: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:04 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:04 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:05 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:05 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:05 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:05 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:06 smithi114 ceph-mon[30820]: pgmap v758: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:06 smithi116 ceph-mon[35411]: pgmap v758: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:08 smithi114 ceph-mon[30820]: pgmap v759: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:08 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:08 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:08 smithi116 ceph-mon[35411]: pgmap v759: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:08 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:08 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:09 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:09 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:10 smithi114 ceph-mon[30820]: pgmap v760: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:10 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:10 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:10 smithi116 ceph-mon[35411]: pgmap v760: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:10 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:10 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:12 smithi114 ceph-mon[30820]: pgmap v761: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:12 smithi116 ceph-mon[35411]: pgmap v761: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:13 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:13 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:13 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:13 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:14 smithi114 ceph-mon[30820]: pgmap v762: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:14 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:14 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:14 smithi116 ceph-mon[35411]: pgmap v762: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:14 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:14 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:16 smithi114 ceph-mon[30820]: pgmap v763: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:16 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:16 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:16 smithi116 ceph-mon[35411]: pgmap v763: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:16 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:16 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:18 smithi116 ceph-mon[35411]: pgmap v764: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:18 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:18 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:18 smithi114 ceph-mon[30820]: pgmap v764: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:18 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:18 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:20 smithi114 ceph-mon[30820]: pgmap v765: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:20 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:20 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:20 smithi116 ceph-mon[35411]: pgmap v765: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:20 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:20 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:21 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:21 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:22 smithi114 ceph-mon[30820]: pgmap v766: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:22 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:22 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:22 smithi116 ceph-mon[35411]: pgmap v766: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:22 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:22 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:24 smithi114 ceph-mon[30820]: pgmap v767: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:24 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:24 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:24 smithi116 ceph-mon[35411]: pgmap v767: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:24 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:24 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:25 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:25 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:25 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:25 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:26 smithi114 ceph-mon[30820]: pgmap v768: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:26 smithi116 ceph-mon[35411]: pgmap v768: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:28 smithi114 ceph-mon[30820]: pgmap v769: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:28 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:28 smithi116 ceph-mon[35411]: pgmap v769: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:28 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:29 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:29 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:30 smithi114 ceph-mon[30820]: pgmap v770: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:30 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:30 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:30 smithi116 ceph-mon[35411]: pgmap v770: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:30 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:30 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:32 smithi114 ceph-mon[30820]: pgmap v771: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:32 smithi116 ceph-mon[35411]: pgmap v771: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:33 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:33 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:33 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:33 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:34 smithi114 ceph-mon[30820]: pgmap v772: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:34 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:34 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:34 smithi116 ceph-mon[35411]: pgmap v772: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:34 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:34 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:04:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:04:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:36 smithi114 ceph-mon[30820]: pgmap v773: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:36 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:36 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:36 smithi116 ceph-mon[35411]: pgmap v773: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:36 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:36 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-mon[30820]: pgmap v774: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:04:38.356+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:04:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:04:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:04:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:04:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:04:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:04:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:04:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:04:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:04:38.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:04:38.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:04:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:04:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:38 smithi116 ceph-mon[35411]: pgmap v774: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:38 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:38 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:04:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:04:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:04:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:39 smithi114 ceph-mon[30820]: pgmap v775: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:39 smithi114 ceph-mon[30820]: pgmap v776: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:04:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:04:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:04:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:04:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:04:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:39 smithi116 ceph-mon[35411]: pgmap v775: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:39 smithi116 ceph-mon[35411]: pgmap v776: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:04:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:40 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:40 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:40 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:40 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:41 smithi114 ceph-mon[30820]: pgmap v777: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:41 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:41 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:41 smithi116 ceph-mon[35411]: pgmap v777: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:41 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:41 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:42 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:42 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:42 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:42 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:43 smithi114 ceph-mon[30820]: pgmap v778: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:43 smithi116 ceph-mon[35411]: pgmap v778: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:44 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:44 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:44 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:44 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:45 smithi114 ceph-mon[30820]: pgmap v779: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:45 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:45 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:45 smithi116 ceph-mon[35411]: pgmap v779: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:45 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:45 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:47 smithi114 ceph-mon[30820]: pgmap v780: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:47 smithi116 ceph-mon[35411]: pgmap v780: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:48 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:48 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:48 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:48 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:49 smithi114 ceph-mon[30820]: pgmap v781: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:49 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:49 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:49 smithi116 ceph-mon[35411]: pgmap v781: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:49 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:49 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:50 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:50 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:50 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:50 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:51 smithi114 ceph-mon[30820]: pgmap v782: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:51 smithi116 ceph-mon[35411]: pgmap v782: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:53 smithi114 ceph-mon[30820]: pgmap v783: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:53 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:53 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:53 smithi116 ceph-mon[35411]: pgmap v783: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:53 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:04:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:53 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:54 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:54 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:54 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:54 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:55 smithi114 ceph-mon[30820]: pgmap v784: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:55 smithi116 ceph-mon[35411]: pgmap v784: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:56 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:56 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:56 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:04:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:56 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:57 smithi114 ceph-mon[30820]: pgmap v785: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:57 smithi116 ceph-mon[35411]: pgmap v785: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:58 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:58 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:58 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:04:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:58 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:04:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:04:59 smithi114 ceph-mon[30820]: pgmap v786: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:04:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:04:59 smithi116 ceph-mon[35411]: pgmap v786: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:00 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:00 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:00 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:00 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:01 smithi114 ceph-mon[30820]: pgmap v787: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:01 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:01 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:01 smithi116 ceph-mon[35411]: pgmap v787: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:01 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:01 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:02 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:02 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:02 smithi114 ceph-mon[30820]: pgmap v788: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:02 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:02 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:02 smithi116 ceph-mon[35411]: pgmap v788: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:04 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:04 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:04 smithi114 ceph-mon[30820]: pgmap v789: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:04 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:04 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:04 smithi116 ceph-mon[35411]: pgmap v789: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:05 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:05 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:05 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:05 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:06 smithi114 ceph-mon[30820]: pgmap v790: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:06 smithi116 ceph-mon[35411]: pgmap v790: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:08 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:08 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:08 smithi114 ceph-mon[30820]: pgmap v791: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:08 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:08 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:08 smithi116 ceph-mon[35411]: pgmap v791: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:09 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:09 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:10 smithi114 ceph-mon[30820]: pgmap v792: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:10 smithi116 ceph-mon[35411]: pgmap v792: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:12 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:12 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:12 smithi114 ceph-mon[30820]: pgmap v793: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:12 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:12 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:12 smithi116 ceph-mon[35411]: pgmap v793: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:13 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:13 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:13 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:13 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:14 smithi114 ceph-mon[30820]: pgmap v794: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:14 smithi116 ceph-mon[35411]: pgmap v794: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:15 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:15 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:15 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:15 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:16 smithi114 ceph-mon[30820]: pgmap v795: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:16 smithi116 ceph-mon[35411]: pgmap v795: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:17 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:17 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:17 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:17 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:18 smithi114 ceph-mon[30820]: pgmap v796: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:18 smithi116 ceph-mon[35411]: pgmap v796: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:19 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:19 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:19 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:19 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:20 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:20 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:20 smithi114 ceph-mon[30820]: pgmap v797: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:20 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:20 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:20 smithi116 ceph-mon[35411]: pgmap v797: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:21 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:21 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:22 smithi114 ceph-mon[30820]: pgmap v798: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:22 smithi116 ceph-mon[35411]: pgmap v798: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:23 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:23 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:23 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:23 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:24 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:24 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:24 smithi114 ceph-mon[30820]: pgmap v799: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:24 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:24 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:24 smithi116 ceph-mon[35411]: pgmap v799: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:27 smithi114 ceph-mon[30820]: pgmap v800: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:27 smithi116 ceph-mon[35411]: pgmap v800: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:28.946 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:28 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:28.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:28.946 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:28 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:29 smithi114 ceph-mon[30820]: pgmap v801: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:29 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:29 smithi116 ceph-mon[35411]: pgmap v801: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:29 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:30 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:30 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:30 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:30 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:31 smithi114 ceph-mon[30820]: pgmap v802: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:31 smithi116 ceph-mon[35411]: pgmap v802: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:33 smithi114 ceph-mon[30820]: pgmap v803: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:33 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:33 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:33 smithi116 ceph-mon[35411]: pgmap v803: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:33 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:33 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:34 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:34 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:34 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:34 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:35 smithi114 ceph-mon[30820]: pgmap v804: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:35 smithi116 ceph-mon[35411]: pgmap v804: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:36 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:36 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:36 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:36 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:37 smithi114 ceph-mon[30820]: pgmap v805: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:37 smithi116 ceph-mon[35411]: pgmap v805: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:38 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:38 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:05:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:38 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:38 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:05:39.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:05:39.220+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:05:39.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:05:39.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:05:39.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:05:39.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:05:39.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:05:39.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:05:39.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:05:39.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:05:39.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:05:39.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:05:39.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:05:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-mon[30820]: pgmap v806: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:05:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:05:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:05:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:39 smithi116 ceph-mon[35411]: pgmap v806: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:05:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:05:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:05:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:40 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:40 smithi114 ceph-mon[30820]: pgmap v807: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:40 smithi114 ceph-mon[30820]: pgmap v808: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:40 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:40 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:40 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:05:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:05:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:05:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:40 smithi116 ceph-mon[35411]: pgmap v807: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:40 smithi116 ceph-mon[35411]: pgmap v808: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:40 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:40 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:41 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:41 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:41 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:41 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:42 smithi114 ceph-mon[30820]: pgmap v809: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:42 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:42 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:42 smithi116 ceph-mon[35411]: pgmap v809: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:42 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:42 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:44 smithi114 ceph-mon[30820]: pgmap v810: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:44 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:44 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:44 smithi116 ceph-mon[35411]: pgmap v810: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:44 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:44 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:45 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:45 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:45 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:45 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:46 smithi114 ceph-mon[30820]: pgmap v811: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:46 smithi116 ceph-mon[35411]: pgmap v811: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:48 smithi114 ceph-mon[30820]: pgmap v812: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:48 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:48 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:48 smithi116 ceph-mon[35411]: pgmap v812: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:48 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:48 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:49 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:49 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:49 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:49 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:50 smithi114 ceph-mon[30820]: pgmap v813: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:50 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:50 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:50 smithi116 ceph-mon[35411]: pgmap v813: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:50 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:50 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:51 smithi114 ceph-mon[30820]: pgmap v814: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:51 smithi116 ceph-mon[35411]: pgmap v814: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:53 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:53 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:53 smithi114 ceph-mon[30820]: pgmap v815: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:53 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:05:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:53 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:53 smithi116 ceph-mon[35411]: pgmap v815: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:54 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:54 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:54 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:54 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:55 smithi114 ceph-mon[30820]: pgmap v816: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:55 smithi116 ceph-mon[35411]: pgmap v816: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:56 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:56 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:56 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:05:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:56 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:57 smithi114 ceph-mon[30820]: pgmap v817: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:57 smithi116 ceph-mon[35411]: pgmap v817: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:58 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:58 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:58 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:05:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:58 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:05:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:05:59 smithi114 ceph-mon[30820]: pgmap v818: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:05:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:05:59 smithi116 ceph-mon[35411]: pgmap v818: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:00 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:00 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:00 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:00 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:01 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:01 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:01 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:01 smithi114 ceph-mon[30820]: pgmap v819: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:01 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:01 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:01.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:01 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:01.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:01 smithi116 ceph-mon[35411]: pgmap v819: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:03 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:03 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:03 smithi114 ceph-mon[30820]: pgmap v820: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:03 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:03 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:03 smithi116 ceph-mon[35411]: pgmap v820: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:05 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:05 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:05 smithi114 ceph-mon[30820]: pgmap v821: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:05 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:05 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:05 smithi116 ceph-mon[35411]: pgmap v821: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:07 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:07 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:07 smithi114 ceph-mon[30820]: pgmap v822: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:07 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:07 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:07 smithi116 ceph-mon[35411]: pgmap v822: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:08 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:08 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:08 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:08 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:09 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:09 smithi114 ceph-mon[30820]: pgmap v823: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:09 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:09 smithi116 ceph-mon[35411]: pgmap v823: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:12 smithi114 ceph-mon[30820]: pgmap v824: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:12 smithi116 ceph-mon[35411]: pgmap v824: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:13 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:13 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:13 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:13 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:14.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:14 smithi114 ceph-mon[30820]: pgmap v825: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:14.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:14 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:14.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:14 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:14.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:14 smithi116 ceph-mon[35411]: pgmap v825: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:14.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:14 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:14.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:14 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:16 smithi114 ceph-mon[30820]: pgmap v826: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:16 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:16 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:16 smithi116 ceph-mon[35411]: pgmap v826: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:16 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:16 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:18 smithi114 ceph-mon[30820]: pgmap v827: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:18 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:18 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:18 smithi116 ceph-mon[35411]: pgmap v827: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:18 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:18 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:20 smithi114 ceph-mon[30820]: pgmap v828: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:20 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:20 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:20 smithi116 ceph-mon[35411]: pgmap v828: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:20 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:20 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:21 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:21 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:22 smithi114 ceph-mon[30820]: pgmap v829: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:22 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:22 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:22 smithi116 ceph-mon[35411]: pgmap v829: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:22 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:22 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:24 smithi114 ceph-mon[30820]: pgmap v830: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:24 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:24 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:24 smithi116 ceph-mon[35411]: pgmap v830: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:24 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:24 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:25 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:25 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:25 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:25 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:26 smithi114 ceph-mon[30820]: pgmap v831: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:26 smithi116 ceph-mon[35411]: pgmap v831: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:28 smithi114 ceph-mon[30820]: pgmap v832: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:28 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:28 smithi116 ceph-mon[35411]: pgmap v832: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:28 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:29 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:29 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:30 smithi114 ceph-mon[30820]: pgmap v833: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:30 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:30 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:30 smithi116 ceph-mon[35411]: pgmap v833: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:30 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:30 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:32 smithi114 ceph-mon[30820]: pgmap v834: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:32 smithi116 ceph-mon[35411]: pgmap v834: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:33 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:33 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:33 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:33 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:34 smithi114 ceph-mon[30820]: pgmap v835: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:34 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:34 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:34 smithi116 ceph-mon[35411]: pgmap v835: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:34 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:34 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:36 smithi116 ceph-mon[35411]: pgmap v836: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:36 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:36 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:36 smithi114 ceph-mon[30820]: pgmap v836: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:36 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:36 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:38 smithi116 ceph-mon[35411]: pgmap v837: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:38 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:38 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:38 smithi114 ceph-mon[30820]: pgmap v837: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:38 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:38 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:06:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:06:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-mon[30820]: pgmap v838: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:06:40.661+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:06:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:06:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:06:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:06:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:40 smithi116 ceph-mon[35411]: pgmap v838: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:40 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:40 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:41 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:41 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:06:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:06:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:06:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:41 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:41 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:06:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:06:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:06:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:42 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:06:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:42 smithi114 ceph-mon[30820]: pgmap v839: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:42 smithi114 ceph-mon[30820]: pgmap v840: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:42 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:42 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:42 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:06:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:06:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:06:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:06:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:42 smithi116 ceph-mon[35411]: pgmap v839: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:42 smithi116 ceph-mon[35411]: pgmap v840: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:42 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:42 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:44.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:44 smithi114 ceph-mon[30820]: pgmap v841: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:44.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:44 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:44.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:44 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:44.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:44 smithi116 ceph-mon[35411]: pgmap v841: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:44.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:44 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:44.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:44 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:45 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:45 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:45 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:45 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:46 smithi114 ceph-mon[30820]: pgmap v842: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:46 smithi116 ceph-mon[35411]: pgmap v842: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:48 smithi114 ceph-mon[30820]: pgmap v843: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:48 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:48 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:48 smithi116 ceph-mon[35411]: pgmap v843: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:48 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:48 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:49 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:49 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:49 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:49 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:50 smithi114 ceph-mon[30820]: pgmap v844: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:50 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:50 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:50 smithi116 ceph-mon[35411]: pgmap v844: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:50 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:50 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:52 smithi114 ceph-mon[30820]: pgmap v845: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:52 smithi116 ceph-mon[35411]: pgmap v845: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:53 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:53 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:53 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:06:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:53 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:54 smithi114 ceph-mon[30820]: pgmap v846: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:54 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:54 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:54 smithi116 ceph-mon[35411]: pgmap v846: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:54 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:54 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:56 smithi114 ceph-mon[30820]: pgmap v847: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:56 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:56 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:56 smithi116 ceph-mon[35411]: pgmap v847: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:56 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:06:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:56 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:58 smithi114 ceph-mon[30820]: pgmap v848: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:58 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:06:58 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:06:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:58 smithi116 ceph-mon[35411]: pgmap v848: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:06:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:58 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:06:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:06:58 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:00 smithi114 ceph-mon[30820]: pgmap v849: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:00 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:00 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:00 smithi116 ceph-mon[35411]: pgmap v849: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:00 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:00 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:01 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:01 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:01 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:01 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:02 smithi114 ceph-mon[30820]: pgmap v850: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:02 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:02 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:02 smithi116 ceph-mon[35411]: pgmap v850: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:02 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:02 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:04 smithi114 ceph-mon[30820]: pgmap v851: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:04 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:04 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:04 smithi116 ceph-mon[35411]: pgmap v851: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:04 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:04 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:05 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:05 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:05 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:05 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:06 smithi114 ceph-mon[30820]: pgmap v852: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:06 smithi116 ceph-mon[35411]: pgmap v852: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:08 smithi114 ceph-mon[30820]: pgmap v853: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:08 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:08 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:08 smithi116 ceph-mon[35411]: pgmap v853: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:08 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:08 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:09 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:09 smithi114 ceph-mon[30820]: pgmap v854: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:09 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:09 smithi116 ceph-mon[35411]: pgmap v854: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:10 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:10 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:10 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:10 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:11 smithi114 ceph-mon[30820]: pgmap v855: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:11 smithi116 ceph-mon[35411]: pgmap v855: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:13 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:13 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:13 smithi114 ceph-mon[30820]: pgmap v856: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:13 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:13 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:13 smithi116 ceph-mon[35411]: pgmap v856: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:14 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:14 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:14 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:14 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:15 smithi114 ceph-mon[30820]: pgmap v857: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:15 smithi116 ceph-mon[35411]: pgmap v857: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:16 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:16 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:16 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:16 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:17 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:17 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:17 smithi114 ceph-mon[30820]: pgmap v858: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:17 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:17 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:17 smithi116 ceph-mon[35411]: pgmap v858: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:19 smithi114 ceph-mon[30820]: pgmap v859: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:19 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:19 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:19 smithi116 ceph-mon[35411]: pgmap v859: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:19 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:19 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:21 smithi114 ceph-mon[30820]: pgmap v860: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:21 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:21 smithi116 ceph-mon[35411]: pgmap v860: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:21 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:23 smithi114 ceph-mon[30820]: pgmap v861: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:23 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:23 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:23 smithi116 ceph-mon[35411]: pgmap v861: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:23 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:23 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:25 smithi114 ceph-mon[30820]: pgmap v862: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:25 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:25 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:25 smithi116 ceph-mon[35411]: pgmap v862: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:25 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:25 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:27 smithi114 ceph-mon[30820]: pgmap v863: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:27 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:27 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:27 smithi116 ceph-mon[35411]: pgmap v863: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:27 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:27 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:28 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:28 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:29 smithi114 ceph-mon[30820]: pgmap v864: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:29 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:29 smithi116 ceph-mon[35411]: pgmap v864: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:29 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:31 smithi114 ceph-mon[30820]: pgmap v865: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:31 smithi116 ceph-mon[35411]: pgmap v865: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:32 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:32 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:33.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:32 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:33.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:32 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:33 smithi114 ceph-mon[30820]: pgmap v866: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:33 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:33 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:33 smithi116 ceph-mon[35411]: pgmap v866: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:33 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:33 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:35 smithi114 ceph-mon[30820]: pgmap v867: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:35 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:35 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:35 smithi116 ceph-mon[35411]: pgmap v867: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:35 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:35 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:37 smithi114 ceph-mon[30820]: pgmap v868: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:37 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:37 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:37 smithi116 ceph-mon[35411]: pgmap v868: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:37 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:37 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:39 smithi114 ceph-mon[30820]: pgmap v869: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:39 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:39 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:39 smithi116 ceph-mon[35411]: pgmap v869: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:39 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:39 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:41 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:41 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:41 smithi114 ceph-mon[30820]: pgmap v870: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:07:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:41 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:41 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:41 smithi116 ceph-mon[35411]: pgmap v870: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:07:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-mon[30820]: pgmap v871: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:44.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:07:43.957+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:07:44.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:07:44.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:07:44.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:07:44.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:07:44.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:07:44.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:07:44.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:07:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:07:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:07:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:43 smithi116 ceph-mon[35411]: pgmap v871: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:43 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:43 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:44 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:07:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:07:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:07:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:44 smithi114 ceph-mon[30820]: pgmap v872: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:44 smithi114 ceph-mon[30820]: pgmap v873: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:07:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:44 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:07:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:07:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:07:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:07:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:07:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:07:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:44 smithi116 ceph-mon[35411]: pgmap v872: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:44 smithi116 ceph-mon[35411]: pgmap v873: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:07:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:45 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:45 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:45 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:45 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:45 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:45 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:46 smithi114 ceph-mon[30820]: pgmap v874: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:46 smithi116 ceph-mon[35411]: pgmap v874: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:47 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:47 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:47 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:47 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:48 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:48 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:48 smithi114 ceph-mon[30820]: pgmap v875: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:48 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:48 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:48 smithi116 ceph-mon[35411]: pgmap v875: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:49 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:49 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:49 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:49 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:50 smithi114 ceph-mon[30820]: pgmap v876: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:50 smithi116 ceph-mon[35411]: pgmap v876: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:52 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:52 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:52 smithi114 ceph-mon[30820]: pgmap v877: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:52 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:07:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:52 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:52 smithi116 ceph-mon[35411]: pgmap v877: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:53 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:53 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:53 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:53 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:54 smithi114 ceph-mon[30820]: pgmap v878: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:54 smithi116 ceph-mon[35411]: pgmap v878: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:55 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:55 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:55 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:07:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:55 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:56 smithi114 ceph-mon[30820]: pgmap v879: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:56 smithi116 ceph-mon[35411]: pgmap v879: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:57 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:57 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:57 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:07:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:57 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:07:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:58 smithi114 ceph-mon[30820]: pgmap v880: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:07:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:58 smithi116 ceph-mon[35411]: pgmap v880: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:59 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:07:59 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:59 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:07:59 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:00 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:00 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:00 smithi114 ceph-mon[30820]: pgmap v881: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:00 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:00 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:00 smithi116 ceph-mon[35411]: pgmap v881: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:01 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:01 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:01 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:01 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:02 smithi114 ceph-mon[30820]: pgmap v882: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:02 smithi116 ceph-mon[35411]: pgmap v882: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:03 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:03 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:03 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:03 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:04 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:04 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:04 smithi114 ceph-mon[30820]: pgmap v883: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:04 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:04 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:04 smithi116 ceph-mon[35411]: pgmap v883: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:07 smithi114 ceph-mon[30820]: pgmap v884: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:07 smithi116 ceph-mon[35411]: pgmap v884: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:08 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:08 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:08 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:08 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:09 smithi114 ceph-mon[30820]: pgmap v885: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:09 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:09 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:09 smithi116 ceph-mon[35411]: pgmap v885: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:09 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:09 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: Marking osd.2 out (has been down for 601 seconds) 2024-09-19T07:08:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: Marking osd.5 out (has been down for 601 seconds) 2024-09-19T07:08:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: Marking osd.8 out (has been down for 601 seconds) 2024-09-19T07:08:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: Marking osd.11 out (has been down for 601 seconds) 2024-09-19T07:08:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: Health check cleared: OSD_DOWN (was: 4 osds down) 2024-09-19T07:08:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: Health check cleared: OSD_HOST_DOWN (was: 1 host (4 osds) down) 2024-09-19T07:08:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:10 smithi114 ceph-mon[30820]: osdmap e100: 12 total, 8 up, 8 in 2024-09-19T07:08:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: Marking osd.2 out (has been down for 601 seconds) 2024-09-19T07:08:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: Marking osd.5 out (has been down for 601 seconds) 2024-09-19T07:08:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: Marking osd.8 out (has been down for 601 seconds) 2024-09-19T07:08:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: Marking osd.11 out (has been down for 601 seconds) 2024-09-19T07:08:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: Health check cleared: OSD_DOWN (was: 4 osds down) 2024-09-19T07:08:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: Health check cleared: OSD_HOST_DOWN (was: 1 host (4 osds) down) 2024-09-19T07:08:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:10 smithi116 ceph-mon[35411]: osdmap e100: 12 total, 8 up, 8 in 2024-09-19T07:08:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:11 smithi114 ceph-mon[30820]: pgmap v887: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:11 smithi114 ceph-mon[30820]: osdmap e101: 12 total, 8 up, 8 in 2024-09-19T07:08:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:11 smithi116 ceph-mon[35411]: pgmap v887: 129 pgs: 15 undersized+peered, 30 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 34 active+undersized+degraded, 31 active+clean; 10 GiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 3 op/s; 2627/10620 objects degraded (24.736%) 2024-09-19T07:08:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:11 smithi116 ceph-mon[35411]: osdmap e101: 12 total, 8 up, 8 in 2024-09-19T07:08:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:12 smithi116 ceph-mon[35411]: osdmap e102: 12 total, 8 up, 8 in 2024-09-19T07:08:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:12 smithi116 ceph-mon[35411]: pgmap v890: 129 pgs: 2 unknown, 14 peering, 12 undersized+peered, 28 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 25 active+undersized+degraded, 29 active+clean; 9.0 GiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 2348/9632 objects degraded (24.377%) 2024-09-19T07:08:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:12 smithi114 ceph-mon[30820]: osdmap e102: 12 total, 8 up, 8 in 2024-09-19T07:08:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:12 smithi114 ceph-mon[30820]: pgmap v890: 129 pgs: 2 unknown, 14 peering, 12 undersized+peered, 28 active+undersized, 3 down, 2 stale+active+clean, 14 undersized+degraded+peered, 25 active+undersized+degraded, 29 active+clean; 9.0 GiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s; 2348/9632 objects degraded (24.377%) 2024-09-19T07:08:14.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:13 smithi114 ceph-mon[30820]: Health check update: Reduced data availability: 32 pgs inactive, 3 pgs down, 7 pgs peering, 2 pgs stale (PG_AVAILABILITY) 2024-09-19T07:08:14.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:13 smithi114 ceph-mon[30820]: Health check update: Degraded data redundancy: 2348/9632 objects degraded (24.377%), 39 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-09-19T07:08:14.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:13 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:14.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:13 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:14.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:13 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:14.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:13 smithi114 ceph-mon[30820]: osdmap e103: 12 total, 8 up, 8 in 2024-09-19T07:08:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:13 smithi116 ceph-mon[35411]: Health check update: Reduced data availability: 32 pgs inactive, 3 pgs down, 7 pgs peering, 2 pgs stale (PG_AVAILABILITY) 2024-09-19T07:08:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:13 smithi116 ceph-mon[35411]: Health check update: Degraded data redundancy: 2348/9632 objects degraded (24.377%), 39 pgs degraded, 79 pgs undersized (PG_DEGRADED) 2024-09-19T07:08:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:13 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:13 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:13 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:13 smithi116 ceph-mon[35411]: osdmap e103: 12 total, 8 up, 8 in 2024-09-19T07:08:15.079 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:14 smithi116 ceph-mon[35411]: osdmap e104: 12 total, 8 up, 8 in 2024-09-19T07:08:15.079 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:14 smithi116 ceph-mon[35411]: pgmap v893: 129 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 14 peering, 2 active+recovering+undersized+degraded+remapped, 2 undersized+peered, 9 active+undersized, 40 active+clean+wait, 3 down, 8 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 5 undersized+degraded+peered, 8 active+undersized+degraded, 29 active+clean; 9.4 GiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 2346/9980 objects degraded (23.507%); 58 MiB/s, 2 keys/s, 23 objects/s recovering 2024-09-19T07:08:15.079 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:14 smithi116 ceph-mon[35411]: osdmap e105: 12 total, 8 up, 8 in 2024-09-19T07:08:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:14 smithi114 ceph-mon[30820]: osdmap e104: 12 total, 8 up, 8 in 2024-09-19T07:08:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:14 smithi114 ceph-mon[30820]: pgmap v893: 129 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 14 peering, 2 active+recovering+undersized+degraded+remapped, 2 undersized+peered, 9 active+undersized, 40 active+clean+wait, 3 down, 8 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 5 undersized+degraded+peered, 8 active+undersized+degraded, 29 active+clean; 9.4 GiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 2346/9980 objects degraded (23.507%); 58 MiB/s, 2 keys/s, 23 objects/s recovering 2024-09-19T07:08:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:14 smithi114 ceph-mon[30820]: osdmap e105: 12 total, 8 up, 8 in 2024-09-19T07:08:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:16 smithi114 ceph-mon[30820]: osdmap e106: 12 total, 8 up, 8 in 2024-09-19T07:08:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:16 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:16 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:16 smithi114 ceph-mon[30820]: pgmap v896: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+degraded+remapped, 68 active+clean+wait, 3 down, 12 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 31 active+clean; 9.4 GiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 669 MiB/s wr, 2.40k op/s; 2109/9980 objects degraded (21.132%); 208 MiB/s, 12 keys/s, 72 objects/s recovering 2024-09-19T07:08:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:16 smithi116 ceph-mon[35411]: osdmap e106: 12 total, 8 up, 8 in 2024-09-19T07:08:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:16 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:16 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:16 smithi116 ceph-mon[35411]: pgmap v896: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+degraded+remapped, 68 active+clean+wait, 3 down, 12 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 31 active+clean; 9.4 GiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 669 MiB/s wr, 2.40k op/s; 2109/9980 objects degraded (21.132%); 208 MiB/s, 12 keys/s, 72 objects/s recovering 2024-09-19T07:08:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:17 smithi114 ceph-mon[30820]: osdmap e107: 12 total, 8 up, 8 in 2024-09-19T07:08:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:17 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:17 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:17 smithi116 ceph-mon[35411]: osdmap e107: 12 total, 8 up, 8 in 2024-09-19T07:08:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:17 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:17 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:18 smithi114 ceph-mon[30820]: osdmap e108: 12 total, 8 up, 8 in 2024-09-19T07:08:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:18 smithi114 ceph-mon[30820]: pgmap v899: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+degraded+remapped, 68 active+clean+wait, 3 down, 12 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 31 active+clean; 9.4 GiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s wr, 40 op/s; 2108/9980 objects degraded (21.122%); 151 MiB/s, 10 keys/s, 49 objects/s recovering 2024-09-19T07:08:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:18 smithi116 ceph-mon[35411]: osdmap e108: 12 total, 8 up, 8 in 2024-09-19T07:08:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:18 smithi116 ceph-mon[35411]: pgmap v899: 129 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 2 peering, 3 active+recovering+undersized+degraded+remapped, 68 active+clean+wait, 3 down, 12 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 31 active+clean; 9.4 GiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s wr, 40 op/s; 2108/9980 objects degraded (21.122%); 151 MiB/s, 10 keys/s, 49 objects/s recovering 2024-09-19T07:08:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:19 smithi114 ceph-mon[30820]: Health check update: Degraded data redundancy: 2108/9980 objects degraded (21.122%), 23 pgs degraded (PG_DEGRADED) 2024-09-19T07:08:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:19 smithi114 ceph-mon[30820]: osdmap e109: 12 total, 8 up, 8 in 2024-09-19T07:08:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:19 smithi114 ceph-mon[30820]: Health check update: Reduced data availability: 1 pg inactive, 3 pgs down, 2 pgs stale (PG_AVAILABILITY) 2024-09-19T07:08:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:19 smithi114 ceph-mon[30820]: osdmap e110: 12 total, 8 up, 8 in 2024-09-19T07:08:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:19 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:19 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:19 smithi116 ceph-mon[35411]: Health check update: Degraded data redundancy: 2108/9980 objects degraded (21.122%), 23 pgs degraded (PG_DEGRADED) 2024-09-19T07:08:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:19 smithi116 ceph-mon[35411]: osdmap e109: 12 total, 8 up, 8 in 2024-09-19T07:08:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:19 smithi116 ceph-mon[35411]: Health check update: Reduced data availability: 1 pg inactive, 3 pgs down, 2 pgs stale (PG_AVAILABILITY) 2024-09-19T07:08:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:19 smithi116 ceph-mon[35411]: osdmap e110: 12 total, 8 up, 8 in 2024-09-19T07:08:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:19 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:19 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:20 smithi114 ceph-mon[30820]: pgmap v902: 129 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 peering, 73 active+clean+wait, 3 down, 10 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 1 active+recovering+undersized+degraded+remapped+wait, 31 active+clean; 9.4 GiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s wr, 7 op/s; 1265/9980 objects degraded (12.675%); 385 MiB/s, 97 objects/s recovering 2024-09-19T07:08:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:20 smithi114 ceph-mon[30820]: osdmap e111: 12 total, 8 up, 8 in 2024-09-19T07:08:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:20 smithi116 ceph-mon[35411]: pgmap v902: 129 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 peering, 73 active+clean+wait, 3 down, 10 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 1 active+recovering+undersized+degraded+remapped+wait, 31 active+clean; 9.4 GiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 2.3 MiB/s wr, 7 op/s; 1265/9980 objects degraded (12.675%); 385 MiB/s, 97 objects/s recovering 2024-09-19T07:08:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:20 smithi116 ceph-mon[35411]: osdmap e111: 12 total, 8 up, 8 in 2024-09-19T07:08:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:21 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:21 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:21 smithi114 ceph-mon[30820]: osdmap e112: 12 total, 8 up, 8 in 2024-09-19T07:08:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:21 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:resolve 2024-09-19T07:08:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:21 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:21 smithi116 ceph-mon[35411]: osdmap e112: 12 total, 8 up, 8 in 2024-09-19T07:08:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:22 smithi114 ceph-mon[30820]: pgmap v905: 129 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 peering, 74 active+clean+wait, 3 down, 9 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 1 active+recovering+undersized+degraded+remapped+wait, 31 active+clean; 9.4 GiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1173/9980 objects degraded (11.754%); 476 MiB/s, 119 objects/s recovering 2024-09-19T07:08:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:22 smithi114 ceph-mon[30820]: osdmap e113: 12 total, 8 up, 8 in 2024-09-19T07:08:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:22 smithi116 ceph-mon[35411]: pgmap v905: 129 pgs: 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 peering, 74 active+clean+wait, 3 down, 9 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 1 active+recovering+undersized+degraded+remapped+wait, 31 active+clean; 9.4 GiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1173/9980 objects degraded (11.754%); 476 MiB/s, 119 objects/s recovering 2024-09-19T07:08:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:22 smithi116 ceph-mon[35411]: osdmap e113: 12 total, 8 up, 8 in 2024-09-19T07:08:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:24 smithi114 ceph-mon[30820]: osdmap e114: 12 total, 8 up, 8 in 2024-09-19T07:08:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:24 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:24 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:24 smithi114 ceph-mon[30820]: pgmap v908: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped+wait, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 peering, 78 active+clean+wait, 3 down, 8 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 31 active+clean; 9.4 GiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 873/9980 objects degraded (8.747%); 329 MiB/s, 82 objects/s recovering 2024-09-19T07:08:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:24 smithi114 ceph-mon[30820]: Health check update: Degraded data redundancy: 1173/9980 objects degraded (11.754%), 15 pgs degraded (PG_DEGRADED) 2024-09-19T07:08:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:24 smithi114 ceph-mon[30820]: osdmap e115: 12 total, 8 up, 8 in 2024-09-19T07:08:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:24 smithi116 ceph-mon[35411]: osdmap e114: 12 total, 8 up, 8 in 2024-09-19T07:08:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:24 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:replay 2024-09-19T07:08:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:24 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:24 smithi116 ceph-mon[35411]: pgmap v908: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped+wait, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 2 peering, 78 active+clean+wait, 3 down, 8 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 31 active+clean; 9.4 GiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 873/9980 objects degraded (8.747%); 329 MiB/s, 82 objects/s recovering 2024-09-19T07:08:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:24 smithi116 ceph-mon[35411]: Health check update: Degraded data redundancy: 1173/9980 objects degraded (11.754%), 15 pgs degraded (PG_DEGRADED) 2024-09-19T07:08:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:24 smithi116 ceph-mon[35411]: osdmap e115: 12 total, 8 up, 8 in 2024-09-19T07:08:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:25 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:25 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:25 smithi116 ceph-mon[35411]: osdmap e116: 12 total, 8 up, 8 in 2024-09-19T07:08:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:25 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:replay 2024-09-19T07:08:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:25 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-09-19T07:08:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:25 smithi114 ceph-mon[30820]: osdmap e116: 12 total, 8 up, 8 in 2024-09-19T07:08:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:26 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:resolve 2024-09-19T07:08:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:26 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:resolve 2024-09-19T07:08:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:26 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:resolve 2 up:active 2024-09-19T07:08:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:26 smithi114 ceph-mon[30820]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 556 secs 2024-09-19T07:08:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:26 smithi114 ceph-mon[30820]: pgmap v911: 129 pgs: 2 active+undersized+remapped+wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 76 active+clean+wait, 3 down, 7 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 37 active+clean; 9.4 GiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 1.6 MiB/s wr, 6 op/s; 655/9980 objects degraded (6.563%); 378 MiB/s, 94 objects/s recovering 2024-09-19T07:08:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:26 smithi114 ceph-mon[30820]: osdmap e117: 12 total, 8 up, 8 in 2024-09-19T07:08:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:26 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:resolve 2024-09-19T07:08:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:26 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:resolve 2024-09-19T07:08:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:26 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:resolve 2 up:active 2024-09-19T07:08:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:26 smithi116 ceph-mon[35411]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 556 secs 2024-09-19T07:08:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:26 smithi116 ceph-mon[35411]: pgmap v911: 129 pgs: 2 active+undersized+remapped+wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 76 active+clean+wait, 3 down, 7 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 37 active+clean; 9.4 GiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 1.6 MiB/s wr, 6 op/s; 655/9980 objects degraded (6.563%); 378 MiB/s, 94 objects/s recovering 2024-09-19T07:08:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:26 smithi116 ceph-mon[35411]: osdmap e117: 12 total, 8 up, 8 in 2024-09-19T07:08:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-09-19T07:08:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:reconnect 2024-09-19T07:08:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:reconnect 2024-09-19T07:08:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:reconnect 2024-09-19T07:08:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:reconnect 2 up:active 2024-09-19T07:08:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 633 secs 2024-09-19T07:08:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: reconnect by client.25246 v1:172.21.15.114:0/1565499477 after 0.0239995 2024-09-19T07:08:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 557 secs 2024-09-19T07:08:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:27 smithi114 ceph-mon[30820]: osdmap e118: 12 total, 8 up, 8 in 2024-09-19T07:08:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-09-19T07:08:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:reconnect 2024-09-19T07:08:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:reconnect 2024-09-19T07:08:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:reconnect 2024-09-19T07:08:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:reconnect 2 up:active 2024-09-19T07:08:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 633 secs 2024-09-19T07:08:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: reconnect by client.25246 v1:172.21.15.114:0/1565499477 after 0.0239995 2024-09-19T07:08:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 557 secs 2024-09-19T07:08:28.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:27 smithi116 ceph-mon[35411]: osdmap e118: 12 total, 8 up, 8 in 2024-09-19T07:08:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: Health check cleared: MDS_SLOW_METADATA_IO (was: 2 MDSs report slow metadata IOs) 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:rejoin 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:rejoin 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:rejoin 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:rejoin 2 up:active 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: pgmap v914: 129 pgs: 1 active+undersized+remapped+wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 64 active+clean+wait, 3 down, 7 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 50 active+clean; 9.4 GiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 4.3 MiB/s wr, 21 op/s; 655/9980 objects degraded (6.563%); 141 MiB/s, 34 objects/s recovering 2024-09-19T07:08:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:28 smithi114 ceph-mon[30820]: osdmap e119: 12 total, 8 up, 8 in 2024-09-19T07:08:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: Health check cleared: MDS_SLOW_METADATA_IO (was: 2 MDSs report slow metadata IOs) 2024-09-19T07:08:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:rejoin 2024-09-19T07:08:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:rejoin 2024-09-19T07:08:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:rejoin 2024-09-19T07:08:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:rejoin 2 up:active 2024-09-19T07:08:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-09-19T07:08:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-09-19T07:08:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-09-19T07:08:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: pgmap v914: 129 pgs: 1 active+undersized+remapped+wait, 1 creating+peering, 1 active+recovery_wait+undersized+degraded+remapped, 64 active+clean+wait, 3 down, 7 active+recovery_wait+undersized+degraded+remapped+wait, 2 stale+active+clean, 50 active+clean; 9.4 GiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 4.3 MiB/s wr, 21 op/s; 655/9980 objects degraded (6.563%); 141 MiB/s, 34 objects/s recovering 2024-09-19T07:08:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:28 smithi116 ceph-mon[35411]: osdmap e119: 12 total, 8 up, 8 in 2024-09-19T07:08:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:29 smithi114 ceph-mon[30820]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-19T07:08:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:active 2024-09-19T07:08:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:active 2024-09-19T07:08:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:29 smithi114 ceph-mon[30820]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:active 2024-09-19T07:08:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:29 smithi114 ceph-mon[30820]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-09-19T07:08:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:29 smithi114 ceph-mon[30820]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-09-19T07:08:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:29 smithi114 ceph-mon[30820]: Health check update: Degraded data redundancy: 655/9980 objects degraded (6.563%), 8 pgs degraded (PG_DEGRADED) 2024-09-19T07:08:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:29 smithi114 ceph-mon[30820]: osdmap e120: 12 total, 8 up, 8 in 2024-09-19T07:08:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:29 smithi116 ceph-mon[35411]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-19T07:08:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6838/3315910714,v1:172.21.15.114:6839/3315910714] up:active 2024-09-19T07:08:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.116:6832/1884054485,v1:172.21.15.116:6833/1884054485] up:active 2024-09-19T07:08:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:29 smithi116 ceph-mon[35411]: mds.? [v2:172.21.15.114:6834/2797655862,v1:172.21.15.114:6835/2797655862] up:active 2024-09-19T07:08:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:29 smithi116 ceph-mon[35411]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-09-19T07:08:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:29 smithi116 ceph-mon[35411]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-09-19T07:08:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:29 smithi116 ceph-mon[35411]: Health check update: Degraded data redundancy: 655/9980 objects degraded (6.563%), 8 pgs degraded (PG_DEGRADED) 2024-09-19T07:08:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:29 smithi116 ceph-mon[35411]: osdmap e120: 12 total, 8 up, 8 in 2024-09-19T07:08:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:30 smithi114 ceph-mon[30820]: pgmap v917: 129 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 12 active+clean+wait, 3 down, 2 stale+active+clean, 106 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 43 MiB/s wr, 199 op/s; 283/9983 objects degraded (2.835%); 371 MiB/s, 92 objects/s recovering 2024-09-19T07:08:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:30 smithi114 ceph-mon[30820]: osdmap e121: 12 total, 8 up, 8 in 2024-09-19T07:08:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:30 smithi116 ceph-mon[35411]: pgmap v917: 129 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 12 active+clean+wait, 3 down, 2 stale+active+clean, 106 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 43 MiB/s wr, 199 op/s; 283/9983 objects degraded (2.835%); 371 MiB/s, 92 objects/s recovering 2024-09-19T07:08:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:30 smithi116 ceph-mon[35411]: osdmap e121: 12 total, 8 up, 8 in 2024-09-19T07:08:32.416 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 07:08:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g[71204]: 2024-09-19T07:08:32.162+0000 7fa9e7400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-19T07:08:32.416 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 07:08:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g[71204]: 2024-09-19T07:08:32.162+0000 7fa9e7400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-19T07:08:32.416 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 07:08:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g[71204]: 2024-09-19T07:08:32.162+0000 7fa9e7400640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-19T07:08:32.416 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 07:08:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g[71204]: 2024-09-19T07:08:32.162+0000 7fa9e7400640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-09-19T07:08:32.417 INFO:journalctl@ceph.mds.g.smithi114.stdout:Sep 19 07:08:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mds-g[71204]: 2024-09-19T07:08:32.162+0000 7fa9e7400640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-09-19T07:08:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:32 smithi114 ceph-mon[30820]: osdmap e122: 12 total, 8 up, 8 in 2024-09-19T07:08:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:32 smithi114 ceph-mon[30820]: pgmap v920: 129 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+clean+wait, 3 down, 2 stale+active+clean, 112 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 48 MiB/s wr, 239 op/s; 283/9983 objects degraded (2.835%); 371 MiB/s, 92 objects/s recovering 2024-09-19T07:08:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:32 smithi116 ceph-mon[35411]: osdmap e122: 12 total, 8 up, 8 in 2024-09-19T07:08:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:32 smithi116 ceph-mon[35411]: pgmap v920: 129 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+clean+wait, 3 down, 2 stale+active+clean, 112 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 48 MiB/s wr, 239 op/s; 283/9983 objects degraded (2.835%); 371 MiB/s, 92 objects/s recovering 2024-09-19T07:08:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:33 smithi116 ceph-mon[35411]: osdmap e123: 12 total, 8 up, 8 in 2024-09-19T07:08:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:33 smithi114 ceph-mon[30820]: osdmap e123: 12 total, 8 up, 8 in 2024-09-19T07:08:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:34 smithi116 ceph-mon[35411]: osdmap e124: 12 total, 8 up, 8 in 2024-09-19T07:08:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:34 smithi116 ceph-mon[35411]: pgmap v923: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+clean+wait, 3 down, 2 stale+active+clean, 112 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 48 MiB/s wr, 257 op/s; 81/9983 objects degraded (0.811%); 199 MiB/s, 50 objects/s recovering 2024-09-19T07:08:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:34 smithi116 ceph-mon[35411]: Health check update: Degraded data redundancy: 283/9983 objects degraded (2.835%), 4 pgs degraded (PG_DEGRADED) 2024-09-19T07:08:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:34 smithi114 ceph-mon[30820]: osdmap e124: 12 total, 8 up, 8 in 2024-09-19T07:08:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:34 smithi114 ceph-mon[30820]: pgmap v923: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 active+clean+wait, 3 down, 2 stale+active+clean, 112 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 MiB/s rd, 48 MiB/s wr, 257 op/s; 81/9983 objects degraded (0.811%); 199 MiB/s, 50 objects/s recovering 2024-09-19T07:08:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:34 smithi114 ceph-mon[30820]: Health check update: Degraded data redundancy: 283/9983 objects degraded (2.835%), 4 pgs degraded (PG_DEGRADED) 2024-09-19T07:08:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:36 smithi114 ceph-mon[30820]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/9983 objects degraded (0.811%), 1 pg degraded) 2024-09-19T07:08:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:36 smithi116 ceph-mon[35411]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 81/9983 objects degraded (0.811%), 1 pg degraded) 2024-09-19T07:08:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:37 smithi114 ceph-mon[30820]: pgmap v924: 129 pgs: 12 active+clean+wait, 3 down, 2 stale+active+clean, 112 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 35 MiB/s wr, 184 op/s; 149 MiB/s, 37 objects/s recovering 2024-09-19T07:08:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:37 smithi116 ceph-mon[35411]: pgmap v924: 129 pgs: 12 active+clean+wait, 3 down, 2 stale+active+clean, 112 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 35 MiB/s wr, 184 op/s; 149 MiB/s, 37 objects/s recovering 2024-09-19T07:08:39.271 INFO:tasks.workunit.client.0.smithi114.stdout:OK 2024-09-19T07:08:39.271 INFO:tasks.workunit.client.0.smithi114.stderr:+ echo OK 2024-09-19T07:08:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:39 smithi114 ceph-mon[30820]: pgmap v925: 129 pgs: 12 active+clean+wait, 3 down, 2 stale+active+clean, 112 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 157 KiB/s rd, 30 MiB/s wr, 155 op/s; 126 MiB/s, 31 objects/s recovering 2024-09-19T07:08:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:39 smithi116 ceph-mon[35411]: pgmap v925: 129 pgs: 12 active+clean+wait, 3 down, 2 stale+active+clean, 112 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 157 KiB/s rd, 30 MiB/s wr, 155 op/s; 126 MiB/s, 31 objects/s recovering 2024-09-19T07:08:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:41 smithi114 ceph-mon[30820]: pgmap v926: 129 pgs: 6 active+clean+wait, 3 down, 2 stale+active+clean, 118 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 164 KiB/s rd, 34 MiB/s wr, 168 op/s; 99 MiB/s, 25 objects/s recovering 2024-09-19T07:08:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:41 smithi116 ceph-mon[35411]: pgmap v926: 129 pgs: 6 active+clean+wait, 3 down, 2 stale+active+clean, 118 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 164 KiB/s rd, 34 MiB/s wr, 168 op/s; 99 MiB/s, 25 objects/s recovering 2024-09-19T07:08:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:43 smithi114 ceph-mon[30820]: pgmap v927: 129 pgs: 5 active+clean+wait, 3 down, 2 stale+active+clean, 119 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 224 KiB/s rd, 27 MiB/s wr, 143 op/s; 85 MiB/s, 21 objects/s recovering 2024-09-19T07:08:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:43 smithi116 ceph-mon[35411]: pgmap v927: 129 pgs: 5 active+clean+wait, 3 down, 2 stale+active+clean, 119 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 224 KiB/s rd, 27 MiB/s wr, 143 op/s; 85 MiB/s, 21 objects/s recovering 2024-09-19T07:08:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:08:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:08:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: pgmap v928: 129 pgs: 5 active+clean+wait, 3 down, 2 stale+active+clean, 119 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 20 MiB/s wr, 99 op/s 2024-09-19T07:08:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:08:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:08:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: pgmap v928: 129 pgs: 5 active+clean+wait, 3 down, 2 stale+active+clean, 119 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 344 KiB/s rd, 20 MiB/s wr, 99 op/s 2024-09-19T07:08:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:08:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:08:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:08:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:46 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:08:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:46 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:08:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:46 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:08:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:46 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:08:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-mon[30820]: pgmap v929: 129 pgs: 4 active+clean+wait, 3 down, 2 stale+active+clean, 120 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 304 KiB/s rd, 20 MiB/s wr, 100 op/s 2024-09-19T07:08:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:08:47.317+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:08:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:08:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:08:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:08:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:08:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:08:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:08:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:08:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:08:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:08:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:08:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:08:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:08:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:08:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:08:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:08:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:08:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:08:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:08:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:08:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:08:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:08:47.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:08:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:08:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:47 smithi116 ceph-mon[35411]: pgmap v929: 129 pgs: 4 active+clean+wait, 3 down, 2 stale+active+clean, 120 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 304 KiB/s rd, 20 MiB/s wr, 100 op/s 2024-09-19T07:08:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:48 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:08:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:08:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:08:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:08:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:08:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:08:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:08:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:08:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:08:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:08:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:08:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:08:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:48 smithi114 ceph-mon[30820]: pgmap v930: 129 pgs: 4 active+clean+wait, 3 down, 2 stale+active+clean, 120 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 242 KiB/s rd, 20 MiB/s wr, 96 op/s 2024-09-19T07:08:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:48 smithi114 ceph-mon[30820]: pgmap v931: 129 pgs: 4 active+clean+wait, 3 down, 2 stale+active+clean, 120 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 293 KiB/s rd, 24 MiB/s wr, 116 op/s 2024-09-19T07:08:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:08:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:48 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:08:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:08:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:08:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:08:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:48 smithi116 ceph-mon[35411]: pgmap v930: 129 pgs: 4 active+clean+wait, 3 down, 2 stale+active+clean, 120 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 242 KiB/s rd, 20 MiB/s wr, 96 op/s 2024-09-19T07:08:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:48 smithi116 ceph-mon[35411]: pgmap v931: 129 pgs: 4 active+clean+wait, 3 down, 2 stale+active+clean, 120 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 293 KiB/s rd, 24 MiB/s wr, 116 op/s 2024-09-19T07:08:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:08:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:50 smithi114 ceph-mon[30820]: pgmap v932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 261 KiB/s rd, 50 MiB/s wr, 245 op/s 2024-09-19T07:08:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:50 smithi116 ceph-mon[35411]: pgmap v932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 261 KiB/s rd, 50 MiB/s wr, 245 op/s 2024-09-19T07:08:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:52 smithi114 ceph-mon[30820]: pgmap v933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 178 KiB/s rd, 64 MiB/s wr, 311 op/s 2024-09-19T07:08:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:52 smithi116 ceph-mon[35411]: pgmap v933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 178 KiB/s rd, 64 MiB/s wr, 311 op/s 2024-09-19T07:08:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:54 smithi114 ceph-mon[30820]: pgmap v934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 60 MiB/s wr, 296 op/s 2024-09-19T07:08:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:54 smithi116 ceph-mon[35411]: pgmap v934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 60 MiB/s wr, 296 op/s 2024-09-19T07:08:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:56 smithi114 ceph-mon[30820]: pgmap v935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 62 MiB/s wr, 307 op/s 2024-09-19T07:08:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:56 smithi116 ceph-mon[35411]: pgmap v935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 62 MiB/s wr, 307 op/s 2024-09-19T07:08:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:08:58 smithi114 ceph-mon[30820]: pgmap v936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 57 MiB/s wr, 281 op/s 2024-09-19T07:08:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:08:58 smithi116 ceph-mon[35411]: pgmap v936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 57 MiB/s wr, 281 op/s 2024-09-19T07:09:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:00 smithi114 ceph-mon[30820]: pgmap v937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 47 MiB/s wr, 235 op/s 2024-09-19T07:09:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:00 smithi116 ceph-mon[35411]: pgmap v937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 47 MiB/s wr, 235 op/s 2024-09-19T07:09:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:02 smithi114 ceph-mon[30820]: pgmap v938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 21 MiB/s wr, 106 op/s 2024-09-19T07:09:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:02 smithi116 ceph-mon[35411]: pgmap v938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 21 MiB/s wr, 106 op/s 2024-09-19T07:09:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:04 smithi114 ceph-mon[30820]: pgmap v939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 8.3 MiB/s wr, 45 op/s 2024-09-19T07:09:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:04 smithi116 ceph-mon[35411]: pgmap v939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 8.3 MiB/s wr, 45 op/s 2024-09-19T07:09:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:06 smithi114 ceph-mon[30820]: pgmap v940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.7 MiB/s wr, 27 op/s 2024-09-19T07:09:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:06 smithi116 ceph-mon[35411]: pgmap v940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.7 MiB/s wr, 27 op/s 2024-09-19T07:09:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:08 smithi114 ceph-mon[30820]: pgmap v941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-19T07:09:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:08 smithi116 ceph-mon[35411]: pgmap v941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-19T07:09:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:10 smithi114 ceph-mon[30820]: pgmap v942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-19T07:09:10.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:10 smithi116 ceph-mon[35411]: pgmap v942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-19T07:09:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:11 smithi114 ceph-mon[30820]: Health check update: Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale (PG_AVAILABILITY) 2024-09-19T07:09:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:11 smithi116 ceph-mon[35411]: Health check update: Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale (PG_AVAILABILITY) 2024-09-19T07:09:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:12 smithi114 ceph-mon[30820]: pgmap v943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-19T07:09:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:12 smithi116 ceph-mon[35411]: pgmap v943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-19T07:09:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:14 smithi114 ceph-mon[30820]: pgmap v944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-19T07:09:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:14 smithi116 ceph-mon[35411]: pgmap v944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-19T07:09:16.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:16 smithi116 ceph-mon[35411]: pgmap v945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-19T07:09:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:16 smithi114 ceph-mon[30820]: pgmap v945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-09-19T07:09:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:18 smithi114 ceph-mon[30820]: pgmap v946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-09-19T07:09:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:18 smithi116 ceph-mon[35411]: pgmap v946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-09-19T07:09:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:20 smithi114 ceph-mon[30820]: pgmap v947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-19T07:09:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:20 smithi116 ceph-mon[35411]: pgmap v947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-19T07:09:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:22 smithi114 ceph-mon[30820]: pgmap v948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-19T07:09:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:22 smithi116 ceph-mon[35411]: pgmap v948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-19T07:09:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:24 smithi114 ceph-mon[30820]: pgmap v949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-09-19T07:09:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:24 smithi116 ceph-mon[35411]: pgmap v949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-09-19T07:09:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:26 smithi114 ceph-mon[30820]: pgmap v950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 853 B/s wr, 7 op/s 2024-09-19T07:09:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:26 smithi116 ceph-mon[35411]: pgmap v950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 853 B/s wr, 7 op/s 2024-09-19T07:09:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:28 smithi114 ceph-mon[30820]: pgmap v951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-19T07:09:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:28 smithi116 ceph-mon[35411]: pgmap v951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-19T07:09:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:30 smithi114 ceph-mon[30820]: pgmap v952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-19T07:09:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:30 smithi116 ceph-mon[35411]: pgmap v952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-19T07:09:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:32 smithi114 ceph-mon[30820]: pgmap v953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-19T07:09:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:32 smithi116 ceph-mon[35411]: pgmap v953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2024-09-19T07:09:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:34 smithi114 ceph-mon[30820]: pgmap v954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-19T07:09:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:34 smithi116 ceph-mon[35411]: pgmap v954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-09-19T07:09:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:36 smithi114 ceph-mon[30820]: pgmap v955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2024-09-19T07:09:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:36 smithi116 ceph-mon[35411]: pgmap v955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2024-09-19T07:09:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:38 smithi114 ceph-mon[30820]: pgmap v956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:09:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:38 smithi116 ceph-mon[35411]: pgmap v956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:09:39.273 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-09-19T07:09:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:40 smithi114 ceph-mon[30820]: pgmap v957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:09:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:40 smithi116 ceph-mon[35411]: pgmap v957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:09:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:42 smithi114 ceph-mon[30820]: pgmap v958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:09:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:42 smithi116 ceph-mon[35411]: pgmap v958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:09:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:44 smithi114 ceph-mon[30820]: pgmap v959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:09:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:44 smithi116 ceph-mon[35411]: pgmap v959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:09:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:46 smithi114 ceph-mon[30820]: pgmap v960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-09-19T07:09:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:46 smithi116 ceph-mon[35411]: pgmap v960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-09-19T07:09:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:09:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:09:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:48 smithi114 ceph-mon[30820]: pgmap v961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-19T07:09:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:48 smithi116 ceph-mon[35411]: pgmap v961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-19T07:09:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-mon[30820]: pgmap v962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:09:50.676+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:09:50.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:09:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:09:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:09:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:50 smithi116 ceph-mon[35411]: pgmap v962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-19T07:09:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:09:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:09:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:09:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:09:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:09:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:09:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:52 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:09:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:09:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:09:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:52 smithi114 ceph-mon[30820]: pgmap v963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 812 B/s wr, 7 op/s 2024-09-19T07:09:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:52 smithi114 ceph-mon[30820]: pgmap v964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 986 B/s wr, 6 op/s 2024-09-19T07:09:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:52 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:09:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:09:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:09:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:09:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:09:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:09:52.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:09:52.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:09:52.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:09:52.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:09:52.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:52 smithi116 ceph-mon[35411]: pgmap v963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 812 B/s wr, 7 op/s 2024-09-19T07:09:52.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:52 smithi116 ceph-mon[35411]: pgmap v964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 986 B/s wr, 6 op/s 2024-09-19T07:09:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:54 smithi114 ceph-mon[30820]: pgmap v965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 986 B/s wr, 6 op/s 2024-09-19T07:09:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:54 smithi116 ceph-mon[35411]: pgmap v965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 986 B/s wr, 6 op/s 2024-09-19T07:09:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:56 smithi114 ceph-mon[30820]: pgmap v966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-19T07:09:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:56 smithi116 ceph-mon[35411]: pgmap v966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 5 op/s 2024-09-19T07:09:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:09:58 smithi114 ceph-mon[30820]: pgmap v967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:09:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:09:58 smithi116 ceph-mon[35411]: pgmap v967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:10:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: pgmap v968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:10:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:10:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: host smithi118 (172.21.15.118) failed check: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: mon.c (rank 1) addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] is down (out of quorum) 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: pg 3.20 is stuck stale for 11m, current state stale+active+clean, last acting [11,2,5] 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: pg 3.25 is stuck stale for 11m, current state stale+active+clean, last acting [11,8,5] 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: pg 4.f is down, acting [9,3,6,0] 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: pg 4.12 is down, acting [9,1,6,0] 2024-09-19T07:10:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:00 smithi114 ceph-mon[30820]: pg 4.1d is down, acting [1,3,7,6] 2024-09-19T07:10:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: pgmap v968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:10:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: Health detail: HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:10:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: [WRN] CEPHADM_HOST_CHECK_FAILED: 1 hosts fail cephadm check 2024-09-19T07:10:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: host smithi118 (172.21.15.118) failed check: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:10:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-19T07:10:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: mon.c (rank 1) addr [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] is down (out of quorum) 2024-09-19T07:10:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:10:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: pg 3.20 is stuck stale for 11m, current state stale+active+clean, last acting [11,2,5] 2024-09-19T07:10:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: pg 3.25 is stuck stale for 11m, current state stale+active+clean, last acting [11,8,5] 2024-09-19T07:10:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: pg 4.f is down, acting [9,3,6,0] 2024-09-19T07:10:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: pg 4.12 is down, acting [9,1,6,0] 2024-09-19T07:10:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:00 smithi116 ceph-mon[35411]: pg 4.1d is down, acting [1,3,7,6] 2024-09-19T07:10:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:02 smithi114 ceph-mon[30820]: pgmap v969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:10:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:02 smithi116 ceph-mon[35411]: pgmap v969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:10:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:04 smithi114 ceph-mon[30820]: pgmap v970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:04 smithi116 ceph-mon[35411]: pgmap v970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:06 smithi114 ceph-mon[30820]: pgmap v971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:10:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:06 smithi116 ceph-mon[35411]: pgmap v971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:10:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:08 smithi114 ceph-mon[30820]: pgmap v972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:10:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:08 smithi116 ceph-mon[35411]: pgmap v972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:10:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:10 smithi114 ceph-mon[30820]: pgmap v973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:10 smithi116 ceph-mon[35411]: pgmap v973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:12 smithi114 ceph-mon[30820]: pgmap v974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:10:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:12 smithi116 ceph-mon[35411]: pgmap v974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:10:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:14 smithi114 ceph-mon[30820]: pgmap v975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:14 smithi116 ceph-mon[35411]: pgmap v975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:16 smithi114 ceph-mon[30820]: pgmap v976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:10:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:16 smithi116 ceph-mon[35411]: pgmap v976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:10:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:18 smithi114 ceph-mon[30820]: pgmap v977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:10:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:18 smithi116 ceph-mon[35411]: pgmap v977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:10:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:20 smithi114 ceph-mon[30820]: pgmap v978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:20 smithi116 ceph-mon[35411]: pgmap v978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:22 smithi114 ceph-mon[30820]: pgmap v979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:10:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:22 smithi116 ceph-mon[35411]: pgmap v979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:10:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:24 smithi114 ceph-mon[30820]: pgmap v980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:10:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:24 smithi116 ceph-mon[35411]: pgmap v980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:10:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:26 smithi114 ceph-mon[30820]: pgmap v981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:10:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:26 smithi116 ceph-mon[35411]: pgmap v981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:10:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:28 smithi114 ceph-mon[30820]: pgmap v982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:10:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:28 smithi116 ceph-mon[35411]: pgmap v982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:10:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:30 smithi114 ceph-mon[30820]: pgmap v983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:10:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:30 smithi116 ceph-mon[35411]: pgmap v983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:10:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:32 smithi114 ceph-mon[30820]: pgmap v984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:10:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:32 smithi116 ceph-mon[35411]: pgmap v984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:10:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:34 smithi114 ceph-mon[30820]: pgmap v985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:34 smithi116 ceph-mon[35411]: pgmap v985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:36 smithi114 ceph-mon[30820]: pgmap v986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:10:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:36 smithi116 ceph-mon[35411]: pgmap v986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:10:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:37 smithi114 ceph-mon[30820]: pgmap v987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:10:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:37 smithi116 ceph-mon[35411]: pgmap v987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:10:39.274 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-09-19T07:10:39.275 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-19T07:10:39.275 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-19T07:10:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:39 smithi114 ceph-mon[30820]: pgmap v988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:39 smithi116 ceph-mon[35411]: pgmap v988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:41 smithi114 ceph-mon[30820]: pgmap v989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:10:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:41 smithi116 ceph-mon[35411]: pgmap v989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:10:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:43 smithi114 ceph-mon[30820]: pgmap v990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:43 smithi116 ceph-mon[35411]: pgmap v990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:10:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:45 smithi114 ceph-mon[30820]: pgmap v991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-19T07:10:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:45 smithi116 ceph-mon[35411]: pgmap v991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-09-19T07:10:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:47 smithi114 ceph-mon[30820]: pgmap v992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-09-19T07:10:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:47 smithi116 ceph-mon[35411]: pgmap v992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 938 B/s wr, 6 op/s 2024-09-19T07:10:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:49 smithi114 ceph-mon[30820]: pgmap v993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-19T07:10:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:49 smithi116 ceph-mon[35411]: pgmap v993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-19T07:10:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:51 smithi114 ceph-mon[30820]: pgmap v994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-19T07:10:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:10:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:51 smithi116 ceph-mon[35411]: pgmap v994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-09-19T07:10:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:10:54.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:53 smithi114 ceph-mon[30820]: pgmap v995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-09-19T07:10:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:53 smithi116 ceph-mon[35411]: pgmap v995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:10:54.036+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:10:54.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:10:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:10:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:10:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:10:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-mon[30820]: pgmap v996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-09-19T07:10:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-mon[30820]: pgmap v997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-19T07:10:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:10:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:10:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:10:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:10:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:10:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:54 smithi116 ceph-mon[35411]: pgmap v996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:54 smithi116 ceph-mon[35411]: pgmap v997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 767 B/s wr, 5 op/s 2024-09-19T07:10:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:10:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:57 smithi114 ceph-mon[30820]: pgmap v998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 548 B/s wr, 7 op/s 2024-09-19T07:10:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:57 smithi116 ceph-mon[35411]: pgmap v998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 548 B/s wr, 7 op/s 2024-09-19T07:10:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:10:59 smithi114 ceph-mon[30820]: pgmap v999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 438 B/s wr, 7 op/s 2024-09-19T07:10:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:10:59 smithi116 ceph-mon[35411]: pgmap v999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 438 B/s wr, 7 op/s 2024-09-19T07:11:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:01 smithi114 ceph-mon[30820]: pgmap v1000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:11:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:01 smithi116 ceph-mon[35411]: pgmap v1000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:11:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:03 smithi114 ceph-mon[30820]: pgmap v1001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:11:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:03 smithi116 ceph-mon[35411]: pgmap v1001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:11:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:05 smithi114 ceph-mon[30820]: pgmap v1002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:11:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:05 smithi116 ceph-mon[35411]: pgmap v1002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:11:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:07 smithi114 ceph-mon[30820]: pgmap v1003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:07 smithi116 ceph-mon[35411]: pgmap v1003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:09 smithi114 ceph-mon[30820]: pgmap v1004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:09 smithi116 ceph-mon[35411]: pgmap v1004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:11 smithi114 ceph-mon[30820]: pgmap v1005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:11 smithi116 ceph-mon[35411]: pgmap v1005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:13 smithi114 ceph-mon[30820]: pgmap v1006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:13 smithi116 ceph-mon[35411]: pgmap v1006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:15 smithi114 ceph-mon[30820]: pgmap v1007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:15 smithi116 ceph-mon[35411]: pgmap v1007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:17 smithi114 ceph-mon[30820]: pgmap v1008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:17 smithi116 ceph-mon[35411]: pgmap v1008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:19 smithi114 ceph-mon[30820]: pgmap v1009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:19 smithi116 ceph-mon[35411]: pgmap v1009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:21 smithi114 ceph-mon[30820]: pgmap v1010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:21 smithi116 ceph-mon[35411]: pgmap v1010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:23 smithi114 ceph-mon[30820]: pgmap v1011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:23 smithi116 ceph-mon[35411]: pgmap v1011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:25.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:25 smithi114 ceph-mon[30820]: pgmap v1012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:25.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:25 smithi116 ceph-mon[35411]: pgmap v1012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:27 smithi114 ceph-mon[30820]: pgmap v1013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:27 smithi116 ceph-mon[35411]: pgmap v1013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:29 smithi114 ceph-mon[30820]: pgmap v1014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:29 smithi116 ceph-mon[35411]: pgmap v1014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:31 smithi114 ceph-mon[30820]: pgmap v1015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:31 smithi116 ceph-mon[35411]: pgmap v1015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:33 smithi116 ceph-mon[35411]: pgmap v1016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:33 smithi114 ceph-mon[30820]: pgmap v1016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:35 smithi116 ceph-mon[35411]: pgmap v1017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:35 smithi114 ceph-mon[30820]: pgmap v1017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:37 smithi116 ceph-mon[35411]: pgmap v1018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:37 smithi114 ceph-mon[30820]: pgmap v1018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:39 smithi114 ceph-mon[30820]: pgmap v1019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:39 smithi116 ceph-mon[35411]: pgmap v1019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:41 smithi114 ceph-mon[30820]: pgmap v1020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:41 smithi116 ceph-mon[35411]: pgmap v1020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:43 smithi114 ceph-mon[30820]: pgmap v1021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:43 smithi116 ceph-mon[35411]: pgmap v1021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:45.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:45 smithi114 ceph-mon[30820]: pgmap v1022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:45 smithi116 ceph-mon[35411]: pgmap v1022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:47.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:47 smithi116 ceph-mon[35411]: pgmap v1023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:47 smithi114 ceph-mon[30820]: pgmap v1023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:49 smithi114 ceph-mon[30820]: pgmap v1024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:49 smithi116 ceph-mon[35411]: pgmap v1024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:51.674 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:51 smithi114 ceph-mon[30820]: pgmap v1025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:51 smithi116 ceph-mon[35411]: pgmap v1025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:53 smithi114 ceph-mon[30820]: pgmap v1026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:53 smithi116 ceph-mon[35411]: pgmap v1026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:11:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:11:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:11:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:55 smithi114 ceph-mon[30820]: pgmap v1027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:55 smithi116 ceph-mon[35411]: pgmap v1027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:11:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-mon[30820]: pgmap v1028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:11:57.397+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:11:57.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:11:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:11:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:11:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:11:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:11:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:11:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:11:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:11:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:57 smithi116 ceph-mon[35411]: pgmap v1028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:11:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:11:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:11:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:11:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:58 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:11:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:11:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:11:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:11:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:11:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:11:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:11:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:11:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:11:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:11:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:58 smithi114 ceph-mon[30820]: pgmap v1029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:11:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:11:58 smithi114 ceph-mon[30820]: pgmap v1030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:11:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:58 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:11:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:11:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:58 smithi116 ceph-mon[35411]: pgmap v1029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:11:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:11:58 smithi116 ceph-mon[35411]: pgmap v1030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:00 smithi114 ceph-mon[30820]: pgmap v1031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:00 smithi116 ceph-mon[35411]: pgmap v1031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:02 smithi114 ceph-mon[30820]: pgmap v1032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:02 smithi116 ceph-mon[35411]: pgmap v1032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:04 smithi114 ceph-mon[30820]: pgmap v1033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:04 smithi116 ceph-mon[35411]: pgmap v1033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:06 smithi114 ceph-mon[30820]: pgmap v1034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:06 smithi116 ceph-mon[35411]: pgmap v1034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:12:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:08 smithi114 ceph-mon[30820]: pgmap v1035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:12:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:08 smithi116 ceph-mon[35411]: pgmap v1035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:12:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:10 smithi114 ceph-mon[30820]: pgmap v1036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:10 smithi116 ceph-mon[35411]: pgmap v1036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:12 smithi114 ceph-mon[30820]: pgmap v1037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:12 smithi116 ceph-mon[35411]: pgmap v1037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:14 smithi114 ceph-mon[30820]: pgmap v1038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:14 smithi116 ceph-mon[35411]: pgmap v1038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:16.151 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:15 smithi114 ceph-mon[30820]: pgmap v1039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:15 smithi116 ceph-mon[35411]: pgmap v1039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:18.881 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:18 smithi116 ceph-mon[35411]: pgmap v1040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:18 smithi114 ceph-mon[30820]: pgmap v1040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:20 smithi114 ceph-mon[30820]: pgmap v1041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:20 smithi116 ceph-mon[35411]: pgmap v1041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:22 smithi114 ceph-mon[30820]: pgmap v1042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:22 smithi116 ceph-mon[35411]: pgmap v1042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:24 smithi114 ceph-mon[30820]: pgmap v1043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:24 smithi116 ceph-mon[35411]: pgmap v1043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:26.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:26 smithi114 ceph-mon[30820]: pgmap v1044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:26 smithi116 ceph-mon[35411]: pgmap v1044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:28 smithi114 ceph-mon[30820]: pgmap v1045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:28 smithi116 ceph-mon[35411]: pgmap v1045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:30 smithi114 ceph-mon[30820]: pgmap v1046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:30 smithi116 ceph-mon[35411]: pgmap v1046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:32 smithi114 ceph-mon[30820]: pgmap v1047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:32 smithi116 ceph-mon[35411]: pgmap v1047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:34 smithi114 ceph-mon[30820]: pgmap v1048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:34 smithi116 ceph-mon[35411]: pgmap v1048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:36 smithi114 ceph-mon[30820]: pgmap v1049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:36 smithi116 ceph-mon[35411]: pgmap v1049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:38 smithi114 ceph-mon[30820]: pgmap v1050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:38 smithi116 ceph-mon[35411]: pgmap v1050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:40 smithi114 ceph-mon[30820]: pgmap v1051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:40 smithi116 ceph-mon[35411]: pgmap v1051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:42 smithi114 ceph-mon[30820]: pgmap v1052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:42 smithi116 ceph-mon[35411]: pgmap v1052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:44 smithi114 ceph-mon[30820]: pgmap v1053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:44 smithi116 ceph-mon[35411]: pgmap v1053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:46 smithi114 ceph-mon[30820]: pgmap v1054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:46 smithi116 ceph-mon[35411]: pgmap v1054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:48 smithi114 ceph-mon[30820]: pgmap v1055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:48 smithi116 ceph-mon[35411]: pgmap v1055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:50 smithi114 ceph-mon[30820]: pgmap v1056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:50 smithi116 ceph-mon[35411]: pgmap v1056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:12:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:52 smithi114 ceph-mon[30820]: pgmap v1057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:52 smithi116 ceph-mon[35411]: pgmap v1057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:12:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:54 smithi114 ceph-mon[30820]: pgmap v1058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:54 smithi116 ceph-mon[35411]: pgmap v1058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:56 smithi114 ceph-mon[30820]: pgmap v1059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:56 smithi116 ceph-mon[35411]: pgmap v1059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:12:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:12:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:12:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:58 smithi114 ceph-mon[30820]: pgmap v1060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:12:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:58 smithi116 ceph-mon[35411]: pgmap v1060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:12:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:12:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:12:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:13:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-mon[30820]: pgmap v1061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:00.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:13:00.692+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:13:00.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:13:00.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:13:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:13:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:00 smithi116 ceph-mon[35411]: pgmap v1061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:13:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:13:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:13:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:13:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:13:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:13:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:02 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:13:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:13:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:13:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:13:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:13:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:02 smithi114 ceph-mon[30820]: pgmap v1062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:13:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:02 smithi114 ceph-mon[30820]: pgmap v1063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:02 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:13:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:13:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:13:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:13:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:13:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:02 smithi116 ceph-mon[35411]: pgmap v1062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:13:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:02 smithi116 ceph-mon[35411]: pgmap v1063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:13:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:04 smithi114 ceph-mon[30820]: pgmap v1064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:13:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:04 smithi116 ceph-mon[35411]: pgmap v1064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:13:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:06 smithi114 ceph-mon[30820]: pgmap v1065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:13:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:06 smithi116 ceph-mon[35411]: pgmap v1065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:13:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:08 smithi114 ceph-mon[30820]: pgmap v1066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:13:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:08 smithi116 ceph-mon[35411]: pgmap v1066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:13:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:10 smithi114 ceph-mon[30820]: pgmap v1067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:13:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:10 smithi116 ceph-mon[35411]: pgmap v1067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:13:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:12 smithi114 ceph-mon[30820]: pgmap v1068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:13:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:12 smithi116 ceph-mon[35411]: pgmap v1068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:13:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:14 smithi114 ceph-mon[30820]: pgmap v1069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:14 smithi116 ceph-mon[35411]: pgmap v1069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:16 smithi114 ceph-mon[30820]: pgmap v1070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:16 smithi116 ceph-mon[35411]: pgmap v1070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:18 smithi114 ceph-mon[30820]: pgmap v1071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:18 smithi116 ceph-mon[35411]: pgmap v1071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:20 smithi114 ceph-mon[30820]: pgmap v1072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:20 smithi116 ceph-mon[35411]: pgmap v1072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:22 smithi114 ceph-mon[30820]: pgmap v1073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:13:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:22 smithi116 ceph-mon[35411]: pgmap v1073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:13:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:24 smithi114 ceph-mon[30820]: pgmap v1074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:24 smithi116 ceph-mon[35411]: pgmap v1074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:26 smithi114 ceph-mon[30820]: pgmap v1075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:26 smithi116 ceph-mon[35411]: pgmap v1075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:28 smithi114 ceph-mon[30820]: pgmap v1076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:28 smithi116 ceph-mon[35411]: pgmap v1076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:30 smithi114 ceph-mon[30820]: pgmap v1077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:30 smithi116 ceph-mon[35411]: pgmap v1077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:32 smithi114 ceph-mon[30820]: pgmap v1078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:13:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:32 smithi116 ceph-mon[35411]: pgmap v1078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:13:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:34 smithi114 ceph-mon[30820]: pgmap v1079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:34 smithi116 ceph-mon[35411]: pgmap v1079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:36 smithi114 ceph-mon[30820]: pgmap v1080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:36 smithi116 ceph-mon[35411]: pgmap v1080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:38 smithi114 ceph-mon[30820]: pgmap v1081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:38 smithi116 ceph-mon[35411]: pgmap v1081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:40 smithi114 ceph-mon[30820]: pgmap v1082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:40 smithi116 ceph-mon[35411]: pgmap v1082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:42 smithi114 ceph-mon[30820]: pgmap v1083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:13:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:42 smithi116 ceph-mon[35411]: pgmap v1083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:13:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:44 smithi114 ceph-mon[30820]: pgmap v1084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:44 smithi116 ceph-mon[35411]: pgmap v1084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:46 smithi114 ceph-mon[30820]: pgmap v1085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:46 smithi116 ceph-mon[35411]: pgmap v1085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:48 smithi114 ceph-mon[30820]: pgmap v1086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:48 smithi116 ceph-mon[35411]: pgmap v1086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:50 smithi114 ceph-mon[30820]: pgmap v1087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:50 smithi116 ceph-mon[35411]: pgmap v1087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:52 smithi114 ceph-mon[30820]: pgmap v1088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:13:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:52 smithi116 ceph-mon[35411]: pgmap v1088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:13:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:54 smithi114 ceph-mon[30820]: pgmap v1089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:54 smithi116 ceph-mon[35411]: pgmap v1089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:13:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:56 smithi114 ceph-mon[30820]: pgmap v1090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:56 smithi116 ceph-mon[35411]: pgmap v1090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:13:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:13:58 smithi114 ceph-mon[30820]: pgmap v1091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:13:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:13:58 smithi116 ceph-mon[35411]: pgmap v1091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:00 smithi114 ceph-mon[30820]: pgmap v1092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:00 smithi116 ceph-mon[35411]: pgmap v1092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:14:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:14:02.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:14:02.132+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:14:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:14:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:14:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:02 smithi116 ceph-mon[35411]: pgmap v1093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:14:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:14:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:14:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-mon[30820]: pgmap v1093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:14:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:14:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:14:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:14:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:14:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:14:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:14:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:14:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:14:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:14:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:14:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:14:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:14:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:03 smithi116 ceph-mon[35411]: pgmap v1094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:14:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:03 smithi116 ceph-mon[35411]: pgmap v1095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:14:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:14:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:14:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:14:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:14:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:14:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:14:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:14:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:14:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:14:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:03 smithi114 ceph-mon[30820]: pgmap v1094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:14:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:03 smithi114 ceph-mon[30820]: pgmap v1095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:14:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:05 smithi116 ceph-mon[35411]: pgmap v1096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:14:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:05 smithi114 ceph-mon[30820]: pgmap v1096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:14:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:07 smithi114 ceph-mon[30820]: pgmap v1097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:14:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:07 smithi116 ceph-mon[35411]: pgmap v1097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:14:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:08 smithi114 ceph-mon[30820]: pgmap v1098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:14:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:08 smithi116 ceph-mon[35411]: pgmap v1098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:14:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:11 smithi114 ceph-mon[30820]: pgmap v1099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:14:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:11 smithi116 ceph-mon[35411]: pgmap v1099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:14:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:13 smithi114 ceph-mon[30820]: pgmap v1100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:14:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:13 smithi116 ceph-mon[35411]: pgmap v1100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:14:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:15 smithi114 ceph-mon[30820]: pgmap v1101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:15 smithi116 ceph-mon[35411]: pgmap v1101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:17.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:17 smithi114 ceph-mon[30820]: pgmap v1102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:17 smithi116 ceph-mon[35411]: pgmap v1102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:19.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:19 smithi116 ceph-mon[35411]: pgmap v1103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:19 smithi114 ceph-mon[30820]: pgmap v1103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:21 smithi114 ceph-mon[30820]: pgmap v1104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:14:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:21 smithi116 ceph-mon[35411]: pgmap v1104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:14:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:23 smithi114 ceph-mon[30820]: pgmap v1105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:23 smithi116 ceph-mon[35411]: pgmap v1105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:25 smithi114 ceph-mon[30820]: pgmap v1106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:25 smithi116 ceph-mon[35411]: pgmap v1106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:27 smithi114 ceph-mon[30820]: pgmap v1107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:27 smithi116 ceph-mon[35411]: pgmap v1107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:29 smithi114 ceph-mon[30820]: pgmap v1108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:29 smithi116 ceph-mon[35411]: pgmap v1108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:31 smithi114 ceph-mon[30820]: pgmap v1109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:14:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:31 smithi116 ceph-mon[35411]: pgmap v1109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:14:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:33 smithi114 ceph-mon[30820]: pgmap v1110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:33 smithi116 ceph-mon[35411]: pgmap v1110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:35 smithi114 ceph-mon[30820]: pgmap v1111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:35 smithi116 ceph-mon[35411]: pgmap v1111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:37 smithi114 ceph-mon[30820]: pgmap v1112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:37 smithi116 ceph-mon[35411]: pgmap v1112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:39 smithi114 ceph-mon[30820]: pgmap v1113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:39 smithi116 ceph-mon[35411]: pgmap v1113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:41 smithi114 ceph-mon[30820]: pgmap v1114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:14:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:41 smithi116 ceph-mon[35411]: pgmap v1114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:14:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:43 smithi114 ceph-mon[30820]: pgmap v1115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:43.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:43 smithi116 ceph-mon[35411]: pgmap v1115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:45 smithi114 ceph-mon[30820]: pgmap v1116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:45 smithi116 ceph-mon[35411]: pgmap v1116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:47 smithi114 ceph-mon[30820]: pgmap v1117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:47 smithi116 ceph-mon[35411]: pgmap v1117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:49 smithi114 ceph-mon[30820]: pgmap v1118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:49 smithi116 ceph-mon[35411]: pgmap v1118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:51 smithi114 ceph-mon[30820]: pgmap v1119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:14:51.739 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:51 smithi116 ceph-mon[35411]: pgmap v1119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:14:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:53 smithi114 ceph-mon[30820]: pgmap v1120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:53 smithi116 ceph-mon[35411]: pgmap v1120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:14:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:55 smithi114 ceph-mon[30820]: pgmap v1121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:55 smithi116 ceph-mon[35411]: pgmap v1121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:57 smithi114 ceph-mon[30820]: pgmap v1122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:57 smithi116 ceph-mon[35411]: pgmap v1122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:14:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:14:59 smithi114 ceph-mon[30820]: pgmap v1123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:14:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:14:59 smithi116 ceph-mon[35411]: pgmap v1123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:01 smithi114 ceph-mon[30820]: pgmap v1124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:01 smithi116 ceph-mon[35411]: pgmap v1124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:15:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:15:03.574 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-mon[30820]: pgmap v1125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:03 smithi116 ceph-mon[35411]: pgmap v1125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:03.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:15:03.572+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:15:03.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:15:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:15:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:15:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:15:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:15:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:15:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:15:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:15:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:15:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:15:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:15:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:15:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:15:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:15:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:04 smithi114 ceph-mon[30820]: pgmap v1126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:15:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:04 smithi114 ceph-mon[30820]: pgmap v1127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:15:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:15:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:15:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:04 smithi116 ceph-mon[35411]: pgmap v1126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:04 smithi116 ceph-mon[35411]: pgmap v1127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:15:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:06 smithi114 ceph-mon[30820]: pgmap v1128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:06 smithi116 ceph-mon[35411]: pgmap v1128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:08 smithi114 ceph-mon[30820]: pgmap v1129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:08 smithi116 ceph-mon[35411]: pgmap v1129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:10 smithi114 ceph-mon[30820]: pgmap v1130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:10 smithi116 ceph-mon[35411]: pgmap v1130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:12 smithi114 ceph-mon[30820]: pgmap v1131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:12 smithi116 ceph-mon[35411]: pgmap v1131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:15:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:14 smithi114 ceph-mon[30820]: pgmap v1132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:15:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:14 smithi116 ceph-mon[35411]: pgmap v1132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:15:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:16 smithi114 ceph-mon[30820]: pgmap v1133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:16 smithi116 ceph-mon[35411]: pgmap v1133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:18 smithi114 ceph-mon[30820]: pgmap v1134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:18 smithi116 ceph-mon[35411]: pgmap v1134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:20 smithi114 ceph-mon[30820]: pgmap v1135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:20 smithi116 ceph-mon[35411]: pgmap v1135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:22 smithi114 ceph-mon[30820]: pgmap v1136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:22 smithi116 ceph-mon[35411]: pgmap v1136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:24 smithi116 ceph-mon[35411]: pgmap v1137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:24 smithi114 ceph-mon[30820]: pgmap v1137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:26 smithi116 ceph-mon[35411]: pgmap v1138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:26 smithi114 ceph-mon[30820]: pgmap v1138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:28 smithi116 ceph-mon[35411]: pgmap v1139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:28 smithi114 ceph-mon[30820]: pgmap v1139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:30 smithi116 ceph-mon[35411]: pgmap v1140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:30 smithi114 ceph-mon[30820]: pgmap v1140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:32 smithi114 ceph-mon[30820]: pgmap v1141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:32 smithi116 ceph-mon[35411]: pgmap v1141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:34 smithi114 ceph-mon[30820]: pgmap v1142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:34 smithi116 ceph-mon[35411]: pgmap v1142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:36 smithi114 ceph-mon[30820]: pgmap v1143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:36 smithi116 ceph-mon[35411]: pgmap v1143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:38 smithi114 ceph-mon[30820]: pgmap v1144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:38 smithi116 ceph-mon[35411]: pgmap v1144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:39 smithi114 ceph-mon[30820]: pgmap v1145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:39 smithi116 ceph-mon[35411]: pgmap v1145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:42 smithi114 ceph-mon[30820]: pgmap v1146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:42 smithi116 ceph-mon[35411]: pgmap v1146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:44 smithi114 ceph-mon[30820]: pgmap v1147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:44 smithi116 ceph-mon[35411]: pgmap v1147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:46 smithi114 ceph-mon[30820]: pgmap v1148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:46 smithi116 ceph-mon[35411]: pgmap v1148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:48 smithi114 ceph-mon[30820]: pgmap v1149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:48 smithi116 ceph-mon[35411]: pgmap v1149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:50 smithi114 ceph-mon[30820]: pgmap v1150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:50 smithi116 ceph-mon[35411]: pgmap v1150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:15:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:52 smithi114 ceph-mon[30820]: pgmap v1151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:52 smithi116 ceph-mon[35411]: pgmap v1151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:15:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:54 smithi116 ceph-mon[35411]: pgmap v1152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:54 smithi114 ceph-mon[30820]: pgmap v1152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:55 smithi114 ceph-mon[30820]: pgmap v1153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:55 smithi116 ceph-mon[35411]: pgmap v1153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:15:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:15:58 smithi114 ceph-mon[30820]: pgmap v1154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:15:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:15:58 smithi116 ceph-mon[35411]: pgmap v1154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:00 smithi114 ceph-mon[30820]: pgmap v1155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:00 smithi116 ceph-mon[35411]: pgmap v1155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:02 smithi114 ceph-mon[30820]: pgmap v1156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:02 smithi116 ceph-mon[35411]: pgmap v1156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:04 smithi114 ceph-mon[30820]: pgmap v1157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:16:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:04 smithi116 ceph-mon[35411]: pgmap v1157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:16:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:16:05.012+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:16:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:16:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:16:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:16:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:16:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:16:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:16:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:16:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:16:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:16:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:16:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:16:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:16:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:16:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:16:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:16:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:16:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:16:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:06 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:16:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:16:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:16:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:16:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:16:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:16:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:16:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:16:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:16:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:16:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:16:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:06 smithi116 ceph-mon[35411]: pgmap v1158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:16:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:06 smithi116 ceph-mon[35411]: pgmap v1159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:06 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:16:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:16:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:16:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:06 smithi114 ceph-mon[30820]: pgmap v1158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:16:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:06 smithi114 ceph-mon[30820]: pgmap v1159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:16:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:08 smithi116 ceph-mon[35411]: pgmap v1160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:16:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:08 smithi114 ceph-mon[30820]: pgmap v1160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:16:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:10 smithi116 ceph-mon[35411]: pgmap v1161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:16:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:10 smithi114 ceph-mon[30820]: pgmap v1161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:16:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:12 smithi114 ceph-mon[30820]: pgmap v1162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:16:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:12 smithi116 ceph-mon[35411]: pgmap v1162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:16:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:14 smithi114 ceph-mon[30820]: pgmap v1163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:16:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:14 smithi116 ceph-mon[35411]: pgmap v1163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:16:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:16 smithi114 ceph-mon[30820]: pgmap v1164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:16:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:16 smithi116 ceph-mon[35411]: pgmap v1164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:16:18.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:18 smithi114 ceph-mon[30820]: pgmap v1165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:18 smithi116 ceph-mon[35411]: pgmap v1165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:20.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:20 smithi116 ceph-mon[35411]: pgmap v1166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:20 smithi114 ceph-mon[30820]: pgmap v1166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:22 smithi114 ceph-mon[30820]: pgmap v1167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:16:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:22 smithi116 ceph-mon[35411]: pgmap v1167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:16:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:24 smithi114 ceph-mon[30820]: pgmap v1168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:24 smithi116 ceph-mon[35411]: pgmap v1168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:26 smithi114 ceph-mon[30820]: pgmap v1169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:26 smithi116 ceph-mon[35411]: pgmap v1169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:28 smithi114 ceph-mon[30820]: pgmap v1170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:28 smithi116 ceph-mon[35411]: pgmap v1170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:29 smithi114 ceph-mon[30820]: pgmap v1171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:29 smithi116 ceph-mon[35411]: pgmap v1171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:32 smithi114 ceph-mon[30820]: pgmap v1172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:16:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:32 smithi116 ceph-mon[35411]: pgmap v1172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:16:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:34 smithi114 ceph-mon[30820]: pgmap v1173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:34 smithi116 ceph-mon[35411]: pgmap v1173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:36 smithi114 ceph-mon[30820]: pgmap v1174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:36 smithi116 ceph-mon[35411]: pgmap v1174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:38 smithi114 ceph-mon[30820]: pgmap v1175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:38 smithi116 ceph-mon[35411]: pgmap v1175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:40 smithi114 ceph-mon[30820]: pgmap v1176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:40 smithi116 ceph-mon[35411]: pgmap v1176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:42 smithi114 ceph-mon[30820]: pgmap v1177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:16:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:42 smithi116 ceph-mon[35411]: pgmap v1177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:16:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:44 smithi114 ceph-mon[30820]: pgmap v1178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:44 smithi116 ceph-mon[35411]: pgmap v1178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:46 smithi114 ceph-mon[30820]: pgmap v1179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:46 smithi116 ceph-mon[35411]: pgmap v1179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:48 smithi114 ceph-mon[30820]: pgmap v1180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:48 smithi116 ceph-mon[35411]: pgmap v1180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:50 smithi114 ceph-mon[30820]: pgmap v1181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:50 smithi116 ceph-mon[35411]: pgmap v1181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:52 smithi114 ceph-mon[30820]: pgmap v1182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:16:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:52 smithi116 ceph-mon[35411]: pgmap v1182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:16:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:54 smithi114 ceph-mon[30820]: pgmap v1183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:54 smithi116 ceph-mon[35411]: pgmap v1183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:16:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:56 smithi114 ceph-mon[30820]: pgmap v1184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:56 smithi116 ceph-mon[35411]: pgmap v1184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:16:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:16:58 smithi114 ceph-mon[30820]: pgmap v1185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:16:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:16:58 smithi116 ceph-mon[35411]: pgmap v1185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:00 smithi114 ceph-mon[30820]: pgmap v1186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:00 smithi116 ceph-mon[35411]: pgmap v1186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:02 smithi114 ceph-mon[30820]: pgmap v1187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:02 smithi116 ceph-mon[35411]: pgmap v1187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:04 smithi114 ceph-mon[30820]: pgmap v1188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:17:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:04 smithi116 ceph-mon[35411]: pgmap v1188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:17:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:17:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:17:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-mon[30820]: pgmap v1189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:17:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:06 smithi116 ceph-mon[35411]: pgmap v1189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:17:06.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:17:06.453+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:17:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:17:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:17:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:17:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:17:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:17:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:17:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:17:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:17:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:07 smithi114 ceph-mon[30820]: pgmap v1190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:07 smithi114 ceph-mon[30820]: pgmap v1191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:17:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:17:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:17:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:17:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:17:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:17:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:17:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:17:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:17:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:17:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:17:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:17:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:17:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:17:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:07 smithi116 ceph-mon[35411]: pgmap v1190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T07:17:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:07 smithi116 ceph-mon[35411]: pgmap v1191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:17:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:17:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:09 smithi114 ceph-mon[30820]: pgmap v1192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:17:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:09 smithi116 ceph-mon[35411]: pgmap v1192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:17:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:11 smithi114 ceph-mon[30820]: pgmap v1193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:17:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:11 smithi116 ceph-mon[35411]: pgmap v1193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:17:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:13 smithi114 ceph-mon[30820]: pgmap v1194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:17:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:13 smithi116 ceph-mon[35411]: pgmap v1194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:17:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:15 smithi114 ceph-mon[30820]: pgmap v1195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:17:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:15 smithi116 ceph-mon[35411]: pgmap v1195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:17:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:17 smithi114 ceph-mon[30820]: pgmap v1196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:17:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:17 smithi116 ceph-mon[35411]: pgmap v1196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:17:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:19 smithi114 ceph-mon[30820]: pgmap v1197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:19 smithi116 ceph-mon[35411]: pgmap v1197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:21 smithi114 ceph-mon[30820]: pgmap v1198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:21 smithi116 ceph-mon[35411]: pgmap v1198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:23 smithi114 ceph-mon[30820]: pgmap v1199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:23 smithi116 ceph-mon[35411]: pgmap v1199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:25 smithi114 ceph-mon[30820]: pgmap v1200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:17:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:25 smithi116 ceph-mon[35411]: pgmap v1200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:17:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:27 smithi114 ceph-mon[30820]: pgmap v1201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:27 smithi116 ceph-mon[35411]: pgmap v1201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:29 smithi114 ceph-mon[30820]: pgmap v1202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:29 smithi116 ceph-mon[35411]: pgmap v1202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:31 smithi114 ceph-mon[30820]: pgmap v1203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:31 smithi116 ceph-mon[35411]: pgmap v1203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:33 smithi114 ceph-mon[30820]: pgmap v1204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:33 smithi116 ceph-mon[35411]: pgmap v1204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:35 smithi114 ceph-mon[30820]: pgmap v1205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:17:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:35 smithi116 ceph-mon[35411]: pgmap v1205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:17:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:37 smithi114 ceph-mon[30820]: pgmap v1206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:37 smithi116 ceph-mon[35411]: pgmap v1206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:39 smithi114 ceph-mon[30820]: pgmap v1207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:39 smithi116 ceph-mon[35411]: pgmap v1207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:41 smithi114 ceph-mon[30820]: pgmap v1208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:41 smithi116 ceph-mon[35411]: pgmap v1208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:43 smithi114 ceph-mon[30820]: pgmap v1209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:43 smithi116 ceph-mon[35411]: pgmap v1209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:45 smithi114 ceph-mon[30820]: pgmap v1210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:17:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:45 smithi116 ceph-mon[35411]: pgmap v1210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:17:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:47 smithi114 ceph-mon[30820]: pgmap v1211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:47 smithi116 ceph-mon[35411]: pgmap v1211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:49 smithi114 ceph-mon[30820]: pgmap v1212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:49 smithi116 ceph-mon[35411]: pgmap v1212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:51 smithi114 ceph-mon[30820]: pgmap v1213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:51 smithi116 ceph-mon[35411]: pgmap v1213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:17:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:53 smithi114 ceph-mon[30820]: pgmap v1214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:53 smithi116 ceph-mon[35411]: pgmap v1214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:55 smithi114 ceph-mon[30820]: pgmap v1215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:17:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:55 smithi116 ceph-mon[35411]: pgmap v1215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:17:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:57 smithi114 ceph-mon[30820]: pgmap v1216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:57 smithi116 ceph-mon[35411]: pgmap v1216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:17:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:17:59 smithi114 ceph-mon[30820]: pgmap v1217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:17:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:17:59 smithi116 ceph-mon[35411]: pgmap v1217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:01 smithi114 ceph-mon[30820]: pgmap v1218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:01 smithi116 ceph-mon[35411]: pgmap v1218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:03 smithi114 ceph-mon[30820]: pgmap v1219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:03 smithi116 ceph-mon[35411]: pgmap v1219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:05 smithi114 ceph-mon[30820]: pgmap v1220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:18:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:05 smithi116 ceph-mon[35411]: pgmap v1220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:18:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:18:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:18:07.894 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-mon[30820]: pgmap v1221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:18:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:07 smithi116 ceph-mon[35411]: pgmap v1221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:18:08.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:18:07.893+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:18:08.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:18:08.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:18:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:18:09.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:09 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:18:09.746 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:18:09.747 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:09 smithi114 ceph-mon[30820]: pgmap v1222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:09.820 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:09 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:18:09.820 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:18:09.820 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:18:09.820 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:18:09.820 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:18:09.820 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:18:09.820 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:18:09.820 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:18:09.821 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:18:09.822 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:18:09.822 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:18:09.822 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:09 smithi116 ceph-mon[35411]: pgmap v1222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:11.290 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:11.290 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:11.290 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:11 smithi114 ceph-mon[30820]: pgmap v1223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:11 smithi116 ceph-mon[35411]: pgmap v1223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:18:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:18:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:12 smithi114 ceph-mon[30820]: pgmap v1224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:18:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:12 smithi114 ceph-mon[30820]: pgmap v1225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:18:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:18:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:18:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:12 smithi116 ceph-mon[35411]: pgmap v1224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:18:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:12 smithi116 ceph-mon[35411]: pgmap v1225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:18:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:18:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:14 smithi114 ceph-mon[30820]: pgmap v1226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T07:18:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:14 smithi116 ceph-mon[35411]: pgmap v1226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T07:18:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:16 smithi114 ceph-mon[30820]: pgmap v1227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:18:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:16 smithi116 ceph-mon[35411]: pgmap v1227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:18:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:18 smithi114 ceph-mon[30820]: pgmap v1228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:18:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:18 smithi116 ceph-mon[35411]: pgmap v1228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:18:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:20 smithi114 ceph-mon[30820]: pgmap v1229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-09-19T07:18:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:20 smithi116 ceph-mon[35411]: pgmap v1229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-09-19T07:18:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:22 smithi114 ceph-mon[30820]: pgmap v1230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:18:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:22 smithi116 ceph-mon[35411]: pgmap v1230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:18:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:24 smithi114 ceph-mon[30820]: pgmap v1231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:24 smithi116 ceph-mon[35411]: pgmap v1231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:26 smithi114 ceph-mon[30820]: pgmap v1232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:26 smithi116 ceph-mon[35411]: pgmap v1232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:28 smithi114 ceph-mon[30820]: pgmap v1233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:28 smithi116 ceph-mon[35411]: pgmap v1233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:30 smithi114 ceph-mon[30820]: pgmap v1234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:18:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:30 smithi116 ceph-mon[35411]: pgmap v1234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:18:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:32 smithi114 ceph-mon[30820]: pgmap v1235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:18:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:32 smithi116 ceph-mon[35411]: pgmap v1235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:18:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:34 smithi114 ceph-mon[30820]: pgmap v1236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:34 smithi116 ceph-mon[35411]: pgmap v1236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:36 smithi114 ceph-mon[30820]: pgmap v1237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:36 smithi116 ceph-mon[35411]: pgmap v1237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:38 smithi114 ceph-mon[30820]: pgmap v1238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:38 smithi116 ceph-mon[35411]: pgmap v1238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:40 smithi114 ceph-mon[30820]: pgmap v1239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:18:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:40 smithi116 ceph-mon[35411]: pgmap v1239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:18:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:42 smithi114 ceph-mon[30820]: pgmap v1240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:18:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:42 smithi116 ceph-mon[35411]: pgmap v1240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:18:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:44 smithi114 ceph-mon[30820]: pgmap v1241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:44 smithi116 ceph-mon[35411]: pgmap v1241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:46 smithi116 ceph-mon[35411]: pgmap v1242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:46 smithi114 ceph-mon[30820]: pgmap v1242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:48 smithi116 ceph-mon[35411]: pgmap v1243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:48 smithi114 ceph-mon[30820]: pgmap v1243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:50 smithi116 ceph-mon[35411]: pgmap v1244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:18:50.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:50 smithi114 ceph-mon[30820]: pgmap v1244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:18:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:52 smithi114 ceph-mon[30820]: pgmap v1245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:18:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:52 smithi116 ceph-mon[35411]: pgmap v1245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:18:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:54 smithi114 ceph-mon[30820]: pgmap v1246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:54 smithi116 ceph-mon[35411]: pgmap v1246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:56 smithi114 ceph-mon[30820]: pgmap v1247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:56 smithi116 ceph-mon[35411]: pgmap v1247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:18:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:18:58 smithi114 ceph-mon[30820]: pgmap v1248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:18:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:18:58 smithi116 ceph-mon[35411]: pgmap v1248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:00 smithi114 ceph-mon[30820]: pgmap v1249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:19:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:00 smithi116 ceph-mon[35411]: pgmap v1249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:19:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:02 smithi114 ceph-mon[30820]: pgmap v1250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:02 smithi116 ceph-mon[35411]: pgmap v1250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:04 smithi114 ceph-mon[30820]: pgmap v1251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:19:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:04 smithi116 ceph-mon[35411]: pgmap v1251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:19:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:06 smithi114 ceph-mon[30820]: pgmap v1252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:19:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:06 smithi116 ceph-mon[35411]: pgmap v1252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:19:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:08 smithi114 ceph-mon[30820]: pgmap v1253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:19:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:08 smithi116 ceph-mon[35411]: pgmap v1253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:19:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:10 smithi114 ceph-mon[30820]: pgmap v1254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:19:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:10 smithi116 ceph-mon[35411]: pgmap v1254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:19:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:19:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:19:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:19:12.405+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:19:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:19:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:19:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:19:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:19:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:19:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:19:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:19:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:19:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:19:12.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:19:12.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:19:12.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:19:12.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:19:12.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:19:12.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:19:12.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:19:12.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:19:12.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:19:12.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:19:12.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:19:12.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:19:12.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:19:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: pgmap v1255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:19:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:19:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:19:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: pgmap v1255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:19:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:19:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:19:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:19:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:13 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:19:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:13 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:19:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:13 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:13 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:13 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:13 smithi114 ceph-mon[30820]: pgmap v1256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T07:19:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:13 smithi114 ceph-mon[30820]: pgmap v1257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:13 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:13 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:13 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:13 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:19:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:19:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:19:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:19:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:19:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:13 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:19:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:13 smithi116 ceph-mon[35411]: pgmap v1256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T07:19:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:13 smithi116 ceph-mon[35411]: pgmap v1257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:19:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:15 smithi114 ceph-mon[30820]: pgmap v1258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-09-19T07:19:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:15 smithi116 ceph-mon[35411]: pgmap v1258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-09-19T07:19:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:17 smithi114 ceph-mon[30820]: pgmap v1259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:19:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:17 smithi116 ceph-mon[35411]: pgmap v1259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:19:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:19 smithi114 ceph-mon[30820]: pgmap v1260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 5 op/s 2024-09-19T07:19:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:19 smithi116 ceph-mon[35411]: pgmap v1260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 5 op/s 2024-09-19T07:19:21.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:20 smithi114 ceph-mon[30820]: pgmap v1261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:19:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:20 smithi116 ceph-mon[35411]: pgmap v1261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:19:23.881 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:23 smithi116 ceph-mon[35411]: pgmap v1262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:19:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:23 smithi114 ceph-mon[30820]: pgmap v1262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:19:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:25 smithi114 ceph-mon[30820]: pgmap v1263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:19:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:25 smithi116 ceph-mon[35411]: pgmap v1263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:19:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:27 smithi114 ceph-mon[30820]: pgmap v1264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:27 smithi116 ceph-mon[35411]: pgmap v1264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:29 smithi114 ceph-mon[30820]: pgmap v1265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:29 smithi116 ceph-mon[35411]: pgmap v1265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:31 smithi114 ceph-mon[30820]: pgmap v1266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:19:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:31 smithi116 ceph-mon[35411]: pgmap v1266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:19:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:33 smithi114 ceph-mon[30820]: pgmap v1267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:33 smithi116 ceph-mon[35411]: pgmap v1267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:35 smithi114 ceph-mon[30820]: pgmap v1268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:19:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:35 smithi116 ceph-mon[35411]: pgmap v1268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:19:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:37 smithi114 ceph-mon[30820]: pgmap v1269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:37 smithi116 ceph-mon[35411]: pgmap v1269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:39 smithi114 ceph-mon[30820]: pgmap v1270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:19:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:39 smithi116 ceph-mon[35411]: pgmap v1270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:19:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:41 smithi114 ceph-mon[30820]: pgmap v1271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:19:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:41 smithi116 ceph-mon[35411]: pgmap v1271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:19:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:43 smithi114 ceph-mon[30820]: pgmap v1272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:19:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:43 smithi116 ceph-mon[35411]: pgmap v1272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:19:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:45 smithi114 ceph-mon[30820]: pgmap v1273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:19:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:45 smithi116 ceph-mon[35411]: pgmap v1273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:19:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:47 smithi114 ceph-mon[30820]: pgmap v1274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:47 smithi116 ceph-mon[35411]: pgmap v1274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:49 smithi114 ceph-mon[30820]: pgmap v1275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:49 smithi116 ceph-mon[35411]: pgmap v1275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:51 smithi114 ceph-mon[30820]: pgmap v1276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:19:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:51 smithi116 ceph-mon[35411]: pgmap v1276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:19:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:53 smithi114 ceph-mon[30820]: pgmap v1277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:53 smithi116 ceph-mon[35411]: pgmap v1277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:55 smithi114 ceph-mon[30820]: pgmap v1278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:19:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:55 smithi116 ceph-mon[35411]: pgmap v1278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:19:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:57 smithi114 ceph-mon[30820]: pgmap v1279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:57 smithi116 ceph-mon[35411]: pgmap v1279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:19:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:19:59 smithi114 ceph-mon[30820]: pgmap v1280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:19:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:19:59 smithi116 ceph-mon[35411]: pgmap v1280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:20:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:20:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:01 smithi114 ceph-mon[30820]: pgmap v1281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:01 smithi116 ceph-mon[35411]: pgmap v1281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:03 smithi114 ceph-mon[30820]: pgmap v1282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:03 smithi116 ceph-mon[35411]: pgmap v1282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:05 smithi114 ceph-mon[30820]: pgmap v1283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:20:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:05 smithi116 ceph-mon[35411]: pgmap v1283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:20:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:07 smithi114 ceph-mon[30820]: pgmap v1284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:20:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:07 smithi116 ceph-mon[35411]: pgmap v1284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:20:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:09 smithi114 ceph-mon[30820]: pgmap v1285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:09 smithi116 ceph-mon[35411]: pgmap v1285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:11 smithi114 ceph-mon[30820]: pgmap v1286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:11 smithi116 ceph-mon[35411]: pgmap v1286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:20:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:20:13.846 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-mon[30820]: pgmap v1287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:13 smithi116 ceph-mon[35411]: pgmap v1287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:14.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:20:13.844+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:20:14.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:20:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:20:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:20:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:20:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:20:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:20:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:20:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:20:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:20:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:20:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:20:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:20:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:20:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:20:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:20:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:20:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:15 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:20:15.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:20:15.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:20:15.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:15 smithi114 ceph-mon[30820]: pgmap v1288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:20:15.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:15 smithi114 ceph-mon[30820]: pgmap v1289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:15 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:20:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:20:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:15 smithi116 ceph-mon[35411]: pgmap v1288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:20:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:15 smithi116 ceph-mon[35411]: pgmap v1289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:20:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:17 smithi114 ceph-mon[30820]: pgmap v1290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:20:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:17 smithi116 ceph-mon[35411]: pgmap v1290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:20:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:19 smithi114 ceph-mon[30820]: pgmap v1291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:20:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:19 smithi116 ceph-mon[35411]: pgmap v1291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:20:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:21 smithi114 ceph-mon[30820]: pgmap v1292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:20:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:21 smithi116 ceph-mon[35411]: pgmap v1292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:20:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:23 smithi114 ceph-mon[30820]: pgmap v1293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:20:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:23 smithi116 ceph-mon[35411]: pgmap v1293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:20:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:25 smithi114 ceph-mon[30820]: pgmap v1294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:20:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:25 smithi116 ceph-mon[35411]: pgmap v1294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:20:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:27 smithi114 ceph-mon[30820]: pgmap v1295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:27 smithi116 ceph-mon[35411]: pgmap v1295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:29 smithi114 ceph-mon[30820]: pgmap v1296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:29 smithi116 ceph-mon[35411]: pgmap v1296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:31 smithi114 ceph-mon[30820]: pgmap v1297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:20:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:31 smithi116 ceph-mon[35411]: pgmap v1297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:20:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:33 smithi114 ceph-mon[30820]: pgmap v1298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:20:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:33 smithi116 ceph-mon[35411]: pgmap v1298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:20:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:35 smithi114 ceph-mon[30820]: pgmap v1299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:35 smithi116 ceph-mon[35411]: pgmap v1299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:37 smithi114 ceph-mon[30820]: pgmap v1300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:37 smithi116 ceph-mon[35411]: pgmap v1300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:39 smithi114 ceph-mon[30820]: pgmap v1301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:39 smithi116 ceph-mon[35411]: pgmap v1301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:41 smithi114 ceph-mon[30820]: pgmap v1302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:20:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:41 smithi116 ceph-mon[35411]: pgmap v1302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:20:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:43 smithi114 ceph-mon[30820]: pgmap v1303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:20:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:43 smithi116 ceph-mon[35411]: pgmap v1303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:20:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:45 smithi114 ceph-mon[30820]: pgmap v1304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:45 smithi116 ceph-mon[35411]: pgmap v1304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:47 smithi114 ceph-mon[30820]: pgmap v1305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:47 smithi116 ceph-mon[35411]: pgmap v1305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:49 smithi114 ceph-mon[30820]: pgmap v1306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:49 smithi116 ceph-mon[35411]: pgmap v1306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:51 smithi114 ceph-mon[30820]: pgmap v1307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:20:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:51 smithi116 ceph-mon[35411]: pgmap v1307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:20:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:53 smithi114 ceph-mon[30820]: pgmap v1308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:20:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:53 smithi116 ceph-mon[35411]: pgmap v1308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:20:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:55 smithi114 ceph-mon[30820]: pgmap v1309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:55 smithi116 ceph-mon[35411]: pgmap v1309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:20:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:57 smithi114 ceph-mon[30820]: pgmap v1310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:57 smithi116 ceph-mon[35411]: pgmap v1310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:20:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:20:59 smithi116 ceph-mon[35411]: pgmap v1311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:20:59 smithi114 ceph-mon[30820]: pgmap v1311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:00 smithi114 ceph-mon[30820]: pgmap v1312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:00 smithi116 ceph-mon[35411]: pgmap v1312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:02 smithi116 ceph-mon[35411]: pgmap v1313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:02 smithi114 ceph-mon[30820]: pgmap v1313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:04 smithi116 ceph-mon[35411]: pgmap v1314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:04 smithi114 ceph-mon[30820]: pgmap v1314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:06 smithi114 ceph-mon[30820]: pgmap v1315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:21:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:06 smithi116 ceph-mon[35411]: pgmap v1315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:21:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:08 smithi114 ceph-mon[30820]: pgmap v1316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:08 smithi116 ceph-mon[35411]: pgmap v1316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:10 smithi114 ceph-mon[30820]: pgmap v1317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:10 smithi116 ceph-mon[35411]: pgmap v1317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:12 smithi114 ceph-mon[30820]: pgmap v1318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:12 smithi116 ceph-mon[35411]: pgmap v1318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:15.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:14 smithi114 ceph-mon[30820]: pgmap v1319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:15.286 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:21:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:14 smithi116 ceph-mon[35411]: pgmap v1319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:21:15.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:21:15.284+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:21:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:21:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:21:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:21:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:21:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:21:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-mon[30820]: pgmap v1320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:21:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-mon[30820]: pgmap v1321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:21:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:21:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:15 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:21:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:21:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:21:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:21:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:21:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:21:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:15 smithi116 ceph-mon[35411]: pgmap v1320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:21:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:15 smithi116 ceph-mon[35411]: pgmap v1321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:21:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:21:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:18 smithi114 ceph-mon[30820]: pgmap v1322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:21:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:18 smithi116 ceph-mon[35411]: pgmap v1322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:21:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:20 smithi114 ceph-mon[30820]: pgmap v1323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:21:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:20 smithi116 ceph-mon[35411]: pgmap v1323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:21:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:22 smithi114 ceph-mon[30820]: pgmap v1324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:21:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:22 smithi116 ceph-mon[35411]: pgmap v1324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:21:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:24 smithi114 ceph-mon[30820]: pgmap v1325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:21:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:24 smithi116 ceph-mon[35411]: pgmap v1325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:21:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:26 smithi114 ceph-mon[30820]: pgmap v1326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:21:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:26 smithi116 ceph-mon[35411]: pgmap v1326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:21:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:28 smithi114 ceph-mon[30820]: pgmap v1327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:21:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:28 smithi116 ceph-mon[35411]: pgmap v1327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:21:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:30 smithi114 ceph-mon[30820]: pgmap v1328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:30 smithi116 ceph-mon[35411]: pgmap v1328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:32 smithi114 ceph-mon[30820]: pgmap v1329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:32 smithi116 ceph-mon[35411]: pgmap v1329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:34 smithi114 ceph-mon[30820]: pgmap v1330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:34.922 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:34 smithi116 ceph-mon[35411]: pgmap v1330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:36 smithi114 ceph-mon[30820]: pgmap v1331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:21:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:36 smithi116 ceph-mon[35411]: pgmap v1331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:21:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:38 smithi114 ceph-mon[30820]: pgmap v1332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:21:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:38 smithi116 ceph-mon[35411]: pgmap v1332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:21:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:40 smithi114 ceph-mon[30820]: pgmap v1333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:40 smithi116 ceph-mon[35411]: pgmap v1333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:42 smithi114 ceph-mon[30820]: pgmap v1334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:42 smithi116 ceph-mon[35411]: pgmap v1334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:44 smithi114 ceph-mon[30820]: pgmap v1335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:44 smithi116 ceph-mon[35411]: pgmap v1335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:46 smithi114 ceph-mon[30820]: pgmap v1336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:21:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:46 smithi116 ceph-mon[35411]: pgmap v1336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:21:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:48 smithi114 ceph-mon[30820]: pgmap v1337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:21:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:48 smithi116 ceph-mon[35411]: pgmap v1337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:21:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:50 smithi114 ceph-mon[30820]: pgmap v1338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:50 smithi116 ceph-mon[35411]: pgmap v1338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:21:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:52 smithi114 ceph-mon[30820]: pgmap v1339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:52 smithi116 ceph-mon[35411]: pgmap v1339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:21:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:54 smithi114 ceph-mon[30820]: pgmap v1340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:54 smithi116 ceph-mon[35411]: pgmap v1340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:21:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:55 smithi114 ceph-mon[30820]: pgmap v1341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:21:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:55 smithi116 ceph-mon[35411]: pgmap v1341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:21:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:21:58 smithi114 ceph-mon[30820]: pgmap v1342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:21:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:21:58 smithi116 ceph-mon[35411]: pgmap v1342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:22:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:00 smithi114 ceph-mon[30820]: pgmap v1343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:00 smithi116 ceph-mon[35411]: pgmap v1343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:02 smithi114 ceph-mon[30820]: pgmap v1344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:02 smithi116 ceph-mon[35411]: pgmap v1344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:04 smithi114 ceph-mon[30820]: pgmap v1345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:04 smithi116 ceph-mon[35411]: pgmap v1345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:06 smithi114 ceph-mon[30820]: pgmap v1346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:22:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:06 smithi116 ceph-mon[35411]: pgmap v1346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:22:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:08 smithi114 ceph-mon[30820]: pgmap v1347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:22:08.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:08 smithi116 ceph-mon[35411]: pgmap v1347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:22:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:10 smithi114 ceph-mon[30820]: pgmap v1348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:10 smithi116 ceph-mon[35411]: pgmap v1348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:12 smithi114 ceph-mon[30820]: pgmap v1349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:12.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:12 smithi116 ceph-mon[35411]: pgmap v1349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:14 smithi114 ceph-mon[30820]: pgmap v1350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:14 smithi116 ceph-mon[35411]: pgmap v1350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-mon[30820]: pgmap v1351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:22:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:22:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:16 smithi116 ceph-mon[35411]: pgmap v1351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:22:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:22:16.725+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:22:17.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:22:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:22:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:22:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:22:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:22:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:22:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:22:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:22:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:22:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:22:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:22:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:22:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:22:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:22:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:22:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:22:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:22:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:22:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:22:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:18 smithi114 ceph-mon[30820]: pgmap v1352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:22:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:22:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:22:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:18 smithi116 ceph-mon[35411]: pgmap v1352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:22:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:22:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:22:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:22:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:19 smithi114 ceph-mon[30820]: pgmap v1353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T07:22:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:19 smithi114 ceph-mon[30820]: pgmap v1354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 5 op/s 2024-09-19T07:22:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:19 smithi116 ceph-mon[35411]: pgmap v1353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T07:22:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:19 smithi116 ceph-mon[35411]: pgmap v1354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 5 op/s 2024-09-19T07:22:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:21 smithi114 ceph-mon[30820]: pgmap v1355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-09-19T07:22:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:21 smithi116 ceph-mon[35411]: pgmap v1355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-09-19T07:22:23.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:23 smithi114 ceph-mon[30820]: pgmap v1356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:22:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:23 smithi116 ceph-mon[35411]: pgmap v1356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:22:25.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:25 smithi116 ceph-mon[35411]: pgmap v1357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-09-19T07:22:25.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:25 smithi114 ceph-mon[30820]: pgmap v1357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-09-19T07:22:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:27 smithi114 ceph-mon[30820]: pgmap v1358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:22:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:27 smithi116 ceph-mon[35411]: pgmap v1358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T07:22:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:29 smithi114 ceph-mon[30820]: pgmap v1359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:22:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:29 smithi116 ceph-mon[35411]: pgmap v1359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:22:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:31 smithi114 ceph-mon[30820]: pgmap v1360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:31 smithi116 ceph-mon[35411]: pgmap v1360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:33 smithi114 ceph-mon[30820]: pgmap v1361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:33 smithi116 ceph-mon[35411]: pgmap v1361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:35 smithi114 ceph-mon[30820]: pgmap v1362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:35 smithi116 ceph-mon[35411]: pgmap v1362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:37 smithi114 ceph-mon[30820]: pgmap v1363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:22:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:37 smithi116 ceph-mon[35411]: pgmap v1363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:22:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:39 smithi114 ceph-mon[30820]: pgmap v1364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:39 smithi116 ceph-mon[35411]: pgmap v1364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:41 smithi114 ceph-mon[30820]: pgmap v1365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:41 smithi116 ceph-mon[35411]: pgmap v1365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:43 smithi114 ceph-mon[30820]: pgmap v1366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:43 smithi116 ceph-mon[35411]: pgmap v1366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:45 smithi114 ceph-mon[30820]: pgmap v1367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:45 smithi116 ceph-mon[35411]: pgmap v1367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:47 smithi114 ceph-mon[30820]: pgmap v1368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:22:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:47 smithi116 ceph-mon[35411]: pgmap v1368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:22:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:49 smithi114 ceph-mon[30820]: pgmap v1369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:49 smithi116 ceph-mon[35411]: pgmap v1369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:51 smithi114 ceph-mon[30820]: pgmap v1370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:51.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:51 smithi116 ceph-mon[35411]: pgmap v1370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:22:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:53 smithi114 ceph-mon[30820]: pgmap v1371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:53 smithi116 ceph-mon[35411]: pgmap v1371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:22:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:55 smithi114 ceph-mon[30820]: pgmap v1372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:55 smithi116 ceph-mon[35411]: pgmap v1372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:57 smithi114 ceph-mon[30820]: pgmap v1373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:22:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:57 smithi116 ceph-mon[35411]: pgmap v1373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:22:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:22:59 smithi114 ceph-mon[30820]: pgmap v1374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:22:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:22:59 smithi116 ceph-mon[35411]: pgmap v1374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:01 smithi114 ceph-mon[30820]: pgmap v1375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:01 smithi116 ceph-mon[35411]: pgmap v1375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:03 smithi114 ceph-mon[30820]: pgmap v1376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:23:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:03 smithi116 ceph-mon[35411]: pgmap v1376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:23:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:05 smithi114 ceph-mon[30820]: pgmap v1377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:05 smithi116 ceph-mon[35411]: pgmap v1377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:07 smithi114 ceph-mon[30820]: pgmap v1378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:07 smithi116 ceph-mon[35411]: pgmap v1378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:09 smithi114 ceph-mon[30820]: pgmap v1379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:09 smithi116 ceph-mon[35411]: pgmap v1379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:11 smithi114 ceph-mon[30820]: pgmap v1380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:11 smithi116 ceph-mon[35411]: pgmap v1380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:13 smithi114 ceph-mon[30820]: pgmap v1381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:23:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:13 smithi116 ceph-mon[35411]: pgmap v1381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:23:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:15 smithi114 ceph-mon[30820]: pgmap v1382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:15 smithi116 ceph-mon[35411]: pgmap v1382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:17 smithi114 ceph-mon[30820]: pgmap v1383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:17 smithi116 ceph-mon[35411]: pgmap v1383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:18.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:23:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:23:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:19 smithi114 ceph-mon[30820]: pgmap v1384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:19 smithi116 ceph-mon[35411]: pgmap v1384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-mon[30820]: pgmap v1385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:23:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:23:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:21.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:23:21.492+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:23:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:23:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:23:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:23:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:21 smithi116 ceph-mon[35411]: pgmap v1385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:23:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:23:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:23:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:22 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:23:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:23:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:23:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:23:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:22 smithi114 ceph-mon[30820]: pgmap v1386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:23:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:22 smithi114 ceph-mon[30820]: pgmap v1387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:22 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:23:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:23:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:22 smithi116 ceph-mon[35411]: pgmap v1386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:23:22.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:22 smithi116 ceph-mon[35411]: pgmap v1387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:24 smithi114 ceph-mon[30820]: pgmap v1388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:24 smithi116 ceph-mon[35411]: pgmap v1388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:26 smithi114 ceph-mon[30820]: pgmap v1389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:26 smithi116 ceph-mon[35411]: pgmap v1389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:28 smithi114 ceph-mon[30820]: pgmap v1390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:28 smithi116 ceph-mon[35411]: pgmap v1390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:30 smithi114 ceph-mon[30820]: pgmap v1391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:30 smithi116 ceph-mon[35411]: pgmap v1391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:23:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:32 smithi114 ceph-mon[30820]: pgmap v1392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:23:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:32 smithi116 ceph-mon[35411]: pgmap v1392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:23:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:34 smithi114 ceph-mon[30820]: pgmap v1393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:34 smithi116 ceph-mon[35411]: pgmap v1393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:36 smithi114 ceph-mon[30820]: pgmap v1394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:36 smithi116 ceph-mon[35411]: pgmap v1394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:38 smithi114 ceph-mon[30820]: pgmap v1395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:38 smithi116 ceph-mon[35411]: pgmap v1395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:40 smithi114 ceph-mon[30820]: pgmap v1396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:40 smithi116 ceph-mon[35411]: pgmap v1396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:42 smithi114 ceph-mon[30820]: pgmap v1397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:23:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:42 smithi116 ceph-mon[35411]: pgmap v1397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:23:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:44 smithi114 ceph-mon[30820]: pgmap v1398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:44 smithi116 ceph-mon[35411]: pgmap v1398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:46 smithi114 ceph-mon[30820]: pgmap v1399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:46 smithi116 ceph-mon[35411]: pgmap v1399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:48 smithi114 ceph-mon[30820]: pgmap v1400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:48 smithi116 ceph-mon[35411]: pgmap v1400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:50 smithi114 ceph-mon[30820]: pgmap v1401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:50 smithi116 ceph-mon[35411]: pgmap v1401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:23:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:52 smithi114 ceph-mon[30820]: pgmap v1402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:23:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:52 smithi116 ceph-mon[35411]: pgmap v1402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:23:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:54 smithi116 ceph-mon[35411]: pgmap v1403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:54 smithi114 ceph-mon[30820]: pgmap v1403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:56 smithi116 ceph-mon[35411]: pgmap v1404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:56 smithi114 ceph-mon[30820]: pgmap v1404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:23:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:23:58 smithi116 ceph-mon[35411]: pgmap v1405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:23:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:23:58 smithi114 ceph-mon[30820]: pgmap v1405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:00 smithi114 ceph-mon[30820]: pgmap v1406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:00 smithi116 ceph-mon[35411]: pgmap v1406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:02 smithi114 ceph-mon[30820]: pgmap v1407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:03.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:02 smithi116 ceph-mon[35411]: pgmap v1407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:04 smithi114 ceph-mon[30820]: pgmap v1408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:04 smithi116 ceph-mon[35411]: pgmap v1408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:06 smithi114 ceph-mon[30820]: pgmap v1409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:24:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:06 smithi116 ceph-mon[35411]: pgmap v1409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:24:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:08 smithi114 ceph-mon[30820]: pgmap v1410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:08 smithi116 ceph-mon[35411]: pgmap v1410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:10 smithi114 ceph-mon[30820]: pgmap v1411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:10 smithi116 ceph-mon[35411]: pgmap v1411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:12 smithi114 ceph-mon[30820]: pgmap v1412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:12 smithi116 ceph-mon[35411]: pgmap v1412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:14 smithi114 ceph-mon[30820]: pgmap v1413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:14 smithi116 ceph-mon[35411]: pgmap v1413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:16 smithi114 ceph-mon[30820]: pgmap v1414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:24:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:16 smithi116 ceph-mon[35411]: pgmap v1414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:24:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:18 smithi114 ceph-mon[30820]: pgmap v1415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:18 smithi116 ceph-mon[35411]: pgmap v1415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:20 smithi114 ceph-mon[30820]: pgmap v1416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:20 smithi116 ceph-mon[35411]: pgmap v1416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:22.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:24:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:24:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:22 smithi114 ceph-mon[30820]: pgmap v1417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:23.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:22 smithi116 ceph-mon[35411]: pgmap v1417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-mon[30820]: pgmap v1418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:25.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:24:24.852+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:24:25.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:24:25.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:24:25.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:24:25.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:24:25.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:24:25.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:24:25.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:24:25.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:24:25.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:24:25.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:24:25.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:24:25.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:24:25.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:24:25.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:24:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:24:25.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:24 smithi116 ceph-mon[35411]: pgmap v1418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:26.079 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:24:26.079 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:24:26.079 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:24:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:24:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:24:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:24:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:24:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:24:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:24:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:24:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:24:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:24:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:24:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:26 smithi114 ceph-mon[30820]: pgmap v1419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:24:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:26 smithi114 ceph-mon[30820]: pgmap v1420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:24:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:24:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:26 smithi116 ceph-mon[35411]: pgmap v1419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:24:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:26 smithi116 ceph-mon[35411]: pgmap v1420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:24:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:28 smithi114 ceph-mon[30820]: pgmap v1421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:24:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:28 smithi116 ceph-mon[35411]: pgmap v1421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:24:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:30 smithi114 ceph-mon[30820]: pgmap v1422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:24:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:30 smithi116 ceph-mon[35411]: pgmap v1422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:24:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:32 smithi114 ceph-mon[30820]: pgmap v1423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:24:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:32 smithi116 ceph-mon[35411]: pgmap v1423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:24:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:34 smithi114 ceph-mon[30820]: pgmap v1424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:24:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:34 smithi116 ceph-mon[35411]: pgmap v1424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:24:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:36 smithi114 ceph-mon[30820]: pgmap v1425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:36 smithi116 ceph-mon[35411]: pgmap v1425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:24:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:38 smithi114 ceph-mon[30820]: pgmap v1426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:38 smithi116 ceph-mon[35411]: pgmap v1426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:40 smithi114 ceph-mon[30820]: pgmap v1427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:40 smithi116 ceph-mon[35411]: pgmap v1427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:42 smithi114 ceph-mon[30820]: pgmap v1428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:24:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:42 smithi116 ceph-mon[35411]: pgmap v1428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:24:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:44 smithi114 ceph-mon[30820]: pgmap v1429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:44 smithi116 ceph-mon[35411]: pgmap v1429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:46 smithi114 ceph-mon[30820]: pgmap v1430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:47.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:46 smithi116 ceph-mon[35411]: pgmap v1430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:48 smithi114 ceph-mon[30820]: pgmap v1431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:48 smithi116 ceph-mon[35411]: pgmap v1431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:50 smithi114 ceph-mon[30820]: pgmap v1432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:51.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:50 smithi116 ceph-mon[35411]: pgmap v1432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:52 smithi114 ceph-mon[30820]: pgmap v1433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:24:53.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:52 smithi116 ceph-mon[35411]: pgmap v1433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:24:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:54 smithi114 ceph-mon[30820]: pgmap v1434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:55.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:54 smithi116 ceph-mon[35411]: pgmap v1434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:24:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:56 smithi114 ceph-mon[30820]: pgmap v1435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:57.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:56 smithi116 ceph-mon[35411]: pgmap v1435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:24:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:58 smithi114 ceph-mon[30820]: pgmap v1436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:24:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:58 smithi116 ceph-mon[35411]: pgmap v1436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:24:59 smithi114 ceph-mon[30820]: pgmap v1437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:24:59 smithi116 ceph-mon[35411]: pgmap v1437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:01 smithi116 ceph-mon[35411]: pgmap v1438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:25:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:01 smithi114 ceph-mon[30820]: pgmap v1438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:25:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:03 smithi114 ceph-mon[30820]: pgmap v1439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:03 smithi116 ceph-mon[35411]: pgmap v1439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:05 smithi114 ceph-mon[30820]: pgmap v1440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:05 smithi116 ceph-mon[35411]: pgmap v1440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:07 smithi114 ceph-mon[30820]: pgmap v1441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:07 smithi116 ceph-mon[35411]: pgmap v1441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:09 smithi114 ceph-mon[30820]: pgmap v1442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:09 smithi116 ceph-mon[35411]: pgmap v1442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:11 smithi114 ceph-mon[30820]: pgmap v1443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:25:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:11 smithi116 ceph-mon[35411]: pgmap v1443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:25:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:13 smithi114 ceph-mon[30820]: pgmap v1444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:14.512 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:13 smithi116 ceph-mon[35411]: pgmap v1444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:15 smithi114 ceph-mon[30820]: pgmap v1445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:15 smithi116 ceph-mon[35411]: pgmap v1445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:17 smithi114 ceph-mon[30820]: pgmap v1446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:17 smithi116 ceph-mon[35411]: pgmap v1446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:19 smithi114 ceph-mon[30820]: pgmap v1447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:19 smithi116 ceph-mon[35411]: pgmap v1447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:21 smithi114 ceph-mon[30820]: pgmap v1448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:25:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:21 smithi116 ceph-mon[35411]: pgmap v1448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:25:24.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:23 smithi114 ceph-mon[30820]: pgmap v1449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:23 smithi116 ceph-mon[35411]: pgmap v1449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:26.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:25 smithi116 ceph-mon[35411]: pgmap v1450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:26.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:25:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:25 smithi114 ceph-mon[30820]: pgmap v1450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:25:28.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:27 smithi114 ceph-mon[30820]: pgmap v1451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:28.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:25:28.213+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:25:28.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:25:28.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:25:28.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:25:28.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:25:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:25:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:27 smithi116 ceph-mon[35411]: pgmap v1451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:25:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:25:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:25:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:25:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:25:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:25:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-mon[30820]: pgmap v1452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-mon[30820]: pgmap v1453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:25:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:28 smithi116 ceph-mon[35411]: pgmap v1452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:25:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:28 smithi116 ceph-mon[35411]: pgmap v1453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:25:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:25:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:31 smithi114 ceph-mon[30820]: pgmap v1454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:25:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:31 smithi116 ceph-mon[35411]: pgmap v1454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:25:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:33 smithi114 ceph-mon[30820]: pgmap v1455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:25:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:33 smithi116 ceph-mon[35411]: pgmap v1455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:25:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:35 smithi114 ceph-mon[30820]: pgmap v1456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:25:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:35 smithi116 ceph-mon[35411]: pgmap v1456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:25:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:37 smithi114 ceph-mon[30820]: pgmap v1457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:25:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:37 smithi116 ceph-mon[35411]: pgmap v1457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:25:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:39 smithi114 ceph-mon[30820]: pgmap v1458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:25:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:39 smithi116 ceph-mon[35411]: pgmap v1458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:25:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:41 smithi114 ceph-mon[30820]: pgmap v1459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:41 smithi116 ceph-mon[35411]: pgmap v1459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:43 smithi114 ceph-mon[30820]: pgmap v1460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:43 smithi116 ceph-mon[35411]: pgmap v1460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:45 smithi114 ceph-mon[30820]: pgmap v1461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:45 smithi116 ceph-mon[35411]: pgmap v1461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:25:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:47 smithi114 ceph-mon[30820]: pgmap v1462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:47 smithi116 ceph-mon[35411]: pgmap v1462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:25:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:49 smithi114 ceph-mon[30820]: pgmap v1463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:49 smithi116 ceph-mon[35411]: pgmap v1463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:25:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:51 smithi114 ceph-mon[30820]: pgmap v1464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:25:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:51 smithi116 ceph-mon[35411]: pgmap v1464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:25:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:53 smithi114 ceph-mon[30820]: pgmap v1465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:25:53.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:53 smithi116 ceph-mon[35411]: pgmap v1465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:25:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:55 smithi114 ceph-mon[30820]: pgmap v1466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:25:55.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:55 smithi116 ceph-mon[35411]: pgmap v1466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:25:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:57 smithi114 ceph-mon[30820]: pgmap v1467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:25:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:57 smithi116 ceph-mon[35411]: pgmap v1467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:25:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:25:59 smithi114 ceph-mon[30820]: pgmap v1468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 85 B/s wr, 4 op/s 2024-09-19T07:25:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:25:59 smithi116 ceph-mon[35411]: pgmap v1468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 85 B/s wr, 4 op/s 2024-09-19T07:26:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:01 smithi114 ceph-mon[30820]: pgmap v1469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:26:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:01 smithi116 ceph-mon[35411]: pgmap v1469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-19T07:26:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:03 smithi114 ceph-mon[30820]: pgmap v1470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:03 smithi116 ceph-mon[35411]: pgmap v1470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:05 smithi114 ceph-mon[30820]: pgmap v1471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:26:05.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:05 smithi116 ceph-mon[35411]: pgmap v1471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:26:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:07 smithi114 ceph-mon[30820]: pgmap v1472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:07 smithi116 ceph-mon[35411]: pgmap v1472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:09 smithi114 ceph-mon[30820]: pgmap v1473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:09.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:09 smithi116 ceph-mon[35411]: pgmap v1473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:11 smithi114 ceph-mon[30820]: pgmap v1474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:11.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:11 smithi116 ceph-mon[35411]: pgmap v1474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:13 smithi114 ceph-mon[30820]: pgmap v1475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:13 smithi116 ceph-mon[35411]: pgmap v1475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:15 smithi114 ceph-mon[30820]: pgmap v1476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:15 smithi116 ceph-mon[35411]: pgmap v1476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:17.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:17 smithi114 ceph-mon[30820]: pgmap v1477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:17 smithi116 ceph-mon[35411]: pgmap v1477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:19 smithi114 ceph-mon[30820]: pgmap v1478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:19 smithi116 ceph-mon[35411]: pgmap v1478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:21 smithi114 ceph-mon[30820]: pgmap v1479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:21 smithi116 ceph-mon[35411]: pgmap v1479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:23 smithi114 ceph-mon[30820]: pgmap v1480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:23 smithi116 ceph-mon[35411]: pgmap v1480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:25 smithi114 ceph-mon[30820]: pgmap v1481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:25 smithi116 ceph-mon[35411]: pgmap v1481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:27 smithi114 ceph-mon[30820]: pgmap v1482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:27.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:27 smithi116 ceph-mon[35411]: pgmap v1482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:26:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:26:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:29 smithi114 ceph-mon[30820]: pgmap v1483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:29 smithi116 ceph-mon[35411]: pgmap v1483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-mon[30820]: pgmap v1484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:31.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:26:31.572+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:26:31.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:26:31.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:26:31.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:26:31.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:26:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:26:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:26:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:26:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:26:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:26:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:31 smithi116 ceph-mon[35411]: pgmap v1484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:32 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:26:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:26:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:26:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:32 smithi114 ceph-mon[30820]: pgmap v1485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:26:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:32 smithi114 ceph-mon[30820]: pgmap v1486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:26:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:26:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:32 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:26:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:26:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:26:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:26:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:26:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:32 smithi116 ceph-mon[35411]: pgmap v1485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:32 smithi116 ceph-mon[35411]: pgmap v1486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:26:32.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:26:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:34 smithi114 ceph-mon[30820]: pgmap v1487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:26:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:34 smithi116 ceph-mon[35411]: pgmap v1487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:26:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:36 smithi114 ceph-mon[30820]: pgmap v1488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:26:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:36 smithi116 ceph-mon[35411]: pgmap v1488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:26:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:38 smithi114 ceph-mon[30820]: pgmap v1489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:26:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:38 smithi116 ceph-mon[35411]: pgmap v1489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:26:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:40 smithi114 ceph-mon[30820]: pgmap v1490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:26:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:40 smithi116 ceph-mon[35411]: pgmap v1490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:26:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:42 smithi114 ceph-mon[30820]: pgmap v1491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:26:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:42 smithi116 ceph-mon[35411]: pgmap v1491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:44 smithi114 ceph-mon[30820]: pgmap v1492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:44 smithi116 ceph-mon[35411]: pgmap v1492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:46 smithi114 ceph-mon[30820]: pgmap v1493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:26:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:46 smithi116 ceph-mon[35411]: pgmap v1493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:26:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:48 smithi114 ceph-mon[30820]: pgmap v1494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:48 smithi116 ceph-mon[35411]: pgmap v1494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:50 smithi116 ceph-mon[35411]: pgmap v1495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:51.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:50 smithi114 ceph-mon[30820]: pgmap v1495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:26:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:52 smithi116 ceph-mon[35411]: pgmap v1496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:52 smithi114 ceph-mon[30820]: pgmap v1496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:26:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:54 smithi116 ceph-mon[35411]: pgmap v1497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:55.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:54 smithi114 ceph-mon[30820]: pgmap v1497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:56.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:56 smithi114 ceph-mon[30820]: pgmap v1498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:26:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:56 smithi116 ceph-mon[35411]: pgmap v1498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:26:58.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:26:58 smithi116 ceph-mon[35411]: pgmap v1499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:26:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:26:58 smithi114 ceph-mon[30820]: pgmap v1499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:00 smithi114 ceph-mon[30820]: pgmap v1500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:00 smithi116 ceph-mon[35411]: pgmap v1500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:03.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:02 smithi114 ceph-mon[30820]: pgmap v1501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:02 smithi116 ceph-mon[35411]: pgmap v1501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:04 smithi114 ceph-mon[30820]: pgmap v1502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:04 smithi116 ceph-mon[35411]: pgmap v1502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:07.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:06 smithi114 ceph-mon[30820]: pgmap v1503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:27:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:06 smithi116 ceph-mon[35411]: pgmap v1503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:27:09.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:08 smithi114 ceph-mon[30820]: pgmap v1504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:08 smithi116 ceph-mon[35411]: pgmap v1504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:10 smithi114 ceph-mon[30820]: pgmap v1505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:10 smithi116 ceph-mon[35411]: pgmap v1505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:12 smithi114 ceph-mon[30820]: pgmap v1506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:12 smithi116 ceph-mon[35411]: pgmap v1506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:14 smithi114 ceph-mon[30820]: pgmap v1507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:15.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:14 smithi116 ceph-mon[35411]: pgmap v1507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:16 smithi114 ceph-mon[30820]: pgmap v1508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:27:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:16 smithi116 ceph-mon[35411]: pgmap v1508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:27:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:18 smithi114 ceph-mon[30820]: pgmap v1509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:19.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:18 smithi116 ceph-mon[35411]: pgmap v1509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:21.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:20 smithi114 ceph-mon[30820]: pgmap v1510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:20 smithi116 ceph-mon[35411]: pgmap v1510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:22 smithi114 ceph-mon[30820]: pgmap v1511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:23.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:22 smithi116 ceph-mon[35411]: pgmap v1511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:24 smithi114 ceph-mon[30820]: pgmap v1512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:24 smithi116 ceph-mon[35411]: pgmap v1512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:26 smithi114 ceph-mon[30820]: pgmap v1513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:27:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:26 smithi116 ceph-mon[35411]: pgmap v1513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:27:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:28 smithi114 ceph-mon[30820]: pgmap v1514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:29.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:28 smithi116 ceph-mon[35411]: pgmap v1514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:31.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:30 smithi114 ceph-mon[30820]: pgmap v1515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:31.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:30 smithi116 ceph-mon[35411]: pgmap v1515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:27:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:27:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:32 smithi114 ceph-mon[30820]: pgmap v1516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:32 smithi116 ceph-mon[35411]: pgmap v1516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-mon[30820]: pgmap v1517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:35.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:27:34.932+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:27:35.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:27:35.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:27:35.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:27:35.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:27:35.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:27:35.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:27:35.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:27:35.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:27:35.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:27:35.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:27:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:27:35.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:34 smithi116 ceph-mon[35411]: pgmap v1517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:27:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:27:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:27:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:27:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:27:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:27:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:27:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:27:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:27:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:27:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:27:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:27:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:27:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:27:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:27:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:27:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:27:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:27:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:36 smithi114 ceph-mon[30820]: pgmap v1518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:27:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:36 smithi114 ceph-mon[30820]: pgmap v1519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:36 smithi116 ceph-mon[35411]: pgmap v1518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:27:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:36 smithi116 ceph-mon[35411]: pgmap v1519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:27:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:38 smithi114 ceph-mon[30820]: pgmap v1520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:27:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:38 smithi116 ceph-mon[35411]: pgmap v1520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:27:41.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:40 smithi114 ceph-mon[30820]: pgmap v1521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:27:41.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:40 smithi116 ceph-mon[35411]: pgmap v1521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:27:43.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:42 smithi114 ceph-mon[30820]: pgmap v1522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:27:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:42 smithi116 ceph-mon[35411]: pgmap v1522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:27:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:44 smithi114 ceph-mon[30820]: pgmap v1523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:27:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:44 smithi116 ceph-mon[35411]: pgmap v1523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:46 smithi114 ceph-mon[30820]: pgmap v1524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:27:47.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:46 smithi116 ceph-mon[35411]: pgmap v1524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:27:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:48 smithi114 ceph-mon[30820]: pgmap v1525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:48 smithi116 ceph-mon[35411]: pgmap v1525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:50 smithi114 ceph-mon[30820]: pgmap v1526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:51.322 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:50 smithi116 ceph-mon[35411]: pgmap v1526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:53.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:52 smithi114 ceph-mon[30820]: pgmap v1527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:27:53.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:52 smithi116 ceph-mon[35411]: pgmap v1527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:27:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:54 smithi114 ceph-mon[30820]: pgmap v1528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:54 smithi116 ceph-mon[35411]: pgmap v1528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:27:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:56 smithi114 ceph-mon[30820]: pgmap v1529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:56 smithi116 ceph-mon[35411]: pgmap v1529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:27:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:27:58 smithi114 ceph-mon[30820]: pgmap v1530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:27:59.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:27:58 smithi116 ceph-mon[35411]: pgmap v1530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:00 smithi114 ceph-mon[30820]: pgmap v1531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:00 smithi116 ceph-mon[35411]: pgmap v1531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:02 smithi114 ceph-mon[30820]: pgmap v1532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:28:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:02 smithi116 ceph-mon[35411]: pgmap v1532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:28:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:04 smithi114 ceph-mon[30820]: pgmap v1533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:04 smithi116 ceph-mon[35411]: pgmap v1533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:06 smithi114 ceph-mon[30820]: pgmap v1534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:06 smithi116 ceph-mon[35411]: pgmap v1534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:08 smithi114 ceph-mon[30820]: pgmap v1535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:08 smithi116 ceph-mon[35411]: pgmap v1535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:10 smithi114 ceph-mon[30820]: pgmap v1536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:11.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:10 smithi116 ceph-mon[35411]: pgmap v1536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:12 smithi114 ceph-mon[30820]: pgmap v1537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:28:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:12 smithi116 ceph-mon[35411]: pgmap v1537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:28:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:13 smithi114 ceph-mon[30820]: pgmap v1538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:13 smithi116 ceph-mon[35411]: pgmap v1538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:16 smithi114 ceph-mon[30820]: pgmap v1539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:16 smithi116 ceph-mon[35411]: pgmap v1539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:18 smithi114 ceph-mon[30820]: pgmap v1540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:18 smithi116 ceph-mon[35411]: pgmap v1540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:20 smithi114 ceph-mon[30820]: pgmap v1541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:20 smithi116 ceph-mon[35411]: pgmap v1541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:22 smithi114 ceph-mon[30820]: pgmap v1542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:28:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:22 smithi116 ceph-mon[35411]: pgmap v1542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:28:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:24 smithi114 ceph-mon[30820]: pgmap v1543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:24 smithi116 ceph-mon[35411]: pgmap v1543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:26 smithi114 ceph-mon[30820]: pgmap v1544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:26 smithi116 ceph-mon[35411]: pgmap v1544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:28 smithi114 ceph-mon[30820]: pgmap v1545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:28 smithi116 ceph-mon[35411]: pgmap v1545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:30 smithi114 ceph-mon[30820]: pgmap v1546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:30 smithi116 ceph-mon[35411]: pgmap v1546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:32 smithi114 ceph-mon[30820]: pgmap v1547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:28:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:32 smithi116 ceph-mon[35411]: pgmap v1547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:28:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:34 smithi114 ceph-mon[30820]: pgmap v1548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:34.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:34 smithi116 ceph-mon[35411]: pgmap v1548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:36.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-mon[30820]: pgmap v1549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:36.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:28:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:36 smithi116 ceph-mon[35411]: pgmap v1549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:28:36.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:28:36.372+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:28:36.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:28:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:28:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:28:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:28:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:37 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:28:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:28:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:28:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:28:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:37 smithi114 ceph-mon[30820]: pgmap v1550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:37 smithi114 ceph-mon[30820]: pgmap v1551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:28:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:37 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:28:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:28:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:28:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:28:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:28:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:37 smithi116 ceph-mon[35411]: pgmap v1550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T07:28:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:37 smithi116 ceph-mon[35411]: pgmap v1551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:28:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:28:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:39 smithi114 ceph-mon[30820]: pgmap v1552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:28:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:39 smithi116 ceph-mon[35411]: pgmap v1552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:28:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:41 smithi114 ceph-mon[30820]: pgmap v1553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:28:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:41 smithi116 ceph-mon[35411]: pgmap v1553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:28:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:43 smithi114 ceph-mon[30820]: pgmap v1554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:28:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:43 smithi116 ceph-mon[35411]: pgmap v1554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:28:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:45 smithi114 ceph-mon[30820]: pgmap v1555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:28:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:45 smithi116 ceph-mon[35411]: pgmap v1555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:28:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:47 smithi114 ceph-mon[30820]: pgmap v1556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:28:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:47 smithi116 ceph-mon[35411]: pgmap v1556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:28:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:49 smithi114 ceph-mon[30820]: pgmap v1557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:49 smithi116 ceph-mon[35411]: pgmap v1557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:51 smithi114 ceph-mon[30820]: pgmap v1558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:28:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:51 smithi116 ceph-mon[35411]: pgmap v1558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:28:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:53 smithi114 ceph-mon[30820]: pgmap v1559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:28:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:53 smithi116 ceph-mon[35411]: pgmap v1559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:28:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:55 smithi114 ceph-mon[30820]: pgmap v1560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:55 smithi116 ceph-mon[35411]: pgmap v1560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:28:57.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:57 smithi114 ceph-mon[30820]: pgmap v1561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:57 smithi116 ceph-mon[35411]: pgmap v1561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:28:59.729 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:28:59 smithi116 ceph-mon[35411]: pgmap v1562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:28:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:28:59 smithi114 ceph-mon[30820]: pgmap v1562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:00 smithi116 ceph-mon[35411]: pgmap v1563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:00 smithi114 ceph-mon[30820]: pgmap v1563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:03 smithi114 ceph-mon[30820]: pgmap v1564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T07:29:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:03 smithi116 ceph-mon[35411]: pgmap v1564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T07:29:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:05 smithi114 ceph-mon[30820]: pgmap v1565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:29:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:05 smithi116 ceph-mon[35411]: pgmap v1565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:29:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:07 smithi114 ceph-mon[30820]: pgmap v1566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:07 smithi116 ceph-mon[35411]: pgmap v1566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:09 smithi114 ceph-mon[30820]: pgmap v1567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:29:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:09 smithi116 ceph-mon[35411]: pgmap v1567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:29:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:11 smithi114 ceph-mon[30820]: pgmap v1568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:11 smithi116 ceph-mon[35411]: pgmap v1568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:13 smithi114 ceph-mon[30820]: pgmap v1569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:29:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:13 smithi116 ceph-mon[35411]: pgmap v1569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:29:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:15 smithi114 ceph-mon[30820]: pgmap v1570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:29:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:15 smithi116 ceph-mon[35411]: pgmap v1570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:29:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:17 smithi114 ceph-mon[30820]: pgmap v1571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:17 smithi116 ceph-mon[35411]: pgmap v1571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:19 smithi114 ceph-mon[30820]: pgmap v1572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:19 smithi116 ceph-mon[35411]: pgmap v1572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:21 smithi114 ceph-mon[30820]: pgmap v1573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:29:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:21 smithi116 ceph-mon[35411]: pgmap v1573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:29:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:23 smithi114 ceph-mon[30820]: pgmap v1574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:29:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:23 smithi116 ceph-mon[35411]: pgmap v1574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:29:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:25 smithi114 ceph-mon[30820]: pgmap v1575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:29:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:25 smithi116 ceph-mon[35411]: pgmap v1575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:29:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:27 smithi114 ceph-mon[30820]: pgmap v1576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:27 smithi116 ceph-mon[35411]: pgmap v1576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:29 smithi114 ceph-mon[30820]: pgmap v1577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:29 smithi116 ceph-mon[35411]: pgmap v1577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:31 smithi114 ceph-mon[30820]: pgmap v1578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:29:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:31 smithi116 ceph-mon[35411]: pgmap v1578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:29:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:33 smithi114 ceph-mon[30820]: pgmap v1579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:29:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:33 smithi116 ceph-mon[35411]: pgmap v1579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:29:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:35 smithi114 ceph-mon[30820]: pgmap v1580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:29:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:35 smithi116 ceph-mon[35411]: pgmap v1580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:29:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:29:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: pgmap v1581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: pgmap v1581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:29:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:38 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:29:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:38 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:29:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:38 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:29:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:38 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:29:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:38 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:29:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:38 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:29:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:38 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:29:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:38 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:29:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-mon[30820]: pgmap v1582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:29:39.668+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:29:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:29:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:29:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:29:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:29:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:29:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:29:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:29:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:29:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:29:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:29:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:29:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:39 smithi116 ceph-mon[35411]: pgmap v1582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:40 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:29:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:29:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:29:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:29:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:40 smithi114 ceph-mon[30820]: pgmap v1583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:40 smithi114 ceph-mon[30820]: pgmap v1584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T07:29:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:40 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:29:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:29:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:29:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:40 smithi116 ceph-mon[35411]: pgmap v1583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T07:29:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:40 smithi116 ceph-mon[35411]: pgmap v1584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T07:29:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:29:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:42 smithi114 ceph-mon[30820]: pgmap v1585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:42 smithi116 ceph-mon[35411]: pgmap v1585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:44 smithi114 ceph-mon[30820]: pgmap v1586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:29:45.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:44 smithi116 ceph-mon[35411]: pgmap v1586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:29:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:46 smithi114 ceph-mon[30820]: pgmap v1587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:47.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:46 smithi116 ceph-mon[35411]: pgmap v1587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:48 smithi114 ceph-mon[30820]: pgmap v1588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:48 smithi116 ceph-mon[35411]: pgmap v1588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:29:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:50 smithi114 ceph-mon[30820]: pgmap v1589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:29:51.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:50 smithi116 ceph-mon[35411]: pgmap v1589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:29:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:52 smithi114 ceph-mon[30820]: pgmap v1590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:52 smithi116 ceph-mon[35411]: pgmap v1590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:29:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:54 smithi114 ceph-mon[30820]: pgmap v1591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:55.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:54 smithi116 ceph-mon[35411]: pgmap v1591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:55 smithi114 ceph-mon[30820]: pgmap v1592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:29:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:55 smithi116 ceph-mon[35411]: pgmap v1592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:29:59.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:29:58 smithi114 ceph-mon[30820]: pgmap v1593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:29:59.131 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:29:58 smithi116 ceph-mon[35411]: pgmap v1593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:00 smithi114 ceph-mon[30820]: pgmap v1594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:30:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:00 smithi116 ceph-mon[35411]: pgmap v1594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:30:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:02 smithi114 ceph-mon[30820]: pgmap v1595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:03.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:02 smithi116 ceph-mon[35411]: pgmap v1595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:04 smithi114 ceph-mon[30820]: pgmap v1596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:05.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:04 smithi116 ceph-mon[35411]: pgmap v1596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:06 smithi114 ceph-mon[30820]: pgmap v1597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:30:07.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:06 smithi116 ceph-mon[35411]: pgmap v1597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:30:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:08 smithi114 ceph-mon[30820]: pgmap v1598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:08 smithi116 ceph-mon[35411]: pgmap v1598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:11.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:10 smithi114 ceph-mon[30820]: pgmap v1599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:10 smithi116 ceph-mon[35411]: pgmap v1599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:13.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:12 smithi114 ceph-mon[30820]: pgmap v1600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:12 smithi116 ceph-mon[35411]: pgmap v1600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:14 smithi114 ceph-mon[30820]: pgmap v1601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:14 smithi116 ceph-mon[35411]: pgmap v1601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:16 smithi114 ceph-mon[30820]: pgmap v1602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:30:17.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:16 smithi116 ceph-mon[35411]: pgmap v1602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:30:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:18 smithi114 ceph-mon[30820]: pgmap v1603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:19.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:18 smithi116 ceph-mon[35411]: pgmap v1603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:20 smithi114 ceph-mon[30820]: pgmap v1604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:20 smithi116 ceph-mon[35411]: pgmap v1604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:22 smithi114 ceph-mon[30820]: pgmap v1605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:22 smithi116 ceph-mon[35411]: pgmap v1605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:24 smithi114 ceph-mon[30820]: pgmap v1606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:24 smithi116 ceph-mon[35411]: pgmap v1606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:27.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:26 smithi114 ceph-mon[30820]: pgmap v1607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:30:27.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:26 smithi116 ceph-mon[35411]: pgmap v1607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:30:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:28 smithi114 ceph-mon[30820]: pgmap v1608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:28 smithi116 ceph-mon[35411]: pgmap v1608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:30 smithi114 ceph-mon[30820]: pgmap v1609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:31.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:30 smithi116 ceph-mon[35411]: pgmap v1609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:32 smithi114 ceph-mon[30820]: pgmap v1610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:32 smithi116 ceph-mon[35411]: pgmap v1610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:34 smithi114 ceph-mon[30820]: pgmap v1611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:35.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:34 smithi116 ceph-mon[35411]: pgmap v1611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:36 smithi114 ceph-mon[30820]: pgmap v1612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:30:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:36 smithi116 ceph-mon[35411]: pgmap v1612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:30:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:38 smithi114 ceph-mon[30820]: pgmap v1613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:39.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:38 smithi116 ceph-mon[35411]: pgmap v1613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:30:40.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:30:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:30:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:40 smithi114 ceph-mon[30820]: pgmap v1614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:40 smithi116 ceph-mon[35411]: pgmap v1614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:42 smithi114 ceph-mon[30820]: pgmap v1615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:43.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:30:43.028+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:30:43.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:30:43.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:30:43.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:30:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:30:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:30:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:30:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:30:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:30:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:30:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:30:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:30:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:42 smithi116 ceph-mon[35411]: pgmap v1615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:30:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:30:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:30:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:30:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:30:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:30:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:44 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:30:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:30:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:30:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:30:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:30:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:30:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:30:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:30:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:30:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:30:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:44 smithi114 ceph-mon[30820]: pgmap v1616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:30:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:44 smithi114 ceph-mon[30820]: pgmap v1617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:44 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:44 smithi116 ceph-mon[35411]: pgmap v1616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:44 smithi116 ceph-mon[35411]: pgmap v1617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:30:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:46 smithi114 ceph-mon[30820]: pgmap v1618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:30:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:46 smithi116 ceph-mon[35411]: pgmap v1618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:30:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:48 smithi114 ceph-mon[30820]: pgmap v1619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:30:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:48 smithi116 ceph-mon[35411]: pgmap v1619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:30:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:50 smithi114 ceph-mon[30820]: pgmap v1620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:30:51.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:50 smithi116 ceph-mon[35411]: pgmap v1620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:30:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:52 smithi114 ceph-mon[30820]: pgmap v1621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:30:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:52 smithi116 ceph-mon[35411]: pgmap v1621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:30:55.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:54 smithi114 ceph-mon[30820]: pgmap v1622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:30:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:54 smithi116 ceph-mon[35411]: pgmap v1622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:30:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:56 smithi114 ceph-mon[30820]: pgmap v1623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:56 smithi116 ceph-mon[35411]: pgmap v1623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:30:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:30:58 smithi114 ceph-mon[30820]: pgmap v1624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:30:59.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:30:58 smithi116 ceph-mon[35411]: pgmap v1624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:00 smithi114 ceph-mon[30820]: pgmap v1625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:00 smithi116 ceph-mon[35411]: pgmap v1625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:01 smithi116 ceph-mon[35411]: pgmap v1626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:01 smithi114 ceph-mon[30820]: pgmap v1626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:04 smithi114 ceph-mon[30820]: pgmap v1627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:04 smithi116 ceph-mon[35411]: pgmap v1627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:06 smithi114 ceph-mon[30820]: pgmap v1628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:06 smithi116 ceph-mon[35411]: pgmap v1628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:08 smithi114 ceph-mon[30820]: pgmap v1629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:08.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:08 smithi116 ceph-mon[35411]: pgmap v1629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:10 smithi114 ceph-mon[30820]: pgmap v1630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:10 smithi116 ceph-mon[35411]: pgmap v1630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:12 smithi114 ceph-mon[30820]: pgmap v1631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:12 smithi116 ceph-mon[35411]: pgmap v1631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:14 smithi114 ceph-mon[30820]: pgmap v1632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:14 smithi116 ceph-mon[35411]: pgmap v1632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:16 smithi114 ceph-mon[30820]: pgmap v1633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:16 smithi116 ceph-mon[35411]: pgmap v1633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:18.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:18 smithi114 ceph-mon[30820]: pgmap v1634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:18.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:18 smithi116 ceph-mon[35411]: pgmap v1634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:20 smithi114 ceph-mon[30820]: pgmap v1635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:20.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:20 smithi116 ceph-mon[35411]: pgmap v1635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:22 smithi114 ceph-mon[30820]: pgmap v1636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:22.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:22 smithi116 ceph-mon[35411]: pgmap v1636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:24 smithi114 ceph-mon[30820]: pgmap v1637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:24.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:24 smithi116 ceph-mon[35411]: pgmap v1637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:26.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:26 smithi114 ceph-mon[30820]: pgmap v1638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:26.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:26 smithi116 ceph-mon[35411]: pgmap v1638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:28 smithi114 ceph-mon[30820]: pgmap v1639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:28 smithi116 ceph-mon[35411]: pgmap v1639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:30 smithi114 ceph-mon[30820]: pgmap v1640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:30 smithi116 ceph-mon[35411]: pgmap v1640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:32 smithi114 ceph-mon[30820]: pgmap v1641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:32.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:32 smithi116 ceph-mon[35411]: pgmap v1641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:34 smithi114 ceph-mon[30820]: pgmap v1642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:34 smithi116 ceph-mon[35411]: pgmap v1642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:36 smithi114 ceph-mon[30820]: pgmap v1643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:36.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:36 smithi116 ceph-mon[35411]: pgmap v1643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:38 smithi114 ceph-mon[30820]: pgmap v1644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:38 smithi116 ceph-mon[35411]: pgmap v1644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:31:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:40 smithi116 ceph-mon[35411]: pgmap v1645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:40 smithi114 ceph-mon[30820]: pgmap v1645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:42 smithi116 ceph-mon[35411]: pgmap v1646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:42 smithi114 ceph-mon[30820]: pgmap v1646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:31:43.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:31:43.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:31:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:44 smithi114 ceph-mon[30820]: pgmap v1647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:44 smithi116 ceph-mon[35411]: pgmap v1647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-mon[30820]: pgmap v1648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:31:46.389+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:31:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:31:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:31:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:31:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:31:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:31:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:31:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:31:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:46 smithi116 ceph-mon[35411]: pgmap v1648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:31:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:47 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:31:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:31:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:31:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:31:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:31:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:47 smithi114 ceph-mon[30820]: pgmap v1649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:47 smithi114 ceph-mon[30820]: pgmap v1650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:31:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:47 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:31:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:31:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:31:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:31:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:31:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:31:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:31:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:31:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:31:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:31:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:31:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:31:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:31:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:31:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:47 smithi116 ceph-mon[35411]: pgmap v1649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:31:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:47 smithi116 ceph-mon[35411]: pgmap v1650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:31:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:31:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:49 smithi114 ceph-mon[30820]: pgmap v1651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:31:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:49 smithi116 ceph-mon[35411]: pgmap v1651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:31:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:51 smithi114 ceph-mon[30820]: pgmap v1652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:31:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:51 smithi116 ceph-mon[35411]: pgmap v1652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:31:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:53 smithi114 ceph-mon[30820]: pgmap v1653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:31:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:53 smithi116 ceph-mon[35411]: pgmap v1653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:31:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:55 smithi114 ceph-mon[30820]: pgmap v1654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:31:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:55 smithi116 ceph-mon[35411]: pgmap v1654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:31:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:57 smithi114 ceph-mon[30820]: pgmap v1655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:31:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:57 smithi116 ceph-mon[35411]: pgmap v1655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:31:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:31:59 smithi114 ceph-mon[30820]: pgmap v1656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:31:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:31:59 smithi116 ceph-mon[35411]: pgmap v1656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:01.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:01 smithi116 ceph-mon[35411]: pgmap v1657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:32:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:01 smithi114 ceph-mon[30820]: pgmap v1657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:32:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:03 smithi114 ceph-mon[30820]: pgmap v1658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:32:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:03 smithi116 ceph-mon[35411]: pgmap v1658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:32:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:05 smithi114 ceph-mon[30820]: pgmap v1659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:05 smithi116 ceph-mon[35411]: pgmap v1659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:07 smithi114 ceph-mon[30820]: pgmap v1660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:07 smithi116 ceph-mon[35411]: pgmap v1660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:09 smithi114 ceph-mon[30820]: pgmap v1661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:09 smithi116 ceph-mon[35411]: pgmap v1661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:11 smithi114 ceph-mon[30820]: pgmap v1662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:32:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:11 smithi116 ceph-mon[35411]: pgmap v1662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:32:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:13 smithi114 ceph-mon[30820]: pgmap v1663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:32:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:13 smithi116 ceph-mon[35411]: pgmap v1663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:32:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:15 smithi114 ceph-mon[30820]: pgmap v1664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:15 smithi116 ceph-mon[35411]: pgmap v1664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:17 smithi114 ceph-mon[30820]: pgmap v1665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:17 smithi116 ceph-mon[35411]: pgmap v1665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:19 smithi114 ceph-mon[30820]: pgmap v1666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:19 smithi116 ceph-mon[35411]: pgmap v1666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:21 smithi114 ceph-mon[30820]: pgmap v1667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:32:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:21 smithi116 ceph-mon[35411]: pgmap v1667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:32:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:23 smithi114 ceph-mon[30820]: pgmap v1668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:32:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:23 smithi116 ceph-mon[35411]: pgmap v1668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:32:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:25 smithi114 ceph-mon[30820]: pgmap v1669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:25 smithi116 ceph-mon[35411]: pgmap v1669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:27 smithi114 ceph-mon[30820]: pgmap v1670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:27 smithi116 ceph-mon[35411]: pgmap v1670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:29 smithi114 ceph-mon[30820]: pgmap v1671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:29 smithi116 ceph-mon[35411]: pgmap v1671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:31 smithi114 ceph-mon[30820]: pgmap v1672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:32:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:31 smithi116 ceph-mon[35411]: pgmap v1672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:32:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:33 smithi114 ceph-mon[30820]: pgmap v1673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:32:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:33 smithi116 ceph-mon[35411]: pgmap v1673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:32:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:35 smithi114 ceph-mon[30820]: pgmap v1674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:35 smithi116 ceph-mon[35411]: pgmap v1674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:37 smithi114 ceph-mon[30820]: pgmap v1675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:37 smithi116 ceph-mon[35411]: pgmap v1675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:39 smithi114 ceph-mon[30820]: pgmap v1676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:39 smithi116 ceph-mon[35411]: pgmap v1676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:41 smithi114 ceph-mon[30820]: pgmap v1677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:32:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:41 smithi116 ceph-mon[35411]: pgmap v1677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:32:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:43 smithi114 ceph-mon[30820]: pgmap v1678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:43 smithi116 ceph-mon[35411]: pgmap v1678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:45 smithi114 ceph-mon[30820]: pgmap v1679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:45 smithi116 ceph-mon[35411]: pgmap v1679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:32:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:32:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:32:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:47 smithi114 ceph-mon[30820]: pgmap v1680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:47 smithi116 ceph-mon[35411]: pgmap v1680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:32:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-mon[30820]: pgmap v1681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:49 smithi116 ceph-mon[35411]: pgmap v1681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:32:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:50.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:32:49.748+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:32:50.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:32:50.156 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:32:50.157 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:32:50.158 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:32:50.158 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:32:50.158 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:32:50.158 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:32:50.158 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:32:50.158 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:32:50.158 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:32:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:32:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:50 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:32:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:32:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:32:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:32:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:32:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:32:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:32:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:32:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:32:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:32:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:32:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:32:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:32:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:50 smithi114 ceph-mon[30820]: pgmap v1682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:32:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:50 smithi114 ceph-mon[30820]: pgmap v1683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:32:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:50 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:32:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:32:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:32:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:50 smithi116 ceph-mon[35411]: pgmap v1682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:32:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:50 smithi116 ceph-mon[35411]: pgmap v1683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:32:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:32:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:52 smithi114 ceph-mon[30820]: pgmap v1684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:32:53.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:52 smithi116 ceph-mon[35411]: pgmap v1684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:32:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:54 smithi114 ceph-mon[30820]: pgmap v1685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T07:32:55.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:54 smithi116 ceph-mon[35411]: pgmap v1685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T07:32:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:56 smithi114 ceph-mon[30820]: pgmap v1686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:32:57.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:56 smithi116 ceph-mon[35411]: pgmap v1686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:32:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:32:58 smithi114 ceph-mon[30820]: pgmap v1687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:32:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:32:58 smithi116 ceph-mon[35411]: pgmap v1687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:33:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:00 smithi114 ceph-mon[30820]: pgmap v1688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:33:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:00 smithi116 ceph-mon[35411]: pgmap v1688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:33:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:02 smithi114 ceph-mon[30820]: pgmap v1689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:03.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:02 smithi116 ceph-mon[35411]: pgmap v1689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:04.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:04 smithi114 ceph-mon[30820]: pgmap v1690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:04 smithi116 ceph-mon[35411]: pgmap v1690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:06 smithi114 ceph-mon[30820]: pgmap v1691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:06 smithi116 ceph-mon[35411]: pgmap v1691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:08 smithi114 ceph-mon[30820]: pgmap v1692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:08 smithi116 ceph-mon[35411]: pgmap v1692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:11.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:10 smithi114 ceph-mon[30820]: pgmap v1693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:11.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:10 smithi116 ceph-mon[35411]: pgmap v1693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:12 smithi114 ceph-mon[30820]: pgmap v1694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:12 smithi116 ceph-mon[35411]: pgmap v1694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:14 smithi114 ceph-mon[30820]: pgmap v1695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:14 smithi116 ceph-mon[35411]: pgmap v1695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:16 smithi114 ceph-mon[30820]: pgmap v1696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:17.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:16 smithi116 ceph-mon[35411]: pgmap v1696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:18 smithi114 ceph-mon[30820]: pgmap v1697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:19.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:18 smithi116 ceph-mon[35411]: pgmap v1697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:20 smithi114 ceph-mon[30820]: pgmap v1698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:21.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:20 smithi116 ceph-mon[35411]: pgmap v1698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:22 smithi114 ceph-mon[30820]: pgmap v1699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:23.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:22 smithi116 ceph-mon[35411]: pgmap v1699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:24 smithi114 ceph-mon[30820]: pgmap v1700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:24 smithi116 ceph-mon[35411]: pgmap v1700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:26 smithi114 ceph-mon[30820]: pgmap v1701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:26 smithi116 ceph-mon[35411]: pgmap v1701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:28 smithi114 ceph-mon[30820]: pgmap v1702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:28 smithi116 ceph-mon[35411]: pgmap v1702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:30 smithi114 ceph-mon[30820]: pgmap v1703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:30 smithi116 ceph-mon[35411]: pgmap v1703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:32 smithi114 ceph-mon[30820]: pgmap v1704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:32 smithi116 ceph-mon[35411]: pgmap v1704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:34 smithi114 ceph-mon[30820]: pgmap v1705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:35.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:34 smithi116 ceph-mon[35411]: pgmap v1705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:36 smithi114 ceph-mon[30820]: pgmap v1706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:37.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:36 smithi116 ceph-mon[35411]: pgmap v1706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:37 smithi114 ceph-mon[30820]: pgmap v1707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:38.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:37 smithi116 ceph-mon[35411]: pgmap v1707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:40 smithi114 ceph-mon[30820]: pgmap v1708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:41.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:40 smithi116 ceph-mon[35411]: pgmap v1708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:42 smithi114 ceph-mon[30820]: pgmap v1709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:42 smithi116 ceph-mon[35411]: pgmap v1709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:44 smithi114 ceph-mon[30820]: pgmap v1710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:45.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:44 smithi116 ceph-mon[35411]: pgmap v1710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:46 smithi114 ceph-mon[30820]: pgmap v1711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:47.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:46 smithi116 ceph-mon[35411]: pgmap v1711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:33:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:48 smithi114 ceph-mon[30820]: pgmap v1712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:48 smithi116 ceph-mon[35411]: pgmap v1712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:33:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:50 smithi114 ceph-mon[30820]: pgmap v1713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:33:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:50 smithi116 ceph-mon[35411]: pgmap v1713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:33:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:33:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:52 smithi114 ceph-mon[30820]: pgmap v1714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:53.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:33:53.045+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:33:53.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:33:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:33:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:33:53.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:52 smithi116 ceph-mon[35411]: pgmap v1714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:33:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:33:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:33:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:33:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:33:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:33:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:33:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:33:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:33:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:33:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:33:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:33:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:33:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:33:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:33:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:33:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:54 smithi114 ceph-mon[30820]: pgmap v1715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:33:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:54 smithi114 ceph-mon[30820]: pgmap v1716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:33:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:54 smithi116 ceph-mon[35411]: pgmap v1715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:33:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:54 smithi116 ceph-mon[35411]: pgmap v1716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:33:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:56 smithi114 ceph-mon[30820]: pgmap v1717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:33:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:56 smithi116 ceph-mon[35411]: pgmap v1717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:33:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:33:58 smithi114 ceph-mon[30820]: pgmap v1718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:33:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:33:58 smithi116 ceph-mon[35411]: pgmap v1718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:34:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:00 smithi114 ceph-mon[30820]: pgmap v1719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:34:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:00 smithi116 ceph-mon[35411]: pgmap v1719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:34:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:02 smithi114 ceph-mon[30820]: pgmap v1720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:34:03.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:02 smithi116 ceph-mon[35411]: pgmap v1720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:34:05.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:04 smithi114 ceph-mon[30820]: pgmap v1721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:34:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:04 smithi116 ceph-mon[35411]: pgmap v1721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:34:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:06 smithi114 ceph-mon[30820]: pgmap v1722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:06 smithi116 ceph-mon[35411]: pgmap v1722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:08 smithi114 ceph-mon[30820]: pgmap v1723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:08 smithi116 ceph-mon[35411]: pgmap v1723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:10 smithi114 ceph-mon[30820]: pgmap v1724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:11.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:10 smithi116 ceph-mon[35411]: pgmap v1724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:12 smithi114 ceph-mon[30820]: pgmap v1725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:12 smithi116 ceph-mon[35411]: pgmap v1725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:14 smithi114 ceph-mon[30820]: pgmap v1726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:14 smithi116 ceph-mon[35411]: pgmap v1726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:16 smithi114 ceph-mon[30820]: pgmap v1727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:16 smithi116 ceph-mon[35411]: pgmap v1727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:17 smithi114 ceph-mon[30820]: pgmap v1728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:18.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:17 smithi116 ceph-mon[35411]: pgmap v1728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:20.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:20 smithi114 ceph-mon[30820]: pgmap v1729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:20.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:20 smithi116 ceph-mon[35411]: pgmap v1729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:22 smithi114 ceph-mon[30820]: pgmap v1730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:22.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:22 smithi116 ceph-mon[35411]: pgmap v1730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:24.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:24 smithi114 ceph-mon[30820]: pgmap v1731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:24.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:24 smithi116 ceph-mon[35411]: pgmap v1731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:26 smithi114 ceph-mon[30820]: pgmap v1732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:26.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:26 smithi116 ceph-mon[35411]: pgmap v1732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:28 smithi114 ceph-mon[30820]: pgmap v1733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:28 smithi116 ceph-mon[35411]: pgmap v1733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:30 smithi114 ceph-mon[30820]: pgmap v1734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:30.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:30 smithi116 ceph-mon[35411]: pgmap v1734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:32 smithi114 ceph-mon[30820]: pgmap v1735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:32 smithi116 ceph-mon[35411]: pgmap v1735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:34 smithi114 ceph-mon[30820]: pgmap v1736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:34 smithi116 ceph-mon[35411]: pgmap v1736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:36 smithi114 ceph-mon[30820]: pgmap v1737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:36.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:36 smithi116 ceph-mon[35411]: pgmap v1737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:38 smithi114 ceph-mon[30820]: pgmap v1738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:38 smithi116 ceph-mon[35411]: pgmap v1738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:40 smithi114 ceph-mon[30820]: pgmap v1739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:40 smithi116 ceph-mon[35411]: pgmap v1739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:42 smithi114 ceph-mon[30820]: pgmap v1740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:42 smithi116 ceph-mon[35411]: pgmap v1740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:44 smithi114 ceph-mon[30820]: pgmap v1741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:44.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:44 smithi116 ceph-mon[35411]: pgmap v1741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:46 smithi114 ceph-mon[30820]: pgmap v1742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:46 smithi116 ceph-mon[35411]: pgmap v1742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:48 smithi116 ceph-mon[35411]: pgmap v1743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:48 smithi114 ceph-mon[30820]: pgmap v1743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:34:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:50 smithi116 ceph-mon[35411]: pgmap v1744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:50 smithi114 ceph-mon[30820]: pgmap v1744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:52 smithi116 ceph-mon[35411]: pgmap v1745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:52 smithi114 ceph-mon[30820]: pgmap v1745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:34:54.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:54 smithi114 ceph-mon[30820]: pgmap v1746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:34:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:54 smithi116 ceph-mon[35411]: pgmap v1746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:34:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:34:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-mon[30820]: pgmap v1747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:56.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:34:56.404+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:34:56.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:34:56.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:34:56.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:34:56.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:34:56.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:34:56.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:34:56.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:34:56.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:34:56.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:34:56.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:34:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:34:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:56 smithi116 ceph-mon[35411]: pgmap v1747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:34:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:57 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:34:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:34:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:34:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:34:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:34:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:34:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:34:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:34:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:34:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:34:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:34:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:34:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:34:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:57 smithi114 ceph-mon[30820]: pgmap v1748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T07:34:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:57 smithi114 ceph-mon[30820]: pgmap v1749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:34:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:57 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:34:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:34:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:34:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:34:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:34:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:34:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:57 smithi116 ceph-mon[35411]: pgmap v1748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T07:34:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:57 smithi116 ceph-mon[35411]: pgmap v1749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:34:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:34:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:34:59 smithi114 ceph-mon[30820]: pgmap v1750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:34:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:34:59 smithi116 ceph-mon[35411]: pgmap v1750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:35:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:01 smithi114 ceph-mon[30820]: pgmap v1751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:35:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:01 smithi116 ceph-mon[35411]: pgmap v1751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:35:03.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:03 smithi116 ceph-mon[35411]: pgmap v1752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:35:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:03 smithi114 ceph-mon[30820]: pgmap v1752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:35:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:05 smithi114 ceph-mon[30820]: pgmap v1753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:35:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:05 smithi116 ceph-mon[35411]: pgmap v1753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:35:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:07 smithi114 ceph-mon[30820]: pgmap v1754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:35:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:07 smithi116 ceph-mon[35411]: pgmap v1754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:35:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:09 smithi114 ceph-mon[30820]: pgmap v1755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:09 smithi116 ceph-mon[35411]: pgmap v1755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:11 smithi114 ceph-mon[30820]: pgmap v1756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:11 smithi116 ceph-mon[35411]: pgmap v1756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:13 smithi114 ceph-mon[30820]: pgmap v1757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:13 smithi116 ceph-mon[35411]: pgmap v1757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:15 smithi114 ceph-mon[30820]: pgmap v1758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:15 smithi116 ceph-mon[35411]: pgmap v1758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:17 smithi114 ceph-mon[30820]: pgmap v1759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:17 smithi116 ceph-mon[35411]: pgmap v1759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:19 smithi114 ceph-mon[30820]: pgmap v1760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:19 smithi116 ceph-mon[35411]: pgmap v1760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:21 smithi114 ceph-mon[30820]: pgmap v1761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:21 smithi116 ceph-mon[35411]: pgmap v1761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:23 smithi114 ceph-mon[30820]: pgmap v1762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:23 smithi116 ceph-mon[35411]: pgmap v1762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:25 smithi114 ceph-mon[30820]: pgmap v1763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:25 smithi116 ceph-mon[35411]: pgmap v1763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:27 smithi114 ceph-mon[30820]: pgmap v1764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:27 smithi116 ceph-mon[35411]: pgmap v1764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:29 smithi114 ceph-mon[30820]: pgmap v1765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:29 smithi116 ceph-mon[35411]: pgmap v1765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:31.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:31 smithi114 ceph-mon[30820]: pgmap v1766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:31.933 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:31 smithi116 ceph-mon[35411]: pgmap v1766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:33 smithi114 ceph-mon[30820]: pgmap v1767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:33 smithi116 ceph-mon[35411]: pgmap v1767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:35 smithi114 ceph-mon[30820]: pgmap v1768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:35 smithi116 ceph-mon[35411]: pgmap v1768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:37 smithi114 ceph-mon[30820]: pgmap v1769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:37 smithi116 ceph-mon[35411]: pgmap v1769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:39 smithi114 ceph-mon[30820]: pgmap v1770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:39 smithi116 ceph-mon[35411]: pgmap v1770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:41 smithi114 ceph-mon[30820]: pgmap v1771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:41 smithi116 ceph-mon[35411]: pgmap v1771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:43.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:43 smithi114 ceph-mon[30820]: pgmap v1772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:43 smithi116 ceph-mon[35411]: pgmap v1772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:45 smithi114 ceph-mon[30820]: pgmap v1773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:45 smithi116 ceph-mon[35411]: pgmap v1773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:47 smithi114 ceph-mon[30820]: pgmap v1774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:47 smithi116 ceph-mon[35411]: pgmap v1774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:49 smithi114 ceph-mon[30820]: pgmap v1775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:49 smithi116 ceph-mon[35411]: pgmap v1775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:51 smithi114 ceph-mon[30820]: pgmap v1776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:51 smithi116 ceph-mon[35411]: pgmap v1776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:35:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:53 smithi114 ceph-mon[30820]: pgmap v1777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:53 smithi116 ceph-mon[35411]: pgmap v1777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:55 smithi114 ceph-mon[30820]: pgmap v1778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:55 smithi116 ceph-mon[35411]: pgmap v1778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:35:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:35:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:35:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:57 smithi114 ceph-mon[30820]: pgmap v1779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:57 smithi116 ceph-mon[35411]: pgmap v1779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:35:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-mon[30820]: pgmap v1780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:35:59.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:35:59.764+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:35:59.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:35:59.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:35:59.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:35:59.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:35:59.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:35:59.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:35:59.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:35:59.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:35:59.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:35:59.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:35:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:35:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:35:59 smithi116 ceph-mon[35411]: pgmap v1780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:00 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:36:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:36:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:36:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:36:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:36:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:36:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:36:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:36:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:36:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:36:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:36:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:36:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:00 smithi114 ceph-mon[30820]: pgmap v1781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:36:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:00 smithi114 ceph-mon[30820]: pgmap v1782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:36:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:00 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:36:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:36:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:36:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:36:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:36:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:00 smithi116 ceph-mon[35411]: pgmap v1781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:36:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:00 smithi116 ceph-mon[35411]: pgmap v1782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:36:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:36:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:02 smithi114 ceph-mon[30820]: pgmap v1783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:36:03.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:02 smithi116 ceph-mon[35411]: pgmap v1783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:36:05.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:04 smithi114 ceph-mon[30820]: pgmap v1784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:36:05.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:04 smithi116 ceph-mon[35411]: pgmap v1784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:36:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:06 smithi114 ceph-mon[30820]: pgmap v1785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:36:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:06 smithi116 ceph-mon[35411]: pgmap v1785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:36:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:08 smithi114 ceph-mon[30820]: pgmap v1786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:36:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:08 smithi116 ceph-mon[35411]: pgmap v1786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:36:11.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:10 smithi114 ceph-mon[30820]: pgmap v1787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:36:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:10 smithi116 ceph-mon[35411]: pgmap v1787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:36:13.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:12 smithi114 ceph-mon[30820]: pgmap v1788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:12 smithi116 ceph-mon[35411]: pgmap v1788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:14 smithi114 ceph-mon[30820]: pgmap v1789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:15.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:14 smithi116 ceph-mon[35411]: pgmap v1789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:17.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:16 smithi114 ceph-mon[30820]: pgmap v1790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:16 smithi116 ceph-mon[35411]: pgmap v1790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:18 smithi114 ceph-mon[30820]: pgmap v1791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:18 smithi116 ceph-mon[35411]: pgmap v1791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:20 smithi114 ceph-mon[30820]: pgmap v1792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:36:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:20 smithi116 ceph-mon[35411]: pgmap v1792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:36:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:22 smithi114 ceph-mon[30820]: pgmap v1793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:22 smithi116 ceph-mon[35411]: pgmap v1793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:25.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:24 smithi114 ceph-mon[30820]: pgmap v1794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:25.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:24 smithi116 ceph-mon[35411]: pgmap v1794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:27.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:26 smithi114 ceph-mon[30820]: pgmap v1795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:26 smithi116 ceph-mon[35411]: pgmap v1795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:28 smithi114 ceph-mon[30820]: pgmap v1796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:36:29.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:28 smithi116 ceph-mon[35411]: pgmap v1796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:36:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:30 smithi114 ceph-mon[30820]: pgmap v1797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:36:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:30 smithi116 ceph-mon[35411]: pgmap v1797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:36:33.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:32 smithi114 ceph-mon[30820]: pgmap v1798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:32 smithi116 ceph-mon[35411]: pgmap v1798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:34 smithi114 ceph-mon[30820]: pgmap v1799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:36:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:34 smithi116 ceph-mon[35411]: pgmap v1799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:36:37.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:36 smithi114 ceph-mon[30820]: pgmap v1800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:36 smithi116 ceph-mon[35411]: pgmap v1800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:38 smithi114 ceph-mon[30820]: pgmap v1801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:38 smithi116 ceph-mon[35411]: pgmap v1801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:40 smithi114 ceph-mon[30820]: pgmap v1802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:36:41.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:40 smithi116 ceph-mon[35411]: pgmap v1802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:36:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:42 smithi114 ceph-mon[30820]: pgmap v1803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:42 smithi116 ceph-mon[35411]: pgmap v1803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:44 smithi114 ceph-mon[30820]: pgmap v1804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:45.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:44 smithi116 ceph-mon[35411]: pgmap v1804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:46 smithi114 ceph-mon[30820]: pgmap v1805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:47.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:46 smithi116 ceph-mon[35411]: pgmap v1805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:48 smithi114 ceph-mon[30820]: pgmap v1806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:48 smithi116 ceph-mon[35411]: pgmap v1806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:50 smithi114 ceph-mon[30820]: pgmap v1807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:36:51.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:50 smithi116 ceph-mon[35411]: pgmap v1807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:36:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:52 smithi116 ceph-mon[35411]: pgmap v1808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:53.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:52 smithi114 ceph-mon[30820]: pgmap v1808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:36:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:54 smithi116 ceph-mon[35411]: pgmap v1809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:54 smithi114 ceph-mon[30820]: pgmap v1809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:56 smithi116 ceph-mon[35411]: pgmap v1810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:56 smithi114 ceph-mon[30820]: pgmap v1810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:36:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:58 smithi114 ceph-mon[30820]: pgmap v1811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:36:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:58 smithi116 ceph-mon[35411]: pgmap v1811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:00.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:59 smithi114 ceph-mon[30820]: pgmap v1812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:36:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:37:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:59 smithi116 ceph-mon[35411]: pgmap v1812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:36:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:37:03.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:02 smithi114 ceph-mon[30820]: pgmap v1813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:02 smithi116 ceph-mon[35411]: pgmap v1813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:03.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:37:03.124+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:37:03.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:37:03.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:37:03.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:37:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:37:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:37:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:37:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:37:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:37:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:37:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:04 smithi114 ceph-mon[30820]: pgmap v1814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:37:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:04 smithi114 ceph-mon[30820]: pgmap v1815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:04 smithi116 ceph-mon[35411]: pgmap v1814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:04 smithi116 ceph-mon[35411]: pgmap v1815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:37:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:06 smithi114 ceph-mon[30820]: pgmap v1816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:37:07.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:06 smithi116 ceph-mon[35411]: pgmap v1816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:37:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:08 smithi114 ceph-mon[30820]: pgmap v1817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:37:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:08 smithi116 ceph-mon[35411]: pgmap v1817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:37:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:10 smithi114 ceph-mon[30820]: pgmap v1818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:37:11.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:10 smithi116 ceph-mon[35411]: pgmap v1818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:37:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:12 smithi114 ceph-mon[30820]: pgmap v1819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:37:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:12 smithi116 ceph-mon[35411]: pgmap v1819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:37:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:14 smithi114 ceph-mon[30820]: pgmap v1820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:37:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:14 smithi116 ceph-mon[35411]: pgmap v1820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:37:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:16 smithi114 ceph-mon[30820]: pgmap v1821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:16 smithi116 ceph-mon[35411]: pgmap v1821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:18 smithi114 ceph-mon[30820]: pgmap v1822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:18 smithi116 ceph-mon[35411]: pgmap v1822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:21.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:20 smithi114 ceph-mon[30820]: pgmap v1823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:20 smithi116 ceph-mon[35411]: pgmap v1823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:22 smithi114 ceph-mon[30820]: pgmap v1824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:37:23.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:22 smithi116 ceph-mon[35411]: pgmap v1824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:37:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:24 smithi114 ceph-mon[30820]: pgmap v1825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:24 smithi116 ceph-mon[35411]: pgmap v1825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:26 smithi114 ceph-mon[30820]: pgmap v1826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:27.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:26 smithi116 ceph-mon[35411]: pgmap v1826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:28 smithi114 ceph-mon[30820]: pgmap v1827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:29.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:28 smithi116 ceph-mon[35411]: pgmap v1827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:30 smithi114 ceph-mon[30820]: pgmap v1828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:31.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:30 smithi116 ceph-mon[35411]: pgmap v1828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:32 smithi114 ceph-mon[30820]: pgmap v1829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:37:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:32 smithi116 ceph-mon[35411]: pgmap v1829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:37:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:34 smithi114 ceph-mon[30820]: pgmap v1830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:35.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:34 smithi116 ceph-mon[35411]: pgmap v1830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:36 smithi116 ceph-mon[35411]: pgmap v1831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:36 smithi114 ceph-mon[30820]: pgmap v1831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:38 smithi116 ceph-mon[35411]: pgmap v1832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:38 smithi114 ceph-mon[30820]: pgmap v1832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:40 smithi116 ceph-mon[35411]: pgmap v1833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:40 smithi114 ceph-mon[30820]: pgmap v1833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:42 smithi114 ceph-mon[30820]: pgmap v1834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:37:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:42 smithi116 ceph-mon[35411]: pgmap v1834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:37:45.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:44 smithi114 ceph-mon[30820]: pgmap v1835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:45.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:44 smithi116 ceph-mon[35411]: pgmap v1835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:46 smithi114 ceph-mon[30820]: pgmap v1836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:47.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:46 smithi116 ceph-mon[35411]: pgmap v1836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:48.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:47 smithi114 ceph-mon[30820]: pgmap v1837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:48.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:47 smithi116 ceph-mon[35411]: pgmap v1837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:50 smithi114 ceph-mon[30820]: pgmap v1838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:50 smithi116 ceph-mon[35411]: pgmap v1838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:52.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:52 smithi114 ceph-mon[30820]: pgmap v1839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:37:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:52 smithi116 ceph-mon[35411]: pgmap v1839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:37:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:54 smithi114 ceph-mon[30820]: pgmap v1840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:54 smithi116 ceph-mon[35411]: pgmap v1840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:37:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:56 smithi114 ceph-mon[30820]: pgmap v1841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:56 smithi116 ceph-mon[35411]: pgmap v1841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:37:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:37:58 smithi114 ceph-mon[30820]: pgmap v1842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:37:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:37:58 smithi116 ceph-mon[35411]: pgmap v1842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:00 smithi114 ceph-mon[30820]: pgmap v1843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:00 smithi116 ceph-mon[35411]: pgmap v1843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:02 smithi114 ceph-mon[30820]: pgmap v1844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:02 smithi116 ceph-mon[35411]: pgmap v1844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:04.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:04 smithi116 ceph-mon[35411]: pgmap v1845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:04.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:38:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:04 smithi114 ceph-mon[30820]: pgmap v1845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:38:06.486 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-mon[30820]: pgmap v1846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:06 smithi116 ceph-mon[35411]: pgmap v1846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:06.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:38:06.484+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:38:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:38:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:38:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:38:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:38:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:38:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:38:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:38:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:38:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:38:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:38:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:38:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:38:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:38:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:38:06.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:38:06.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:38:06.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:38:06.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:38:06.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:38:06.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:38:06.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:38:06.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:38:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:38:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:38:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:38:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:07 smithi114 ceph-mon[30820]: pgmap v1847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:38:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:38:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:38:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:38:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:07 smithi116 ceph-mon[35411]: pgmap v1847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:38:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:09 smithi114 ceph-mon[30820]: pgmap v1848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:38:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:09 smithi116 ceph-mon[35411]: pgmap v1848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:38:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:11 smithi114 ceph-mon[30820]: pgmap v1849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T07:38:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:11 smithi116 ceph-mon[35411]: pgmap v1849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T07:38:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:13 smithi114 ceph-mon[30820]: pgmap v1850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:38:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:13 smithi116 ceph-mon[35411]: pgmap v1850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:38:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:15 smithi114 ceph-mon[30820]: pgmap v1851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:38:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:15 smithi116 ceph-mon[35411]: pgmap v1851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:38:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:17 smithi114 ceph-mon[30820]: pgmap v1852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:38:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:17 smithi116 ceph-mon[35411]: pgmap v1852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T07:38:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:19 smithi114 ceph-mon[30820]: pgmap v1853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:19 smithi116 ceph-mon[35411]: pgmap v1853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:21 smithi114 ceph-mon[30820]: pgmap v1854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:21 smithi116 ceph-mon[35411]: pgmap v1854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:23 smithi114 ceph-mon[30820]: pgmap v1855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:23 smithi116 ceph-mon[35411]: pgmap v1855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:25 smithi114 ceph-mon[30820]: pgmap v1856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:25 smithi116 ceph-mon[35411]: pgmap v1856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:27 smithi114 ceph-mon[30820]: pgmap v1857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:27 smithi116 ceph-mon[35411]: pgmap v1857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:29 smithi114 ceph-mon[30820]: pgmap v1858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:29 smithi116 ceph-mon[35411]: pgmap v1858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:31 smithi114 ceph-mon[30820]: pgmap v1859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:31 smithi116 ceph-mon[35411]: pgmap v1859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:33 smithi114 ceph-mon[30820]: pgmap v1860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:33 smithi116 ceph-mon[35411]: pgmap v1860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:35 smithi114 ceph-mon[30820]: pgmap v1861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:35 smithi116 ceph-mon[35411]: pgmap v1861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:37.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:37 smithi114 ceph-mon[30820]: pgmap v1862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:37 smithi116 ceph-mon[35411]: pgmap v1862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:39 smithi114 ceph-mon[30820]: pgmap v1863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:39 smithi116 ceph-mon[35411]: pgmap v1863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:41 smithi114 ceph-mon[30820]: pgmap v1864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:41 smithi116 ceph-mon[35411]: pgmap v1864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:43 smithi114 ceph-mon[30820]: pgmap v1865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:43 smithi116 ceph-mon[35411]: pgmap v1865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:45 smithi114 ceph-mon[30820]: pgmap v1866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:45 smithi116 ceph-mon[35411]: pgmap v1866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:47 smithi114 ceph-mon[30820]: pgmap v1867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:47 smithi116 ceph-mon[35411]: pgmap v1867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:49 smithi114 ceph-mon[30820]: pgmap v1868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:49 smithi116 ceph-mon[35411]: pgmap v1868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:51.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:51 smithi114 ceph-mon[30820]: pgmap v1869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:51 smithi116 ceph-mon[35411]: pgmap v1869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:38:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:53 smithi114 ceph-mon[30820]: pgmap v1870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:53 smithi116 ceph-mon[35411]: pgmap v1870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:55 smithi114 ceph-mon[30820]: pgmap v1871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:55 smithi116 ceph-mon[35411]: pgmap v1871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:38:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:57 smithi114 ceph-mon[30820]: pgmap v1872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:57 smithi116 ceph-mon[35411]: pgmap v1872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:38:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:38:59 smithi114 ceph-mon[30820]: pgmap v1873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:38:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:38:59 smithi116 ceph-mon[35411]: pgmap v1873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:01 smithi114 ceph-mon[30820]: pgmap v1874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:01 smithi116 ceph-mon[35411]: pgmap v1874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:03.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:03 smithi114 ceph-mon[30820]: pgmap v1875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:03 smithi116 ceph-mon[35411]: pgmap v1875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:05 smithi114 ceph-mon[30820]: pgmap v1876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:39:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:05 smithi116 ceph-mon[35411]: pgmap v1876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:39:06.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:39:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:39:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:07 smithi114 ceph-mon[30820]: pgmap v1877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:07 smithi116 ceph-mon[35411]: pgmap v1877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-mon[30820]: pgmap v1878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:09.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:39:09.844+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:39:09.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:39:09.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:39:09.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:39:09.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:39:09.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:39:09.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:39:09.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:39:09.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:39:09.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:39:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:39:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:09 smithi116 ceph-mon[35411]: pgmap v1878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:10 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:39:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:39:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:39:10.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:39:10.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:10 smithi116 ceph-mon[35411]: pgmap v1879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:39:10.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:10 smithi116 ceph-mon[35411]: pgmap v1880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:39:10.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:39:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:10 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:39:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:39:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:39:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:39:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:10 smithi114 ceph-mon[30820]: pgmap v1879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:10 smithi114 ceph-mon[30820]: pgmap v1880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:39:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:39:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:12 smithi114 ceph-mon[30820]: pgmap v1881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:39:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:12 smithi116 ceph-mon[35411]: pgmap v1881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:39:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:14 smithi114 ceph-mon[30820]: pgmap v1882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:39:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:14 smithi116 ceph-mon[35411]: pgmap v1882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:39:17.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:16 smithi114 ceph-mon[30820]: pgmap v1883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:39:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:16 smithi116 ceph-mon[35411]: pgmap v1883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:39:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:18 smithi116 ceph-mon[35411]: pgmap v1884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:39:19.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:18 smithi114 ceph-mon[30820]: pgmap v1884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:39:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:20 smithi116 ceph-mon[35411]: pgmap v1885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:39:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:20 smithi114 ceph-mon[30820]: pgmap v1885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:39:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:22 smithi116 ceph-mon[35411]: pgmap v1886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:22 smithi114 ceph-mon[30820]: pgmap v1886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:24 smithi114 ceph-mon[30820]: pgmap v1887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:24 smithi116 ceph-mon[35411]: pgmap v1887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:26 smithi114 ceph-mon[30820]: pgmap v1888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:26 smithi116 ceph-mon[35411]: pgmap v1888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:28 smithi114 ceph-mon[30820]: pgmap v1889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:28 smithi116 ceph-mon[35411]: pgmap v1889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:30 smithi114 ceph-mon[30820]: pgmap v1890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:39:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:30 smithi116 ceph-mon[35411]: pgmap v1890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:39:33.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:32 smithi114 ceph-mon[30820]: pgmap v1891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:33.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:32 smithi116 ceph-mon[35411]: pgmap v1891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:35.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:34 smithi114 ceph-mon[30820]: pgmap v1892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:35.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:34 smithi116 ceph-mon[35411]: pgmap v1892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:37.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:36 smithi114 ceph-mon[30820]: pgmap v1893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:36 smithi116 ceph-mon[35411]: pgmap v1893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:38 smithi114 ceph-mon[30820]: pgmap v1894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:39.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:38 smithi116 ceph-mon[35411]: pgmap v1894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:40 smithi114 ceph-mon[30820]: pgmap v1895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:39:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:40 smithi116 ceph-mon[35411]: pgmap v1895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:39:42.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:42 smithi114 ceph-mon[30820]: pgmap v1896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:42.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:42 smithi116 ceph-mon[35411]: pgmap v1896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:44 smithi116 ceph-mon[35411]: pgmap v1897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:44 smithi114 ceph-mon[30820]: pgmap v1897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:46 smithi116 ceph-mon[35411]: pgmap v1898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:47.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:46 smithi114 ceph-mon[30820]: pgmap v1898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:48 smithi116 ceph-mon[35411]: pgmap v1899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:49.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:48 smithi114 ceph-mon[30820]: pgmap v1899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:51.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:50 smithi114 ceph-mon[30820]: pgmap v1900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:39:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:50 smithi116 ceph-mon[35411]: pgmap v1900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:39:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:52 smithi114 ceph-mon[30820]: pgmap v1901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:52 smithi116 ceph-mon[35411]: pgmap v1901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:39:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:54 smithi114 ceph-mon[30820]: pgmap v1902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:55.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:54 smithi116 ceph-mon[35411]: pgmap v1902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:56 smithi114 ceph-mon[30820]: pgmap v1903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:56 smithi116 ceph-mon[35411]: pgmap v1903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:39:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:39:58 smithi114 ceph-mon[30820]: pgmap v1904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:39:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:39:58 smithi116 ceph-mon[35411]: pgmap v1904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:00 smithi114 ceph-mon[30820]: pgmap v1905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:40:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:00 smithi116 ceph-mon[35411]: pgmap v1905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:40:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:02 smithi114 ceph-mon[30820]: pgmap v1906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:40:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:02 smithi116 ceph-mon[35411]: pgmap v1906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:40:05.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:04 smithi114 ceph-mon[30820]: pgmap v1907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:05.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:04 smithi116 ceph-mon[35411]: pgmap v1907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:07.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:06 smithi114 ceph-mon[30820]: pgmap v1908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:40:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:06 smithi116 ceph-mon[35411]: pgmap v1908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:40:09.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:08 smithi114 ceph-mon[30820]: pgmap v1909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:08 smithi116 ceph-mon[35411]: pgmap v1909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:10 smithi114 ceph-mon[30820]: pgmap v1910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:40:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:10 smithi116 ceph-mon[35411]: pgmap v1910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:40:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:11 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:40:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:40:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:11 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:40:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:12 smithi114 ceph-mon[30820]: pgmap v1911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:40:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:12 smithi116 ceph-mon[35411]: pgmap v1911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:40:13.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:40:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:40:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:40:13.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:40:13.204+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:40:13.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:40:13.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:40:13.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:40:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:40:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:40:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:40:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:40:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:40:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:40:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:40:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:40:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:40:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:40:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:40:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:40:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:40:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:40:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:40:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:40:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:40:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:40:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:40:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:40:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:40:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:14 smithi114 ceph-mon[30820]: pgmap v1912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:40:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:14 smithi114 ceph-mon[30820]: pgmap v1913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:40:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:40:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:40:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:40:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:40:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:40:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:40:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:40:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:40:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:40:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:40:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:14 smithi116 ceph-mon[35411]: pgmap v1912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:40:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:14 smithi116 ceph-mon[35411]: pgmap v1913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:40:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:16 smithi114 ceph-mon[30820]: pgmap v1914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:40:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:16 smithi116 ceph-mon[35411]: pgmap v1914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:40:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:18 smithi114 ceph-mon[30820]: pgmap v1915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:40:18.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:18 smithi116 ceph-mon[35411]: pgmap v1915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:40:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:20 smithi114 ceph-mon[30820]: pgmap v1916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:40:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:20 smithi116 ceph-mon[35411]: pgmap v1916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:40:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:22 smithi114 ceph-mon[30820]: pgmap v1917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:40:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:22 smithi116 ceph-mon[35411]: pgmap v1917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:40:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:24 smithi114 ceph-mon[30820]: pgmap v1918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:40:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:24 smithi116 ceph-mon[35411]: pgmap v1918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:40:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:26 smithi114 ceph-mon[30820]: pgmap v1919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:40:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:26 smithi116 ceph-mon[35411]: pgmap v1919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:40:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:28 smithi114 ceph-mon[30820]: pgmap v1920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:40:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:28 smithi116 ceph-mon[35411]: pgmap v1920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:40:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:30 smithi114 ceph-mon[30820]: pgmap v1921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:30 smithi116 ceph-mon[35411]: pgmap v1921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:32 smithi114 ceph-mon[30820]: pgmap v1922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:40:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:32 smithi116 ceph-mon[35411]: pgmap v1922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:40:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:34 smithi114 ceph-mon[30820]: pgmap v1923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:34 smithi116 ceph-mon[35411]: pgmap v1923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:36 smithi114 ceph-mon[30820]: pgmap v1924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:40:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:36 smithi116 ceph-mon[35411]: pgmap v1924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:40:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:38 smithi114 ceph-mon[30820]: pgmap v1925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:40:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:38 smithi116 ceph-mon[35411]: pgmap v1925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:40:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:40 smithi114 ceph-mon[30820]: pgmap v1926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:40 smithi116 ceph-mon[35411]: pgmap v1926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:42 smithi114 ceph-mon[30820]: pgmap v1927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:40:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:42 smithi116 ceph-mon[35411]: pgmap v1927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:40:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:44 smithi114 ceph-mon[30820]: pgmap v1928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:44 smithi116 ceph-mon[35411]: pgmap v1928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:46 smithi114 ceph-mon[30820]: pgmap v1929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:40:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:46 smithi116 ceph-mon[35411]: pgmap v1929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:40:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:48 smithi114 ceph-mon[30820]: pgmap v1930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:40:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:48 smithi116 ceph-mon[35411]: pgmap v1930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:40:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:50 smithi114 ceph-mon[30820]: pgmap v1931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:50 smithi116 ceph-mon[35411]: pgmap v1931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:40:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:52 smithi114 ceph-mon[30820]: pgmap v1932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:40:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:52 smithi116 ceph-mon[35411]: pgmap v1932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:54 smithi114 ceph-mon[30820]: pgmap v1933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:54 smithi116 ceph-mon[35411]: pgmap v1933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:40:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:56 smithi114 ceph-mon[30820]: pgmap v1934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:40:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:56 smithi116 ceph-mon[35411]: pgmap v1934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:40:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:40:58 smithi114 ceph-mon[30820]: pgmap v1935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:40:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:40:58 smithi116 ceph-mon[35411]: pgmap v1935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:41:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:00 smithi114 ceph-mon[30820]: pgmap v1936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:00 smithi116 ceph-mon[35411]: pgmap v1936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:02.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:02 smithi114 ceph-mon[30820]: pgmap v1937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:02 smithi116 ceph-mon[35411]: pgmap v1937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:04.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:04 smithi114 ceph-mon[30820]: pgmap v1938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:04 smithi116 ceph-mon[35411]: pgmap v1938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:06.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:06 smithi114 ceph-mon[30820]: pgmap v1939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:41:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:06 smithi116 ceph-mon[35411]: pgmap v1939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:41:08.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:08 smithi114 ceph-mon[30820]: pgmap v1940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:41:08.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:08 smithi116 ceph-mon[35411]: pgmap v1940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:41:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:10 smithi114 ceph-mon[30820]: pgmap v1941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:10 smithi116 ceph-mon[35411]: pgmap v1941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:12.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:12 smithi114 ceph-mon[30820]: pgmap v1942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:12.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:12 smithi116 ceph-mon[35411]: pgmap v1942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:13.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:41:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:41:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:14 smithi114 ceph-mon[30820]: pgmap v1943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:14 smithi116 ceph-mon[35411]: pgmap v1943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-mon[30820]: pgmap v1944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:41:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:41:16.564+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:41:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:41:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:41:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:41:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:41:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:41:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:41:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:41:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:41:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:41:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:41:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:41:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:16 smithi116 ceph-mon[35411]: pgmap v1944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:41:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:41:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:41:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:41:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:17 smithi114 ceph-mon[30820]: pgmap v1945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T07:41:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:17 smithi114 ceph-mon[30820]: pgmap v1946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:41:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:41:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:41:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:17 smithi116 ceph-mon[35411]: pgmap v1945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T07:41:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:17 smithi116 ceph-mon[35411]: pgmap v1946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:41:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:41:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:19 smithi114 ceph-mon[30820]: pgmap v1947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:41:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:19 smithi116 ceph-mon[35411]: pgmap v1947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:41:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:21 smithi114 ceph-mon[30820]: pgmap v1948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:41:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:21 smithi116 ceph-mon[35411]: pgmap v1948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:41:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:23 smithi114 ceph-mon[30820]: pgmap v1949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:41:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:23 smithi116 ceph-mon[35411]: pgmap v1949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:41:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:25 smithi114 ceph-mon[30820]: pgmap v1950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:41:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:25 smithi116 ceph-mon[35411]: pgmap v1950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:41:27.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:27 smithi114 ceph-mon[30820]: pgmap v1951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:41:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:27 smithi116 ceph-mon[35411]: pgmap v1951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:41:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:29 smithi114 ceph-mon[30820]: pgmap v1952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:29 smithi116 ceph-mon[35411]: pgmap v1952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:31.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:31 smithi114 ceph-mon[30820]: pgmap v1953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:41:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:31 smithi116 ceph-mon[35411]: pgmap v1953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:41:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:33 smithi114 ceph-mon[30820]: pgmap v1954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:33 smithi116 ceph-mon[35411]: pgmap v1954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:35 smithi114 ceph-mon[30820]: pgmap v1955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:35 smithi116 ceph-mon[35411]: pgmap v1955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:37 smithi114 ceph-mon[30820]: pgmap v1956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:37 smithi116 ceph-mon[35411]: pgmap v1956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:39 smithi114 ceph-mon[30820]: pgmap v1957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:39 smithi116 ceph-mon[35411]: pgmap v1957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:41 smithi114 ceph-mon[30820]: pgmap v1958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:41:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:41 smithi116 ceph-mon[35411]: pgmap v1958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:41:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:43 smithi116 ceph-mon[35411]: pgmap v1959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:43 smithi114 ceph-mon[30820]: pgmap v1959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:45 smithi116 ceph-mon[35411]: pgmap v1960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:45 smithi114 ceph-mon[30820]: pgmap v1960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:47 smithi116 ceph-mon[35411]: pgmap v1961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:47 smithi114 ceph-mon[30820]: pgmap v1961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:49 smithi114 ceph-mon[30820]: pgmap v1962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:49 smithi116 ceph-mon[35411]: pgmap v1962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:51 smithi114 ceph-mon[30820]: pgmap v1963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:41:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:51 smithi116 ceph-mon[35411]: pgmap v1963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:41:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:53 smithi114 ceph-mon[30820]: pgmap v1964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:53 smithi116 ceph-mon[35411]: pgmap v1964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:41:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:55 smithi114 ceph-mon[30820]: pgmap v1965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:55 smithi116 ceph-mon[35411]: pgmap v1965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:41:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:57 smithi114 ceph-mon[30820]: pgmap v1966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:41:58.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:57 smithi116 ceph-mon[35411]: pgmap v1966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:41:59 smithi114 ceph-mon[30820]: pgmap v1967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:41:59 smithi116 ceph-mon[35411]: pgmap v1967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:01 smithi114 ceph-mon[30820]: pgmap v1968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:01 smithi116 ceph-mon[35411]: pgmap v1968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:03 smithi114 ceph-mon[30820]: pgmap v1969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:03 smithi116 ceph-mon[35411]: pgmap v1969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:05 smithi114 ceph-mon[30820]: pgmap v1970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:42:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:05 smithi116 ceph-mon[35411]: pgmap v1970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:42:07.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:07 smithi114 ceph-mon[30820]: pgmap v1971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:07.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:07 smithi116 ceph-mon[35411]: pgmap v1971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:10.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:09 smithi114 ceph-mon[30820]: pgmap v1972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:09 smithi116 ceph-mon[35411]: pgmap v1972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:11 smithi114 ceph-mon[30820]: pgmap v1973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:11 smithi116 ceph-mon[35411]: pgmap v1973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:14.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:13 smithi114 ceph-mon[30820]: pgmap v1974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:13 smithi116 ceph-mon[35411]: pgmap v1974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:15 smithi114 ceph-mon[30820]: pgmap v1975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:42:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:15 smithi116 ceph-mon[35411]: pgmap v1975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:42:17.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:42:17.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:42:18.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:17 smithi114 ceph-mon[30820]: pgmap v1976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:17 smithi116 ceph-mon[35411]: pgmap v1976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:42:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:42:19.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:42:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:42:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-mon[30820]: pgmap v1977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:20.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:42:19.924+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:42:20.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:42:20.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:42:20.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:42:20.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:42:20.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:42:20.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:42:20.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:42:20.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:42:20.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:42:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:42:20.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:19 smithi116 ceph-mon[35411]: pgmap v1977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:20 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:42:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:42:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:42:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:42:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:42:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:42:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:42:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:42:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:42:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:42:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:42:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:42:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:20 smithi114 ceph-mon[30820]: pgmap v1978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:42:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:20 smithi114 ceph-mon[30820]: pgmap v1979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:42:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:20 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:42:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:42:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:42:21.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:42:21.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:20 smithi116 ceph-mon[35411]: pgmap v1978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:42:21.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:20 smithi116 ceph-mon[35411]: pgmap v1979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:42:21.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:42:23.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:22 smithi114 ceph-mon[30820]: pgmap v1980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:42:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:22 smithi116 ceph-mon[35411]: pgmap v1980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:42:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:24 smithi114 ceph-mon[30820]: pgmap v1981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:42:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:24 smithi116 ceph-mon[35411]: pgmap v1981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:42:26.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:26 smithi114 ceph-mon[30820]: pgmap v1982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:42:26.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:26 smithi116 ceph-mon[35411]: pgmap v1982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:42:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:28 smithi114 ceph-mon[30820]: pgmap v1983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:42:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:28 smithi116 ceph-mon[35411]: pgmap v1983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:42:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:30 smithi114 ceph-mon[30820]: pgmap v1984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:42:30.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:30 smithi116 ceph-mon[35411]: pgmap v1984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:42:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:32 smithi114 ceph-mon[30820]: pgmap v1985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:32 smithi116 ceph-mon[35411]: pgmap v1985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:34.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:34 smithi114 ceph-mon[30820]: pgmap v1986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:34.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:34 smithi116 ceph-mon[35411]: pgmap v1986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:36 smithi114 ceph-mon[30820]: pgmap v1987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:37.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:36 smithi116 ceph-mon[35411]: pgmap v1987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:38.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:37 smithi114 ceph-mon[30820]: pgmap v1988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:37 smithi116 ceph-mon[35411]: pgmap v1988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:40 smithi114 ceph-mon[30820]: pgmap v1989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:42:41.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:40 smithi116 ceph-mon[35411]: pgmap v1989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:42:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:41 smithi114 ceph-mon[30820]: pgmap v1990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:41 smithi116 ceph-mon[35411]: pgmap v1990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:44 smithi114 ceph-mon[30820]: pgmap v1991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:44 smithi116 ceph-mon[35411]: pgmap v1991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:46 smithi114 ceph-mon[30820]: pgmap v1992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:46.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:46 smithi116 ceph-mon[35411]: pgmap v1992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:48 smithi114 ceph-mon[30820]: pgmap v1993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:48.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:48 smithi116 ceph-mon[35411]: pgmap v1993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:50.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:50 smithi114 ceph-mon[30820]: pgmap v1994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:42:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:50 smithi116 ceph-mon[35411]: pgmap v1994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:42:52.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:52 smithi114 ceph-mon[30820]: pgmap v1995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:52.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:52 smithi116 ceph-mon[35411]: pgmap v1995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:42:54.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:54 smithi114 ceph-mon[30820]: pgmap v1996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:54.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:54 smithi116 ceph-mon[35411]: pgmap v1996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:56 smithi114 ceph-mon[30820]: pgmap v1997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:56 smithi116 ceph-mon[35411]: pgmap v1997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:42:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:42:58 smithi114 ceph-mon[30820]: pgmap v1998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:42:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:42:58 smithi116 ceph-mon[35411]: pgmap v1998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:00 smithi114 ceph-mon[30820]: pgmap v1999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:00 smithi116 ceph-mon[35411]: pgmap v1999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:02.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:02 smithi114 ceph-mon[30820]: pgmap v2000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:02 smithi116 ceph-mon[35411]: pgmap v2000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:04 smithi114 ceph-mon[30820]: pgmap v2001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:04 smithi116 ceph-mon[35411]: pgmap v2001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:06 smithi114 ceph-mon[30820]: pgmap v2002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:43:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:06 smithi116 ceph-mon[35411]: pgmap v2002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:43:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:08 smithi114 ceph-mon[30820]: pgmap v2003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:08.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:08 smithi116 ceph-mon[35411]: pgmap v2003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:10 smithi114 ceph-mon[30820]: pgmap v2004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:10 smithi116 ceph-mon[35411]: pgmap v2004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:12.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:12 smithi114 ceph-mon[30820]: pgmap v2005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:12 smithi116 ceph-mon[35411]: pgmap v2005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:14 smithi114 ceph-mon[30820]: pgmap v2006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:14 smithi116 ceph-mon[35411]: pgmap v2006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:16 smithi114 ceph-mon[30820]: pgmap v2007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:43:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:16 smithi116 ceph-mon[35411]: pgmap v2007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:43:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:18 smithi114 ceph-mon[30820]: pgmap v2008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:18.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:18 smithi116 ceph-mon[35411]: pgmap v2008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:20.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:20 smithi114 ceph-mon[30820]: pgmap v2009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:20 smithi116 ceph-mon[35411]: pgmap v2009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:21.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:43:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:43:22.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:22 smithi114 ceph-mon[30820]: pgmap v2010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:22 smithi116 ceph-mon[35411]: pgmap v2010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:43:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:43:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:23.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:43:23.284+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:43:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:43:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:43:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:43:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:43:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:43:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:43:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:43:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:43:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:24 smithi114 ceph-mon[30820]: pgmap v2011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:43:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:24 smithi114 ceph-mon[30820]: pgmap v2012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:43:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:24 smithi116 ceph-mon[35411]: pgmap v2011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:43:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:24 smithi116 ceph-mon[35411]: pgmap v2012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:43:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:26 smithi114 ceph-mon[30820]: pgmap v2013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:43:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:26 smithi116 ceph-mon[35411]: pgmap v2013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:43:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:28 smithi114 ceph-mon[30820]: pgmap v2014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:43:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:28 smithi116 ceph-mon[35411]: pgmap v2014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:43:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:30 smithi114 ceph-mon[30820]: pgmap v2015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:43:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:30 smithi116 ceph-mon[35411]: pgmap v2015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:43:32.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:32 smithi114 ceph-mon[30820]: pgmap v2016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:43:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:32 smithi116 ceph-mon[35411]: pgmap v2016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:43:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:34 smithi114 ceph-mon[30820]: pgmap v2017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:43:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:34 smithi116 ceph-mon[35411]: pgmap v2017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:43:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:36 smithi114 ceph-mon[30820]: pgmap v2018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:43:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:36 smithi116 ceph-mon[35411]: pgmap v2018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:43:38.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:38 smithi114 ceph-mon[30820]: pgmap v2019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:43:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:38 smithi116 ceph-mon[35411]: pgmap v2019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:43:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:40 smithi114 ceph-mon[30820]: pgmap v2020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:40 smithi116 ceph-mon[35411]: pgmap v2020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:42 smithi114 ceph-mon[30820]: pgmap v2021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:42 smithi116 ceph-mon[35411]: pgmap v2021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:44 smithi114 ceph-mon[30820]: pgmap v2022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:44 smithi116 ceph-mon[35411]: pgmap v2022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:46 smithi114 ceph-mon[30820]: pgmap v2023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:43:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:46 smithi116 ceph-mon[35411]: pgmap v2023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:43:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:48 smithi114 ceph-mon[30820]: pgmap v2024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:43:48.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:48 smithi116 ceph-mon[35411]: pgmap v2024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:43:50.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:50 smithi114 ceph-mon[30820]: pgmap v2025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:50 smithi116 ceph-mon[35411]: pgmap v2025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:43:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:52 smithi114 ceph-mon[30820]: pgmap v2026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:52 smithi116 ceph-mon[35411]: pgmap v2026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:43:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:54 smithi114 ceph-mon[30820]: pgmap v2027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:54 smithi116 ceph-mon[35411]: pgmap v2027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:43:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:56 smithi114 ceph-mon[30820]: pgmap v2028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:43:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:56 smithi116 ceph-mon[35411]: pgmap v2028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:43:58.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:43:58 smithi114 ceph-mon[30820]: pgmap v2029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:43:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:43:58 smithi116 ceph-mon[35411]: pgmap v2029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:44:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:00 smithi114 ceph-mon[30820]: pgmap v2030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:00 smithi116 ceph-mon[35411]: pgmap v2030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:02 smithi114 ceph-mon[30820]: pgmap v2031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:02.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:02 smithi116 ceph-mon[35411]: pgmap v2031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:04 smithi114 ceph-mon[30820]: pgmap v2032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:04 smithi116 ceph-mon[35411]: pgmap v2032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:06 smithi114 ceph-mon[30820]: pgmap v2033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:44:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:06 smithi116 ceph-mon[35411]: pgmap v2033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:44:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:08 smithi114 ceph-mon[30820]: pgmap v2034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:44:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:08 smithi116 ceph-mon[35411]: pgmap v2034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:44:10.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:10 smithi114 ceph-mon[30820]: pgmap v2035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:10.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:10 smithi116 ceph-mon[35411]: pgmap v2035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:12.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:12 smithi114 ceph-mon[30820]: pgmap v2036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:12.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:12 smithi116 ceph-mon[35411]: pgmap v2036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:14.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:14 smithi114 ceph-mon[30820]: pgmap v2037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:14.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:14 smithi116 ceph-mon[35411]: pgmap v2037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:16 smithi114 ceph-mon[30820]: pgmap v2038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:44:16.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:16 smithi116 ceph-mon[35411]: pgmap v2038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:44:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:18 smithi114 ceph-mon[30820]: pgmap v2039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:44:18.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:18 smithi116 ceph-mon[35411]: pgmap v2039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:44:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:20 smithi114 ceph-mon[30820]: pgmap v2040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:20 smithi116 ceph-mon[35411]: pgmap v2040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:22 smithi114 ceph-mon[30820]: pgmap v2041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:22 smithi116 ceph-mon[35411]: pgmap v2041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:44:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:44:25.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:24 smithi114 ceph-mon[30820]: pgmap v2042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:24 smithi116 ceph-mon[35411]: pgmap v2042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-mon[30820]: pgmap v2043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:44:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:44:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:44:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:26.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:44:26.580+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:44:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:44:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:44:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:44:27.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:26 smithi116 ceph-mon[35411]: pgmap v2043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:44:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:44:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:44:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:44:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:44:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:27 smithi114 ceph-mon[30820]: pgmap v2044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T07:44:28.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:27 smithi114 ceph-mon[30820]: pgmap v2045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:44:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:27 smithi116 ceph-mon[35411]: pgmap v2044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T07:44:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:27 smithi116 ceph-mon[35411]: pgmap v2045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:44:30.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:29 smithi114 ceph-mon[30820]: pgmap v2046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:44:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:29 smithi116 ceph-mon[35411]: pgmap v2046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:44:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:31 smithi114 ceph-mon[30820]: pgmap v2047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:44:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:31 smithi116 ceph-mon[35411]: pgmap v2047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:44:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:33 smithi114 ceph-mon[30820]: pgmap v2048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:44:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:33 smithi116 ceph-mon[35411]: pgmap v2048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:44:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:35 smithi114 ceph-mon[30820]: pgmap v2049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T07:44:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:35 smithi116 ceph-mon[35411]: pgmap v2049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T07:44:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:37 smithi114 ceph-mon[30820]: pgmap v2050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:44:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:37 smithi116 ceph-mon[35411]: pgmap v2050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:44:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:39 smithi114 ceph-mon[30820]: pgmap v2051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:39 smithi116 ceph-mon[35411]: pgmap v2051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:41 smithi114 ceph-mon[30820]: pgmap v2052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:44:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:41 smithi116 ceph-mon[35411]: pgmap v2052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:44:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:43 smithi114 ceph-mon[30820]: pgmap v2053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:43 smithi116 ceph-mon[35411]: pgmap v2053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:45 smithi114 ceph-mon[30820]: pgmap v2054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:45 smithi116 ceph-mon[35411]: pgmap v2054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:47 smithi114 ceph-mon[30820]: pgmap v2055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:47 smithi116 ceph-mon[35411]: pgmap v2055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:49 smithi114 ceph-mon[30820]: pgmap v2056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:50.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:49 smithi116 ceph-mon[35411]: pgmap v2056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:52.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:51 smithi114 ceph-mon[30820]: pgmap v2057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:44:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:51 smithi116 ceph-mon[35411]: pgmap v2057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:44:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:53 smithi114 ceph-mon[30820]: pgmap v2058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:53 smithi116 ceph-mon[35411]: pgmap v2058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:44:56.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:55 smithi114 ceph-mon[30820]: pgmap v2059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:55 smithi116 ceph-mon[35411]: pgmap v2059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:44:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:57 smithi114 ceph-mon[30820]: pgmap v2060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:44:58.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:57 smithi116 ceph-mon[35411]: pgmap v2060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:44:59 smithi114 ceph-mon[30820]: pgmap v2061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:44:59 smithi116 ceph-mon[35411]: pgmap v2061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:01 smithi114 ceph-mon[30820]: pgmap v2062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:01 smithi116 ceph-mon[35411]: pgmap v2062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:03.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:03 smithi114 ceph-mon[30820]: pgmap v2063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:03 smithi116 ceph-mon[35411]: pgmap v2063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:05 smithi114 ceph-mon[30820]: pgmap v2064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:45:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:05 smithi116 ceph-mon[35411]: pgmap v2064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:45:07.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:07 smithi114 ceph-mon[30820]: pgmap v2065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:07 smithi116 ceph-mon[35411]: pgmap v2065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:09 smithi114 ceph-mon[30820]: pgmap v2066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:09 smithi116 ceph-mon[35411]: pgmap v2066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:11 smithi116 ceph-mon[35411]: pgmap v2067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:12.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:11 smithi114 ceph-mon[30820]: pgmap v2067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:13 smithi116 ceph-mon[35411]: pgmap v2068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:14.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:13 smithi114 ceph-mon[30820]: pgmap v2068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:15 smithi116 ceph-mon[35411]: pgmap v2069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:45:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:15 smithi114 ceph-mon[30820]: pgmap v2069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:45:18.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:17 smithi114 ceph-mon[30820]: pgmap v2070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:18.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:17 smithi116 ceph-mon[35411]: pgmap v2070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:19 smithi114 ceph-mon[30820]: pgmap v2071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:45:20.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:19 smithi116 ceph-mon[35411]: pgmap v2071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:45:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:21 smithi114 ceph-mon[30820]: pgmap v2072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:21 smithi116 ceph-mon[35411]: pgmap v2072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:23 smithi114 ceph-mon[30820]: pgmap v2073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:45:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:23 smithi116 ceph-mon[35411]: pgmap v2073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:45:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:25 smithi114 ceph-mon[30820]: pgmap v2074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:45:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:25 smithi116 ceph-mon[35411]: pgmap v2074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:45:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:27 smithi114 ceph-mon[30820]: pgmap v2075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:45:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:27 smithi116 ceph-mon[35411]: pgmap v2075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:45:30.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-mon[30820]: pgmap v2076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:30.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:45:29.941+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:45:30.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:45:30.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:45:30.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:45:30.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:45:30.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:45:30.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:45:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:45:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:45:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:45:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:45:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:45:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:45:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:29 smithi116 ceph-mon[35411]: pgmap v2076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:30 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:45:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:45:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:45:31.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:45:31.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:45:31.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:45:31.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:30 smithi114 ceph-mon[30820]: pgmap v2077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:45:31.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:30 smithi114 ceph-mon[30820]: pgmap v2078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:45:31.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:30 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:45:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:30 smithi116 ceph-mon[35411]: pgmap v2077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T07:45:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:30 smithi116 ceph-mon[35411]: pgmap v2078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:45:31.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:45:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:33 smithi114 ceph-mon[30820]: pgmap v2079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:45:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:33 smithi116 ceph-mon[35411]: pgmap v2079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:45:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:35 smithi114 ceph-mon[30820]: pgmap v2080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:45:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:35 smithi116 ceph-mon[35411]: pgmap v2080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:45:37.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:37 smithi114 ceph-mon[30820]: pgmap v2081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:45:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:37 smithi116 ceph-mon[35411]: pgmap v2081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:45:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:39 smithi114 ceph-mon[30820]: pgmap v2082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:45:39.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:39 smithi116 ceph-mon[35411]: pgmap v2082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:45:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:41 smithi114 ceph-mon[30820]: pgmap v2083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:45:41.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:41 smithi116 ceph-mon[35411]: pgmap v2083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:45:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:43 smithi114 ceph-mon[30820]: pgmap v2084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:43.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:43 smithi116 ceph-mon[35411]: pgmap v2084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:45 smithi114 ceph-mon[30820]: pgmap v2085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:45:45.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:45 smithi116 ceph-mon[35411]: pgmap v2085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:45:46.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:46 smithi114 ceph-mon[30820]: pgmap v2086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:46 smithi116 ceph-mon[35411]: pgmap v2086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:49 smithi114 ceph-mon[30820]: pgmap v2087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:45:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:49 smithi116 ceph-mon[35411]: pgmap v2087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:45:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:51 smithi114 ceph-mon[30820]: pgmap v2088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:45:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:51 smithi116 ceph-mon[35411]: pgmap v2088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:45:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:53 smithi114 ceph-mon[30820]: pgmap v2089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:53 smithi116 ceph-mon[35411]: pgmap v2089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:45:55.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:55 smithi114 ceph-mon[30820]: pgmap v2090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:55 smithi116 ceph-mon[35411]: pgmap v2090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:56.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:56 smithi114 ceph-mon[30820]: pgmap v2091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:56 smithi116 ceph-mon[35411]: pgmap v2091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:45:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:45:59 smithi114 ceph-mon[30820]: pgmap v2092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:45:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:45:59 smithi116 ceph-mon[35411]: pgmap v2092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:01 smithi114 ceph-mon[30820]: pgmap v2093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:01 smithi116 ceph-mon[35411]: pgmap v2093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:03.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:03 smithi114 ceph-mon[30820]: pgmap v2094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:03 smithi116 ceph-mon[35411]: pgmap v2094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:05 smithi114 ceph-mon[30820]: pgmap v2095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:05 smithi116 ceph-mon[35411]: pgmap v2095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:07 smithi114 ceph-mon[30820]: pgmap v2096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:46:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:07 smithi116 ceph-mon[35411]: pgmap v2096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:46:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:09 smithi114 ceph-mon[30820]: pgmap v2097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:09 smithi116 ceph-mon[35411]: pgmap v2097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:10 smithi114 ceph-mon[30820]: pgmap v2098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:10 smithi116 ceph-mon[35411]: pgmap v2098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:13 smithi114 ceph-mon[30820]: pgmap v2099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:13 smithi116 ceph-mon[35411]: pgmap v2099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:15.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:15 smithi114 ceph-mon[30820]: pgmap v2100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:15 smithi116 ceph-mon[35411]: pgmap v2100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:17.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:17 smithi114 ceph-mon[30820]: pgmap v2101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:46:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:17 smithi116 ceph-mon[35411]: pgmap v2101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:46:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:19 smithi114 ceph-mon[30820]: pgmap v2102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:19.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:19 smithi116 ceph-mon[35411]: pgmap v2102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:21 smithi114 ceph-mon[30820]: pgmap v2103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:21 smithi116 ceph-mon[35411]: pgmap v2103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:22.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:22 smithi114 ceph-mon[30820]: pgmap v2104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:22.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:22 smithi116 ceph-mon[35411]: pgmap v2104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:25 smithi114 ceph-mon[30820]: pgmap v2105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:25 smithi116 ceph-mon[35411]: pgmap v2105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:27 smithi114 ceph-mon[30820]: pgmap v2106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:46:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:27 smithi116 ceph-mon[35411]: pgmap v2106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:46:29.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:29 smithi114 ceph-mon[30820]: pgmap v2107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:29 smithi116 ceph-mon[35411]: pgmap v2107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:31.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:31 smithi114 ceph-mon[30820]: pgmap v2108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:46:31.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:31 smithi116 ceph-mon[35411]: pgmap v2108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:46:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:32 smithi114 ceph-mon[30820]: pgmap v2109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:32.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:32 smithi116 ceph-mon[35411]: pgmap v2109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:46:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:46:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:46:33.300+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:46:33.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:46:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:46:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:46:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:46:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:46:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:46:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:46:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:46:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:46:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:34 smithi114 ceph-mon[30820]: pgmap v2110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:46:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:34 smithi114 ceph-mon[30820]: pgmap v2111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:46:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:46:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:46:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:46:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:46:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:46:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:46:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:46:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:46:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:34 smithi116 ceph-mon[35411]: pgmap v2110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:46:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:34 smithi116 ceph-mon[35411]: pgmap v2111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:46:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:36 smithi114 ceph-mon[30820]: pgmap v2112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:46:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:36 smithi116 ceph-mon[35411]: pgmap v2112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:46:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:38 smithi114 ceph-mon[30820]: pgmap v2113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:46:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:38 smithi116 ceph-mon[35411]: pgmap v2113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:46:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:40 smithi114 ceph-mon[30820]: pgmap v2114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:46:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:40 smithi116 ceph-mon[35411]: pgmap v2114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:46:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:42 smithi114 ceph-mon[30820]: pgmap v2115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:46:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:42 smithi116 ceph-mon[35411]: pgmap v2115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:46:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:44 smithi114 ceph-mon[30820]: pgmap v2116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:46:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:44 smithi116 ceph-mon[35411]: pgmap v2116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:46:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:46 smithi114 ceph-mon[30820]: pgmap v2117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:46:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:46 smithi116 ceph-mon[35411]: pgmap v2117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:46:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:48 smithi114 ceph-mon[30820]: pgmap v2118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:46:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:48 smithi116 ceph-mon[35411]: pgmap v2118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:46:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:50 smithi114 ceph-mon[30820]: pgmap v2119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:50 smithi116 ceph-mon[35411]: pgmap v2119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:46:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:52 smithi114 ceph-mon[30820]: pgmap v2120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:52 smithi116 ceph-mon[35411]: pgmap v2120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:46:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:54 smithi114 ceph-mon[30820]: pgmap v2121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:54 smithi116 ceph-mon[35411]: pgmap v2121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:46:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:56 smithi114 ceph-mon[30820]: pgmap v2122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:46:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:56 smithi116 ceph-mon[35411]: pgmap v2122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:46:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:46:58 smithi114 ceph-mon[30820]: pgmap v2123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:46:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:46:58 smithi116 ceph-mon[35411]: pgmap v2123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:47:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:00 smithi116 ceph-mon[35411]: pgmap v2124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:47:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:00 smithi114 ceph-mon[30820]: pgmap v2124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:47:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:02 smithi116 ceph-mon[35411]: pgmap v2125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:02 smithi114 ceph-mon[30820]: pgmap v2125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:04 smithi116 ceph-mon[35411]: pgmap v2126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:04 smithi114 ceph-mon[30820]: pgmap v2126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:06 smithi116 ceph-mon[35411]: pgmap v2127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:47:06.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:06 smithi114 ceph-mon[30820]: pgmap v2127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:47:08.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:08 smithi114 ceph-mon[30820]: pgmap v2128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:47:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:08 smithi116 ceph-mon[35411]: pgmap v2128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:47:10.717 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:10 smithi114 ceph-mon[30820]: pgmap v2129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:47:10.729 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:10 smithi116 ceph-mon[35411]: pgmap v2129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:47:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:12 smithi114 ceph-mon[30820]: pgmap v2130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:12 smithi116 ceph-mon[35411]: pgmap v2130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:14 smithi114 ceph-mon[30820]: pgmap v2131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:14.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:14 smithi116 ceph-mon[35411]: pgmap v2131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:16.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:16 smithi114 ceph-mon[30820]: pgmap v2132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:47:16.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:16 smithi116 ceph-mon[35411]: pgmap v2132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:47:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:18 smithi114 ceph-mon[30820]: pgmap v2133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:47:18.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:18 smithi116 ceph-mon[35411]: pgmap v2133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:47:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:20 smithi114 ceph-mon[30820]: pgmap v2134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:47:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:20 smithi116 ceph-mon[35411]: pgmap v2134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:47:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:22 smithi114 ceph-mon[30820]: pgmap v2135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:22 smithi116 ceph-mon[35411]: pgmap v2135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:24 smithi114 ceph-mon[30820]: pgmap v2136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:24 smithi116 ceph-mon[35411]: pgmap v2136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:26.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:26 smithi114 ceph-mon[30820]: pgmap v2137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:26 smithi116 ceph-mon[35411]: pgmap v2137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:28 smithi114 ceph-mon[30820]: pgmap v2138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T07:47:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:28 smithi116 ceph-mon[35411]: pgmap v2138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T07:47:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:30 smithi114 ceph-mon[30820]: pgmap v2139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:47:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:30 smithi116 ceph-mon[35411]: pgmap v2139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:47:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:32 smithi114 ceph-mon[30820]: pgmap v2140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:32 smithi116 ceph-mon[35411]: pgmap v2140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:47:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:47:34.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:34 smithi114 ceph-mon[30820]: pgmap v2141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:47:34.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:34 smithi116 ceph-mon[35411]: pgmap v2141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:47:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-mon[30820]: pgmap v2142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:47:36.660+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:47:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:47:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:47:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:47:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:47:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:47:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:47:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:47:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:47:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:47:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:47:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:47:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:47:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:47:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:47:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:47:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:36 smithi116 ceph-mon[35411]: pgmap v2142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:37 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:47:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:47:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:47:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:47:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:47:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:37 smithi114 ceph-mon[30820]: pgmap v2143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T07:47:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:37 smithi114 ceph-mon[30820]: pgmap v2144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:47:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:37 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:47:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:37 smithi116 ceph-mon[35411]: pgmap v2143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T07:47:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:37 smithi116 ceph-mon[35411]: pgmap v2144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:37.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:47:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:39 smithi114 ceph-mon[30820]: pgmap v2145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:47:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:39 smithi116 ceph-mon[35411]: pgmap v2145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:47:41.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:41 smithi116 ceph-mon[35411]: pgmap v2146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:41 smithi114 ceph-mon[30820]: pgmap v2146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:43 smithi114 ceph-mon[30820]: pgmap v2147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:43 smithi116 ceph-mon[35411]: pgmap v2147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:47:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:45 smithi114 ceph-mon[30820]: pgmap v2148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:47:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:45 smithi116 ceph-mon[35411]: pgmap v2148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:47:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:47 smithi114 ceph-mon[30820]: pgmap v2149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:47:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:47 smithi116 ceph-mon[35411]: pgmap v2149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:47:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:49 smithi114 ceph-mon[30820]: pgmap v2150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:50.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:49 smithi116 ceph-mon[35411]: pgmap v2150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:51 smithi114 ceph-mon[30820]: pgmap v2151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:47:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:51 smithi116 ceph-mon[35411]: pgmap v2151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:47:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:53 smithi114 ceph-mon[30820]: pgmap v2152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:53 smithi116 ceph-mon[35411]: pgmap v2152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:47:56.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:55 smithi114 ceph-mon[30820]: pgmap v2153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:47:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:55 smithi116 ceph-mon[35411]: pgmap v2153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:47:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:57 smithi114 ceph-mon[30820]: pgmap v2154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:47:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:57 smithi116 ceph-mon[35411]: pgmap v2154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:47:59 smithi114 ceph-mon[30820]: pgmap v2155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:00.380 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:47:59 smithi116 ceph-mon[35411]: pgmap v2155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:01 smithi114 ceph-mon[30820]: pgmap v2156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:01 smithi116 ceph-mon[35411]: pgmap v2156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:03 smithi114 ceph-mon[30820]: pgmap v2157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:03 smithi116 ceph-mon[35411]: pgmap v2157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:05 smithi114 ceph-mon[30820]: pgmap v2158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:48:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:05 smithi116 ceph-mon[35411]: pgmap v2158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:48:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:07 smithi114 ceph-mon[30820]: pgmap v2159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:07 smithi116 ceph-mon[35411]: pgmap v2159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:10.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:09 smithi114 ceph-mon[30820]: pgmap v2160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:09 smithi116 ceph-mon[35411]: pgmap v2160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:12.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:11 smithi114 ceph-mon[30820]: pgmap v2161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:11 smithi116 ceph-mon[35411]: pgmap v2161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:13 smithi114 ceph-mon[30820]: pgmap v2162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:13 smithi116 ceph-mon[35411]: pgmap v2162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:15 smithi114 ceph-mon[30820]: pgmap v2163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:48:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:15 smithi116 ceph-mon[35411]: pgmap v2163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:48:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:17 smithi114 ceph-mon[30820]: pgmap v2164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:17 smithi116 ceph-mon[35411]: pgmap v2164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:19 smithi114 ceph-mon[30820]: pgmap v2165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:19 smithi116 ceph-mon[35411]: pgmap v2165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:21 smithi114 ceph-mon[30820]: pgmap v2166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:21 smithi116 ceph-mon[35411]: pgmap v2166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:23 smithi114 ceph-mon[30820]: pgmap v2167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:23 smithi116 ceph-mon[35411]: pgmap v2167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:25 smithi114 ceph-mon[30820]: pgmap v2168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:48:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:25 smithi116 ceph-mon[35411]: pgmap v2168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:48:28.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:27 smithi114 ceph-mon[30820]: pgmap v2169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:27 smithi116 ceph-mon[35411]: pgmap v2169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:30.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:29 smithi114 ceph-mon[30820]: pgmap v2170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:29 smithi116 ceph-mon[35411]: pgmap v2170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:31 smithi114 ceph-mon[30820]: pgmap v2171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:31 smithi116 ceph-mon[35411]: pgmap v2171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:33 smithi114 ceph-mon[30820]: pgmap v2172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:33 smithi116 ceph-mon[35411]: pgmap v2172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:36.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:35 smithi114 ceph-mon[30820]: pgmap v2173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:48:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:35 smithi116 ceph-mon[35411]: pgmap v2173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:48:37.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:48:37.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:48:38.119 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:37 smithi114 ceph-mon[30820]: pgmap v2174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:37 smithi116 ceph-mon[35411]: pgmap v2174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:48:40.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:39 smithi114 ceph-mon[30820]: pgmap v2175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:40.102 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:39 smithi116 ceph-mon[35411]: pgmap v2175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:40.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:48:40.020+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:48:40.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:48:40.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:48:40.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:48:40.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:48:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:48:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:48:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:48:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:48:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:48:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:48:41.085 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:48:41.086 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:41.087 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:40 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:48:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:48:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:48:41.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:41.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:41 smithi114 ceph-mon[30820]: pgmap v2176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:41 smithi116 ceph-mon[35411]: pgmap v2176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:48:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:48:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:42 smithi114 ceph-mon[30820]: pgmap v2177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:48:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:42 smithi114 ceph-mon[30820]: pgmap v2178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:48:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:48:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:48:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:42 smithi116 ceph-mon[35411]: pgmap v2177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:48:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:42 smithi116 ceph-mon[35411]: pgmap v2178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:48:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:48:45.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:44 smithi114 ceph-mon[30820]: pgmap v2179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T07:48:45.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:44 smithi116 ceph-mon[35411]: pgmap v2179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T07:48:47.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:46 smithi114 ceph-mon[30820]: pgmap v2180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:48:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:46 smithi116 ceph-mon[35411]: pgmap v2180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:48:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:48 smithi114 ceph-mon[30820]: pgmap v2181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:48:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:48 smithi116 ceph-mon[35411]: pgmap v2181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T07:48:51.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:50 smithi114 ceph-mon[30820]: pgmap v2182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:48:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:50 smithi116 ceph-mon[35411]: pgmap v2182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:48:53.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:52 smithi114 ceph-mon[30820]: pgmap v2183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:48:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:52 smithi116 ceph-mon[35411]: pgmap v2183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:48:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:54 smithi114 ceph-mon[30820]: pgmap v2184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:55.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:54 smithi116 ceph-mon[35411]: pgmap v2184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:56 smithi114 ceph-mon[30820]: pgmap v2185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:56 smithi116 ceph-mon[35411]: pgmap v2185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:48:59.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:48:58 smithi114 ceph-mon[30820]: pgmap v2186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:48:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:48:58 smithi116 ceph-mon[35411]: pgmap v2186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:00 smithi114 ceph-mon[30820]: pgmap v2187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:00 smithi116 ceph-mon[35411]: pgmap v2187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:02 smithi114 ceph-mon[30820]: pgmap v2188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:02 smithi116 ceph-mon[35411]: pgmap v2188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:05.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:04 smithi114 ceph-mon[30820]: pgmap v2189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:04 smithi116 ceph-mon[35411]: pgmap v2189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:06 smithi114 ceph-mon[30820]: pgmap v2190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:49:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:06 smithi116 ceph-mon[35411]: pgmap v2190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:49:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:08 smithi114 ceph-mon[30820]: pgmap v2191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:08 smithi116 ceph-mon[35411]: pgmap v2191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:11.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:10 smithi114 ceph-mon[30820]: pgmap v2192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:11.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:10 smithi116 ceph-mon[35411]: pgmap v2192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:13.233 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:12 smithi116 ceph-mon[35411]: pgmap v2193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:12 smithi114 ceph-mon[30820]: pgmap v2193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:15.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:14 smithi114 ceph-mon[30820]: pgmap v2194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:14 smithi116 ceph-mon[35411]: pgmap v2194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:17.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:16 smithi114 ceph-mon[30820]: pgmap v2195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:49:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:16 smithi116 ceph-mon[35411]: pgmap v2195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:49:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:18 smithi114 ceph-mon[30820]: pgmap v2196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:19.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:18 smithi116 ceph-mon[35411]: pgmap v2196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:20 smithi114 ceph-mon[30820]: pgmap v2197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:20 smithi116 ceph-mon[35411]: pgmap v2197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:22 smithi114 ceph-mon[30820]: pgmap v2198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:22 smithi116 ceph-mon[35411]: pgmap v2198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:25.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:25 smithi114 ceph-mon[30820]: pgmap v2199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:25 smithi116 ceph-mon[35411]: pgmap v2199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:27 smithi114 ceph-mon[30820]: pgmap v2200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:49:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:27 smithi116 ceph-mon[35411]: pgmap v2200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:49:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:29 smithi114 ceph-mon[30820]: pgmap v2201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:29 smithi116 ceph-mon[35411]: pgmap v2201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:31.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:31 smithi114 ceph-mon[30820]: pgmap v2202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:31.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:31 smithi116 ceph-mon[35411]: pgmap v2202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:33 smithi114 ceph-mon[30820]: pgmap v2203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:33.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:33 smithi116 ceph-mon[35411]: pgmap v2203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:35 smithi114 ceph-mon[30820]: pgmap v2204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:35.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:35 smithi116 ceph-mon[35411]: pgmap v2204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:37.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:37 smithi114 ceph-mon[30820]: pgmap v2205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:49:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:37 smithi116 ceph-mon[35411]: pgmap v2205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:49:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:39 smithi114 ceph-mon[30820]: pgmap v2206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:39.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:39 smithi116 ceph-mon[35411]: pgmap v2206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:41.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:41 smithi114 ceph-mon[30820]: pgmap v2207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:41.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:41 smithi116 ceph-mon[35411]: pgmap v2207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:49:42.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:49:42.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:49:43.328 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:43 smithi116 ceph-mon[35411]: pgmap v2208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:43.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:43 smithi114 ceph-mon[30820]: pgmap v2208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:49:45.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-mon[30820]: pgmap v2209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:45.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:49:45.236+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:49:45.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:49:45.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:49:45.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:49:45.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:49:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:49:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:49:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:49:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:49:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:49:45.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:45 smithi116 ceph-mon[35411]: pgmap v2209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:49:46.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:49:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:49:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:49:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:49:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:49:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:49:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:47 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:49:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:49:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:49:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:49:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:49:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:49:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:49:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:49:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:49:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:49:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:49:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:49:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:49:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:47 smithi114 ceph-mon[30820]: pgmap v2210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:49:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:47 smithi114 ceph-mon[30820]: pgmap v2211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:47 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:49:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:49:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:49:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:47 smithi116 ceph-mon[35411]: pgmap v2210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:49:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:47 smithi116 ceph-mon[35411]: pgmap v2211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:49:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:49 smithi114 ceph-mon[30820]: pgmap v2212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:49:49.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:49 smithi116 ceph-mon[35411]: pgmap v2212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:49:51.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:51 smithi114 ceph-mon[30820]: pgmap v2213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:49:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:51 smithi116 ceph-mon[35411]: pgmap v2213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:49:53.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:53 smithi114 ceph-mon[30820]: pgmap v2214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:49:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:53 smithi116 ceph-mon[35411]: pgmap v2214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:49:55.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:55 smithi114 ceph-mon[30820]: pgmap v2215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:49:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:55 smithi116 ceph-mon[35411]: pgmap v2215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:49:57.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:57 smithi114 ceph-mon[30820]: pgmap v2216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:49:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:57 smithi116 ceph-mon[35411]: pgmap v2216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:49:58.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:49:58 smithi114 ceph-mon[30820]: pgmap v2217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:49:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:49:58 smithi116 ceph-mon[35411]: pgmap v2217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:00 smithi114 ceph-mon[30820]: pgmap v2218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:50:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:00 smithi116 ceph-mon[35411]: pgmap v2218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T07:50:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:02 smithi114 ceph-mon[30820]: pgmap v2219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:02 smithi116 ceph-mon[35411]: pgmap v2219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:04 smithi114 ceph-mon[30820]: pgmap v2220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:04 smithi116 ceph-mon[35411]: pgmap v2220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:06 smithi114 ceph-mon[30820]: pgmap v2221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:06 smithi116 ceph-mon[35411]: pgmap v2221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:08 smithi114 ceph-mon[30820]: pgmap v2222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:08.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:08 smithi116 ceph-mon[35411]: pgmap v2222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:10.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:10 smithi114 ceph-mon[30820]: pgmap v2223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:10 smithi116 ceph-mon[35411]: pgmap v2223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:12 smithi114 ceph-mon[30820]: pgmap v2224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:12 smithi116 ceph-mon[35411]: pgmap v2224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:14 smithi114 ceph-mon[30820]: pgmap v2225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:14.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:14 smithi116 ceph-mon[35411]: pgmap v2225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:16 smithi114 ceph-mon[30820]: pgmap v2226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:16 smithi116 ceph-mon[35411]: pgmap v2226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:18 smithi114 ceph-mon[30820]: pgmap v2227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:18.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:18 smithi116 ceph-mon[35411]: pgmap v2227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:20 smithi114 ceph-mon[30820]: pgmap v2228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:20 smithi116 ceph-mon[35411]: pgmap v2228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:22 smithi114 ceph-mon[30820]: pgmap v2229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:22 smithi116 ceph-mon[35411]: pgmap v2229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:24 smithi114 ceph-mon[30820]: pgmap v2230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:24 smithi116 ceph-mon[35411]: pgmap v2230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:26 smithi114 ceph-mon[30820]: pgmap v2231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:26 smithi116 ceph-mon[35411]: pgmap v2231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:28 smithi114 ceph-mon[30820]: pgmap v2232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:28 smithi116 ceph-mon[35411]: pgmap v2232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:30 smithi114 ceph-mon[30820]: pgmap v2233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:30 smithi116 ceph-mon[35411]: pgmap v2233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:32 smithi114 ceph-mon[30820]: pgmap v2234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:32 smithi116 ceph-mon[35411]: pgmap v2234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:34 smithi114 ceph-mon[30820]: pgmap v2235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:34 smithi116 ceph-mon[35411]: pgmap v2235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:36 smithi114 ceph-mon[30820]: pgmap v2236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:36 smithi116 ceph-mon[35411]: pgmap v2236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:38 smithi114 ceph-mon[30820]: pgmap v2237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:38 smithi116 ceph-mon[35411]: pgmap v2237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:40.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:40 smithi114 ceph-mon[30820]: pgmap v2238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:40 smithi116 ceph-mon[35411]: pgmap v2238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:50:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:42 smithi114 ceph-mon[30820]: pgmap v2239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:42 smithi116 ceph-mon[35411]: pgmap v2239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:50:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:44 smithi114 ceph-mon[30820]: pgmap v2240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:44 smithi116 ceph-mon[35411]: pgmap v2240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:50:45.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:50:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: pgmap v2241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: pgmap v2241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T07:50:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:47 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:50:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:47 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:50:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:47 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T07:50:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:47 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T07:50:48.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-mon[30820]: pgmap v2242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:48.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:50:48.597+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:50:48.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:50:48.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:50:48.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:50:48.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:50:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:50:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:50:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:50:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:50:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:50:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:50:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:50:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:50:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:48 smithi116 ceph-mon[35411]: pgmap v2242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:50:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:50:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:50:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:50:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:50:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:49 smithi114 ceph-mon[30820]: pgmap v2243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:49 smithi114 ceph-mon[30820]: pgmap v2244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:50:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:50:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:50:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:50:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:49 smithi116 ceph-mon[35411]: pgmap v2243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:49 smithi116 ceph-mon[35411]: pgmap v2244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:50:51.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:51 smithi114 ceph-mon[30820]: pgmap v2245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:50:51.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:51 smithi116 ceph-mon[35411]: pgmap v2245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:50:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:53 smithi114 ceph-mon[30820]: pgmap v2246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:50:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:53 smithi116 ceph-mon[35411]: pgmap v2246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:50:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:55 smithi116 ceph-mon[35411]: pgmap v2247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:50:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:55 smithi114 ceph-mon[30820]: pgmap v2247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:50:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:57 smithi116 ceph-mon[35411]: pgmap v2248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:50:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:57 smithi114 ceph-mon[30820]: pgmap v2248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:50:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:50:59 smithi116 ceph-mon[35411]: pgmap v2249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:51:00.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:50:59 smithi114 ceph-mon[30820]: pgmap v2249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:51:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:01 smithi114 ceph-mon[30820]: pgmap v2250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:01 smithi116 ceph-mon[35411]: pgmap v2250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:03 smithi114 ceph-mon[30820]: pgmap v2251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:03 smithi116 ceph-mon[35411]: pgmap v2251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:05 smithi114 ceph-mon[30820]: pgmap v2252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:05 smithi116 ceph-mon[35411]: pgmap v2252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:07 smithi114 ceph-mon[30820]: pgmap v2253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:07 smithi116 ceph-mon[35411]: pgmap v2253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:09 smithi114 ceph-mon[30820]: pgmap v2254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:09 smithi116 ceph-mon[35411]: pgmap v2254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:12.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:11 smithi114 ceph-mon[30820]: pgmap v2255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:11 smithi116 ceph-mon[35411]: pgmap v2255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:13.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:13 smithi114 ceph-mon[30820]: pgmap v2256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:13 smithi116 ceph-mon[35411]: pgmap v2256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:15.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:15 smithi116 ceph-mon[35411]: pgmap v2257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:16.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:15 smithi114 ceph-mon[30820]: pgmap v2257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:17 smithi114 ceph-mon[30820]: pgmap v2258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:18.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:17 smithi116 ceph-mon[35411]: pgmap v2258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:19 smithi114 ceph-mon[30820]: pgmap v2259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:19 smithi116 ceph-mon[35411]: pgmap v2259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:21 smithi114 ceph-mon[30820]: pgmap v2260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:21 smithi116 ceph-mon[35411]: pgmap v2260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:23 smithi114 ceph-mon[30820]: pgmap v2261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:23 smithi116 ceph-mon[35411]: pgmap v2261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:25 smithi114 ceph-mon[30820]: pgmap v2262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:25 smithi116 ceph-mon[35411]: pgmap v2262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:27 smithi114 ceph-mon[30820]: pgmap v2263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:27 smithi116 ceph-mon[35411]: pgmap v2263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:29 smithi114 ceph-mon[30820]: pgmap v2264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:29 smithi116 ceph-mon[35411]: pgmap v2264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:31 smithi114 ceph-mon[30820]: pgmap v2265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:31 smithi116 ceph-mon[35411]: pgmap v2265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:33 smithi114 ceph-mon[30820]: pgmap v2266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:33 smithi116 ceph-mon[35411]: pgmap v2266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:35 smithi114 ceph-mon[30820]: pgmap v2267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:35 smithi116 ceph-mon[35411]: pgmap v2267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:38.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:37 smithi114 ceph-mon[30820]: pgmap v2268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:37 smithi116 ceph-mon[35411]: pgmap v2268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:39 smithi114 ceph-mon[30820]: pgmap v2269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:39 smithi116 ceph-mon[35411]: pgmap v2269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:41 smithi114 ceph-mon[30820]: pgmap v2270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:41 smithi116 ceph-mon[35411]: pgmap v2270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:43 smithi114 ceph-mon[30820]: pgmap v2271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:43 smithi116 ceph-mon[35411]: pgmap v2271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:45 smithi114 ceph-mon[30820]: pgmap v2272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:45 smithi116 ceph-mon[35411]: pgmap v2272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:51:48.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:47 smithi114 ceph-mon[30820]: pgmap v2273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:47 smithi116 ceph-mon[35411]: pgmap v2273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:51:49.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:51:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:51:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:49 smithi114 ceph-mon[30820]: pgmap v2274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:49 smithi116 ceph-mon[35411]: pgmap v2274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:51:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-mon[30820]: pgmap v2275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:52.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:51:51.957+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:51:52.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:51:52.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:51:52.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:51:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:51:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:51:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:51:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:51:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:51:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:51:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:51:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:51 smithi116 ceph-mon[35411]: pgmap v2275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:51:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:52 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:51:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:51:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:51:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:51:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:51:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:51:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:51:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:51:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:51:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:51:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:51:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:51:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:52 smithi114 ceph-mon[30820]: pgmap v2276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:51:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:52 smithi114 ceph-mon[30820]: pgmap v2277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:51:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:51:53.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:52 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:51:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:52 smithi116 ceph-mon[35411]: pgmap v2276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:52 smithi116 ceph-mon[35411]: pgmap v2277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:51:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:51:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:55 smithi114 ceph-mon[30820]: pgmap v2278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:51:55.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:55 smithi116 ceph-mon[35411]: pgmap v2278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:51:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:57 smithi114 ceph-mon[30820]: pgmap v2279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:51:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:57 smithi116 ceph-mon[35411]: pgmap v2279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:51:59.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:51:59 smithi114 ceph-mon[30820]: pgmap v2280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:51:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:51:59 smithi116 ceph-mon[35411]: pgmap v2280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:52:01.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:01 smithi114 ceph-mon[30820]: pgmap v2281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:52:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:01 smithi116 ceph-mon[35411]: pgmap v2281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:52:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:03 smithi114 ceph-mon[30820]: pgmap v2282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:52:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:03 smithi116 ceph-mon[35411]: pgmap v2282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:52:05.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:05 smithi114 ceph-mon[30820]: pgmap v2283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:05 smithi116 ceph-mon[35411]: pgmap v2283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:07 smithi114 ceph-mon[30820]: pgmap v2284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:07 smithi116 ceph-mon[35411]: pgmap v2284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:09 smithi114 ceph-mon[30820]: pgmap v2285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:09 smithi116 ceph-mon[35411]: pgmap v2285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:11 smithi114 ceph-mon[30820]: pgmap v2286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:11.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:11 smithi116 ceph-mon[35411]: pgmap v2286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:13 smithi114 ceph-mon[30820]: pgmap v2287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:13 smithi116 ceph-mon[35411]: pgmap v2287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:15 smithi114 ceph-mon[30820]: pgmap v2288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:15 smithi116 ceph-mon[35411]: pgmap v2288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:17 smithi114 ceph-mon[30820]: pgmap v2289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:17 smithi116 ceph-mon[35411]: pgmap v2289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:19.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:19 smithi114 ceph-mon[30820]: pgmap v2290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:19.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:19 smithi116 ceph-mon[35411]: pgmap v2290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:21 smithi114 ceph-mon[30820]: pgmap v2291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:21 smithi116 ceph-mon[35411]: pgmap v2291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:23 smithi114 ceph-mon[30820]: pgmap v2292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:23.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:23 smithi116 ceph-mon[35411]: pgmap v2292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:25.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:25 smithi114 ceph-mon[30820]: pgmap v2293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:25 smithi116 ceph-mon[35411]: pgmap v2293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:27 smithi114 ceph-mon[30820]: pgmap v2294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:27 smithi116 ceph-mon[35411]: pgmap v2294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:29 smithi114 ceph-mon[30820]: pgmap v2295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:29 smithi116 ceph-mon[35411]: pgmap v2295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:31 smithi114 ceph-mon[30820]: pgmap v2296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:31.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:31 smithi116 ceph-mon[35411]: pgmap v2296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:33 smithi114 ceph-mon[30820]: pgmap v2297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:33.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:33 smithi116 ceph-mon[35411]: pgmap v2297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:35.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:35 smithi114 ceph-mon[30820]: pgmap v2298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:35.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:35 smithi116 ceph-mon[35411]: pgmap v2298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:37 smithi114 ceph-mon[30820]: pgmap v2299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:37.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:37 smithi116 ceph-mon[35411]: pgmap v2299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:39 smithi114 ceph-mon[30820]: pgmap v2300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:39.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:39 smithi116 ceph-mon[35411]: pgmap v2300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:41.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:41 smithi114 ceph-mon[30820]: pgmap v2301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:41.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:41 smithi116 ceph-mon[35411]: pgmap v2301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:42.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:42 smithi114 ceph-mon[30820]: pgmap v2302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:42.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:42 smithi116 ceph-mon[35411]: pgmap v2302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:45 smithi114 ceph-mon[30820]: pgmap v2303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:45 smithi116 ceph-mon[35411]: pgmap v2303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:47.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:47 smithi114 ceph-mon[30820]: pgmap v2304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:47.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:47 smithi116 ceph-mon[35411]: pgmap v2304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:52:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:49 smithi114 ceph-mon[30820]: pgmap v2305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:49 smithi116 ceph-mon[35411]: pgmap v2305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:51 smithi114 ceph-mon[30820]: pgmap v2306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:51 smithi116 ceph-mon[35411]: pgmap v2306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:52:53.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:53 smithi114 ceph-mon[30820]: pgmap v2307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:52:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:53 smithi116 ceph-mon[35411]: pgmap v2307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:52:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:52:55.317 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-mon[30820]: pgmap v2308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:55.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:55 smithi116 ceph-mon[35411]: pgmap v2308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:52:55.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:52:55.317+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:52:55.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:52:55.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:52:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:52:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:52:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:52:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:52:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:52:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:52:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:52:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:52:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:52:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:52:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:52:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:52:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:52:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:52:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:57 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:52:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:52:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:52:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:52:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:52:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:52:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:52:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:52:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:52:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:52:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:52:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:57 smithi114 ceph-mon[30820]: pgmap v2309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:52:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:57 smithi114 ceph-mon[30820]: pgmap v2310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:52:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:57 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:52:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:52:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:57 smithi116 ceph-mon[35411]: pgmap v2309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:57 smithi116 ceph-mon[35411]: pgmap v2310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:52:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:52:59 smithi114 ceph-mon[30820]: pgmap v2311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:52:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:52:59 smithi116 ceph-mon[35411]: pgmap v2311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:53:01.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:01 smithi114 ceph-mon[30820]: pgmap v2312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:53:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:01 smithi116 ceph-mon[35411]: pgmap v2312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:53:03.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:03 smithi114 ceph-mon[30820]: pgmap v2313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:53:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:03 smithi116 ceph-mon[35411]: pgmap v2313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:53:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:05 smithi114 ceph-mon[30820]: pgmap v2314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:53:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:05 smithi116 ceph-mon[35411]: pgmap v2314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:53:07.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:07 smithi114 ceph-mon[30820]: pgmap v2315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:53:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:07 smithi116 ceph-mon[35411]: pgmap v2315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:53:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:09 smithi114 ceph-mon[30820]: pgmap v2316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:09 smithi116 ceph-mon[35411]: pgmap v2316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:11 smithi114 ceph-mon[30820]: pgmap v2317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:11.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:11 smithi116 ceph-mon[35411]: pgmap v2317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:13 smithi114 ceph-mon[30820]: pgmap v2318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:13 smithi116 ceph-mon[35411]: pgmap v2318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:15 smithi114 ceph-mon[30820]: pgmap v2319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:15 smithi116 ceph-mon[35411]: pgmap v2319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:17 smithi114 ceph-mon[30820]: pgmap v2320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:17 smithi116 ceph-mon[35411]: pgmap v2320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:19 smithi114 ceph-mon[30820]: pgmap v2321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:19.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:19 smithi116 ceph-mon[35411]: pgmap v2321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:21 smithi114 ceph-mon[30820]: pgmap v2322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:21 smithi116 ceph-mon[35411]: pgmap v2322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:23 smithi114 ceph-mon[30820]: pgmap v2323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:23.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:23 smithi116 ceph-mon[35411]: pgmap v2323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:25 smithi114 ceph-mon[30820]: pgmap v2324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:25 smithi116 ceph-mon[35411]: pgmap v2324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:27 smithi114 ceph-mon[30820]: pgmap v2325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:27 smithi116 ceph-mon[35411]: pgmap v2325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:28 smithi114 ceph-mon[30820]: pgmap v2326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:28 smithi116 ceph-mon[35411]: pgmap v2326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:30 smithi114 ceph-mon[30820]: pgmap v2327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:30 smithi116 ceph-mon[35411]: pgmap v2327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:32 smithi114 ceph-mon[30820]: pgmap v2328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:32 smithi116 ceph-mon[35411]: pgmap v2328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:34 smithi114 ceph-mon[30820]: pgmap v2329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:34 smithi116 ceph-mon[35411]: pgmap v2329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:36 smithi114 ceph-mon[30820]: pgmap v2330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:36 smithi116 ceph-mon[35411]: pgmap v2330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:38 smithi114 ceph-mon[30820]: pgmap v2331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:38 smithi116 ceph-mon[35411]: pgmap v2331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:40 smithi114 ceph-mon[30820]: pgmap v2332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:40 smithi116 ceph-mon[35411]: pgmap v2332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:42 smithi114 ceph-mon[30820]: pgmap v2333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:42 smithi116 ceph-mon[35411]: pgmap v2333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:44 smithi116 ceph-mon[35411]: pgmap v2334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:44.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:44 smithi114 ceph-mon[30820]: pgmap v2334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:46 smithi116 ceph-mon[35411]: pgmap v2335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:46.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:46 smithi114 ceph-mon[30820]: pgmap v2335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:48 smithi116 ceph-mon[35411]: pgmap v2336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:48 smithi114 ceph-mon[30820]: pgmap v2336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:50 smithi114 ceph-mon[30820]: pgmap v2337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:50 smithi116 ceph-mon[35411]: pgmap v2337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:53:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:52 smithi114 ceph-mon[30820]: pgmap v2338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:52 smithi116 ceph-mon[35411]: pgmap v2338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:53:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:54 smithi114 ceph-mon[30820]: pgmap v2339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:54 smithi116 ceph-mon[35411]: pgmap v2339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:53:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:56 smithi116 ceph-mon[35411]: pgmap v2340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:56.523 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:53:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:56 smithi114 ceph-mon[30820]: pgmap v2340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:53:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:53:58.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:53:58.677+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:53:59.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:53:59.121 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:53:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:53:59.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:59 smithi114 ceph-mon[30820]: pgmap v2341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:53:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:53:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:53:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:53:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:59 smithi116 ceph-mon[35411]: pgmap v2341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:53:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:53:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:53:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:53:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:54:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:00 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:54:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:54:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:54:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:54:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:54:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:54:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:54:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:54:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:54:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:54:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:54:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:00 smithi114 ceph-mon[30820]: pgmap v2342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:54:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:00 smithi114 ceph-mon[30820]: pgmap v2343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:54:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:00 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:54:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:54:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:54:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:54:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:54:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:54:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:54:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:54:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:54:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:54:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:54:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:00 smithi116 ceph-mon[35411]: pgmap v2342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:54:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:00 smithi116 ceph-mon[35411]: pgmap v2343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:54:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:01 smithi114 ceph-mon[30820]: pgmap v2344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:54:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:01 smithi116 ceph-mon[35411]: pgmap v2344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:54:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:03 smithi114 ceph-mon[30820]: pgmap v2345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:54:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:03 smithi116 ceph-mon[35411]: pgmap v2345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:54:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:05 smithi114 ceph-mon[30820]: pgmap v2346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:54:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:05 smithi116 ceph-mon[35411]: pgmap v2346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:54:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:07 smithi114 ceph-mon[30820]: pgmap v2347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:54:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:07 smithi116 ceph-mon[35411]: pgmap v2347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T07:54:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:09 smithi114 ceph-mon[30820]: pgmap v2348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:54:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:09 smithi116 ceph-mon[35411]: pgmap v2348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:54:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:11 smithi114 ceph-mon[30820]: pgmap v2349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:11 smithi116 ceph-mon[35411]: pgmap v2349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:13 smithi114 ceph-mon[30820]: pgmap v2350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:54:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:13 smithi116 ceph-mon[35411]: pgmap v2350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:54:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:15 smithi114 ceph-mon[30820]: pgmap v2351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:54:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:15 smithi116 ceph-mon[35411]: pgmap v2351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:54:18.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:17 smithi114 ceph-mon[30820]: pgmap v2352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:17 smithi116 ceph-mon[35411]: pgmap v2352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:19 smithi114 ceph-mon[30820]: pgmap v2353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:20.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:19 smithi116 ceph-mon[35411]: pgmap v2353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:21 smithi114 ceph-mon[30820]: pgmap v2354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:21 smithi116 ceph-mon[35411]: pgmap v2354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:23 smithi114 ceph-mon[30820]: pgmap v2355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:23 smithi116 ceph-mon[35411]: pgmap v2355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:25 smithi114 ceph-mon[30820]: pgmap v2356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:54:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:25 smithi116 ceph-mon[35411]: pgmap v2356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:54:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:27 smithi114 ceph-mon[30820]: pgmap v2357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:27 smithi116 ceph-mon[35411]: pgmap v2357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:29 smithi114 ceph-mon[30820]: pgmap v2358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:29 smithi116 ceph-mon[35411]: pgmap v2358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:31 smithi114 ceph-mon[30820]: pgmap v2359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:31 smithi116 ceph-mon[35411]: pgmap v2359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:33 smithi114 ceph-mon[30820]: pgmap v2360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:33 smithi116 ceph-mon[35411]: pgmap v2360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:35 smithi114 ceph-mon[30820]: pgmap v2361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:54:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:35 smithi116 ceph-mon[35411]: pgmap v2361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:54:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:37 smithi114 ceph-mon[30820]: pgmap v2362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:37 smithi116 ceph-mon[35411]: pgmap v2362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:39 smithi114 ceph-mon[30820]: pgmap v2363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:39 smithi116 ceph-mon[35411]: pgmap v2363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:41 smithi114 ceph-mon[30820]: pgmap v2364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:41 smithi116 ceph-mon[35411]: pgmap v2364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:43 smithi114 ceph-mon[30820]: pgmap v2365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:43 smithi116 ceph-mon[35411]: pgmap v2365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:45 smithi114 ceph-mon[30820]: pgmap v2366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:54:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:45 smithi116 ceph-mon[35411]: pgmap v2366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:54:48.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:47 smithi114 ceph-mon[30820]: pgmap v2367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:47 smithi116 ceph-mon[35411]: pgmap v2367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:49 smithi114 ceph-mon[30820]: pgmap v2368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:50.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:49 smithi116 ceph-mon[35411]: pgmap v2368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:51 smithi114 ceph-mon[30820]: pgmap v2369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:51 smithi116 ceph-mon[35411]: pgmap v2369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:54:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:53 smithi114 ceph-mon[30820]: pgmap v2370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:53 smithi116 ceph-mon[35411]: pgmap v2370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:54:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:55 smithi114 ceph-mon[30820]: pgmap v2371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:54:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:55 smithi116 ceph-mon[35411]: pgmap v2371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:54:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:57 smithi114 ceph-mon[30820]: pgmap v2372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:58.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:57 smithi116 ceph-mon[35411]: pgmap v2372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:54:59.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:54:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:54:59.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:54:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:55:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:00 smithi114 ceph-mon[30820]: pgmap v2373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:00 smithi116 ceph-mon[35411]: pgmap v2373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:02 smithi114 ceph-mon[30820]: pgmap v2374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:55:01.973+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:55:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:55:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:55:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:55:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:55:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:55:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:55:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:55:02.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:55:02.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:55:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:55:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:02 smithi116 ceph-mon[35411]: pgmap v2374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:55:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:55:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:55:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:55:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:55:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:55:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:55:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:55:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:55:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:55:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:55:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:55:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:55:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:55:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:55:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:03 smithi114 ceph-mon[30820]: pgmap v2375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:55:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:03 smithi114 ceph-mon[30820]: pgmap v2376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:55:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:55:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:03 smithi116 ceph-mon[35411]: pgmap v2375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:55:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:03 smithi116 ceph-mon[35411]: pgmap v2376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:55:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:05 smithi114 ceph-mon[30820]: pgmap v2377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:55:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:05 smithi116 ceph-mon[35411]: pgmap v2377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:55:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:07 smithi114 ceph-mon[30820]: pgmap v2378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:55:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:07 smithi116 ceph-mon[35411]: pgmap v2378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:55:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:09 smithi114 ceph-mon[30820]: pgmap v2379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:55:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:09 smithi116 ceph-mon[35411]: pgmap v2379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:55:11.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:11 smithi114 ceph-mon[30820]: pgmap v2380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:55:11.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:11 smithi116 ceph-mon[35411]: pgmap v2380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:55:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:13 smithi114 ceph-mon[30820]: pgmap v2381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:55:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:13 smithi116 ceph-mon[35411]: pgmap v2381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:14 smithi114 ceph-mon[30820]: pgmap v2382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:14 smithi116 ceph-mon[35411]: pgmap v2382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:17 smithi114 ceph-mon[30820]: pgmap v2383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:17 smithi116 ceph-mon[35411]: pgmap v2383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:19 smithi114 ceph-mon[30820]: pgmap v2384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:19 smithi116 ceph-mon[35411]: pgmap v2384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:21 smithi114 ceph-mon[30820]: pgmap v2385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:55:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:21 smithi116 ceph-mon[35411]: pgmap v2385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:55:23.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:23 smithi114 ceph-mon[30820]: pgmap v2386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:55:23.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:23 smithi116 ceph-mon[35411]: pgmap v2386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:55:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:25 smithi114 ceph-mon[30820]: pgmap v2387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:25 smithi116 ceph-mon[35411]: pgmap v2387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:27 smithi114 ceph-mon[30820]: pgmap v2388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:27 smithi116 ceph-mon[35411]: pgmap v2388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:29 smithi114 ceph-mon[30820]: pgmap v2389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:55:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:29 smithi116 ceph-mon[35411]: pgmap v2389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:55:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:31 smithi114 ceph-mon[30820]: pgmap v2390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:55:31.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:31 smithi116 ceph-mon[35411]: pgmap v2390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T07:55:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:33 smithi114 ceph-mon[30820]: pgmap v2391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:55:33.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:33 smithi116 ceph-mon[35411]: pgmap v2391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:55:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:35 smithi114 ceph-mon[30820]: pgmap v2392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:55:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:35 smithi116 ceph-mon[35411]: pgmap v2392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T07:55:37.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:37 smithi114 ceph-mon[30820]: pgmap v2393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:37 smithi116 ceph-mon[35411]: pgmap v2393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:39 smithi114 ceph-mon[30820]: pgmap v2394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:39.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:39 smithi116 ceph-mon[35411]: pgmap v2394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:41 smithi114 ceph-mon[30820]: pgmap v2395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:55:41.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:41 smithi116 ceph-mon[35411]: pgmap v2395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:55:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:43 smithi114 ceph-mon[30820]: pgmap v2396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:55:43.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:43 smithi116 ceph-mon[35411]: pgmap v2396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:55:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:45 smithi114 ceph-mon[30820]: pgmap v2397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:45.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:45 smithi116 ceph-mon[35411]: pgmap v2397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:47 smithi114 ceph-mon[30820]: pgmap v2398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:47.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:47 smithi116 ceph-mon[35411]: pgmap v2398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:49 smithi114 ceph-mon[30820]: pgmap v2399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:49 smithi116 ceph-mon[35411]: pgmap v2399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:51 smithi114 ceph-mon[30820]: pgmap v2400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:55:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:51 smithi116 ceph-mon[35411]: pgmap v2400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:55:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:53 smithi114 ceph-mon[30820]: pgmap v2401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:55:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:53 smithi116 ceph-mon[35411]: pgmap v2401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:55:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:55 smithi114 ceph-mon[30820]: pgmap v2402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:55 smithi116 ceph-mon[35411]: pgmap v2402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:57 smithi114 ceph-mon[30820]: pgmap v2403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:57 smithi116 ceph-mon[35411]: pgmap v2403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:55:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:55:59 smithi114 ceph-mon[30820]: pgmap v2404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:55:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:55:59 smithi116 ceph-mon[35411]: pgmap v2404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:01 smithi114 ceph-mon[30820]: pgmap v2405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:01 smithi116 ceph-mon[35411]: pgmap v2405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:02.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:56:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:03 smithi114 ceph-mon[30820]: pgmap v2406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:03 smithi116 ceph-mon[35411]: pgmap v2406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:05.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-mon[30820]: pgmap v2407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:05.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:56:05.332+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:56:05.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:56:05.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:56:05.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:56:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:56:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:56:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:05 smithi116 ceph-mon[35411]: pgmap v2407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:56:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:56:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:56:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:56:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:56:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:56:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:56:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:56:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:56:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:56:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:56:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:56:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:56:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:56:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:56:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:56:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:07 smithi114 ceph-mon[30820]: pgmap v2408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:56:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:07 smithi114 ceph-mon[30820]: pgmap v2409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:56:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:07 smithi116 ceph-mon[35411]: pgmap v2408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:56:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:07 smithi116 ceph-mon[35411]: pgmap v2409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:56:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:08 smithi114 ceph-mon[30820]: pgmap v2410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:56:08.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:08 smithi116 ceph-mon[35411]: pgmap v2410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:56:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:10 smithi114 ceph-mon[30820]: pgmap v2411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:56:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:10 smithi116 ceph-mon[35411]: pgmap v2411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:56:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:12 smithi114 ceph-mon[30820]: pgmap v2412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:56:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:12 smithi116 ceph-mon[35411]: pgmap v2412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:56:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:14 smithi114 ceph-mon[30820]: pgmap v2413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:56:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:14 smithi116 ceph-mon[35411]: pgmap v2413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:56:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:16 smithi114 ceph-mon[30820]: pgmap v2414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:56:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:16 smithi116 ceph-mon[35411]: pgmap v2414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:56:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:18 smithi114 ceph-mon[30820]: pgmap v2415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:18 smithi116 ceph-mon[35411]: pgmap v2415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:20 smithi114 ceph-mon[30820]: pgmap v2416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:20 smithi116 ceph-mon[35411]: pgmap v2416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:22 smithi116 ceph-mon[35411]: pgmap v2417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:22.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:22 smithi114 ceph-mon[30820]: pgmap v2417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:24 smithi116 ceph-mon[35411]: pgmap v2418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:24 smithi114 ceph-mon[30820]: pgmap v2418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:26 smithi116 ceph-mon[35411]: pgmap v2419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:56:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:26 smithi114 ceph-mon[30820]: pgmap v2419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:56:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:28 smithi114 ceph-mon[30820]: pgmap v2420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:28 smithi116 ceph-mon[35411]: pgmap v2420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:30 smithi114 ceph-mon[30820]: pgmap v2421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:30 smithi116 ceph-mon[35411]: pgmap v2421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:32 smithi114 ceph-mon[30820]: pgmap v2422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:32 smithi116 ceph-mon[35411]: pgmap v2422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:34 smithi114 ceph-mon[30820]: pgmap v2423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:34 smithi116 ceph-mon[35411]: pgmap v2423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:36 smithi114 ceph-mon[30820]: pgmap v2424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:56:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:36 smithi116 ceph-mon[35411]: pgmap v2424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:56:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:38 smithi114 ceph-mon[30820]: pgmap v2425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:38 smithi116 ceph-mon[35411]: pgmap v2425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:40 smithi114 ceph-mon[30820]: pgmap v2426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:40.990 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:40 smithi116 ceph-mon[35411]: pgmap v2426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:42 smithi114 ceph-mon[30820]: pgmap v2427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:42 smithi116 ceph-mon[35411]: pgmap v2427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:44.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:44 smithi114 ceph-mon[30820]: pgmap v2428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:44 smithi116 ceph-mon[35411]: pgmap v2428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:46 smithi114 ceph-mon[30820]: pgmap v2429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:56:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:46 smithi116 ceph-mon[35411]: pgmap v2429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:56:48.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:48 smithi114 ceph-mon[30820]: pgmap v2430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:48.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:48 smithi116 ceph-mon[35411]: pgmap v2430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:50 smithi114 ceph-mon[30820]: pgmap v2431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:50 smithi116 ceph-mon[35411]: pgmap v2431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:56:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:52 smithi114 ceph-mon[30820]: pgmap v2432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:52 smithi116 ceph-mon[35411]: pgmap v2432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:56:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:54 smithi114 ceph-mon[30820]: pgmap v2433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:54 smithi116 ceph-mon[35411]: pgmap v2433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:56:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:56 smithi114 ceph-mon[30820]: pgmap v2434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:56:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:56 smithi116 ceph-mon[35411]: pgmap v2434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:56:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:56:58 smithi114 ceph-mon[30820]: pgmap v2435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:56:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:56:58 smithi116 ceph-mon[35411]: pgmap v2435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:57:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:00 smithi114 ceph-mon[30820]: pgmap v2436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:00 smithi116 ceph-mon[35411]: pgmap v2436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:02 smithi114 ceph-mon[30820]: pgmap v2437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:02 smithi116 ceph-mon[35411]: pgmap v2437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:04 smithi114 ceph-mon[30820]: pgmap v2438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:04 smithi116 ceph-mon[35411]: pgmap v2438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:57:06.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:06 smithi114 ceph-mon[30820]: pgmap v2439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:57:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:06 smithi116 ceph-mon[35411]: pgmap v2439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:57:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-mon[30820]: pgmap v2440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:57:08.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:57:08.692+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:57:08.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:57:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:57:08.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:57:08.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:57:08.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:57:08.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:57:08.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:57:08.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:57:08.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:57:08.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:57:08.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:57:08.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:57:08.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:57:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:57:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:08 smithi116 ceph-mon[35411]: pgmap v2440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:57:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:09 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:57:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:57:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:57:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:57:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:09 smithi114 ceph-mon[30820]: pgmap v2441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:09 smithi114 ceph-mon[30820]: pgmap v2442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:57:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:09 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:57:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:57:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:57:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:09 smithi116 ceph-mon[35411]: pgmap v2441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:57:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:09 smithi116 ceph-mon[35411]: pgmap v2442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:57:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:57:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:11 smithi114 ceph-mon[30820]: pgmap v2443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:11 smithi116 ceph-mon[35411]: pgmap v2443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:57:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:13 smithi114 ceph-mon[30820]: pgmap v2444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:57:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:13 smithi116 ceph-mon[35411]: pgmap v2444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:57:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:15 smithi114 ceph-mon[30820]: pgmap v2445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:57:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:15 smithi116 ceph-mon[35411]: pgmap v2445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T07:57:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:17 smithi114 ceph-mon[30820]: pgmap v2446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:57:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:17 smithi116 ceph-mon[35411]: pgmap v2446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:57:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:19 smithi114 ceph-mon[30820]: pgmap v2447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:57:20.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:19 smithi116 ceph-mon[35411]: pgmap v2447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:57:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:21 smithi114 ceph-mon[30820]: pgmap v2448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:57:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:21 smithi116 ceph-mon[35411]: pgmap v2448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:57:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:23 smithi114 ceph-mon[30820]: pgmap v2449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:23 smithi116 ceph-mon[35411]: pgmap v2449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:25 smithi114 ceph-mon[30820]: pgmap v2450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:25 smithi116 ceph-mon[35411]: pgmap v2450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:27 smithi114 ceph-mon[30820]: pgmap v2451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:27 smithi116 ceph-mon[35411]: pgmap v2451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:29 smithi114 ceph-mon[30820]: pgmap v2452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:29 smithi116 ceph-mon[35411]: pgmap v2452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:31 smithi114 ceph-mon[30820]: pgmap v2453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:57:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:31 smithi116 ceph-mon[35411]: pgmap v2453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:57:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:33 smithi114 ceph-mon[30820]: pgmap v2454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:33 smithi116 ceph-mon[35411]: pgmap v2454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:35 smithi114 ceph-mon[30820]: pgmap v2455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:35 smithi116 ceph-mon[35411]: pgmap v2455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:37 smithi114 ceph-mon[30820]: pgmap v2456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:37 smithi116 ceph-mon[35411]: pgmap v2456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:39 smithi114 ceph-mon[30820]: pgmap v2457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:39 smithi116 ceph-mon[35411]: pgmap v2457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:41 smithi114 ceph-mon[30820]: pgmap v2458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:57:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:41 smithi116 ceph-mon[35411]: pgmap v2458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:57:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:43 smithi114 ceph-mon[30820]: pgmap v2459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:43 smithi116 ceph-mon[35411]: pgmap v2459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:45 smithi114 ceph-mon[30820]: pgmap v2460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:45 smithi116 ceph-mon[35411]: pgmap v2460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:48.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:47 smithi114 ceph-mon[30820]: pgmap v2461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:47 smithi116 ceph-mon[35411]: pgmap v2461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:49 smithi114 ceph-mon[30820]: pgmap v2462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:49 smithi116 ceph-mon[35411]: pgmap v2462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:51 smithi114 ceph-mon[30820]: pgmap v2463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:57:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:51 smithi116 ceph-mon[35411]: pgmap v2463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:57:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:53 smithi114 ceph-mon[30820]: pgmap v2464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:53 smithi116 ceph-mon[35411]: pgmap v2464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:57:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:55 smithi114 ceph-mon[30820]: pgmap v2465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:55 smithi116 ceph-mon[35411]: pgmap v2465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:57:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:57 smithi114 ceph-mon[30820]: pgmap v2466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:57:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:57 smithi116 ceph-mon[35411]: pgmap v2466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:57:59 smithi114 ceph-mon[30820]: pgmap v2467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:57:59 smithi116 ceph-mon[35411]: pgmap v2467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:01 smithi114 ceph-mon[30820]: pgmap v2468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:01 smithi116 ceph-mon[35411]: pgmap v2468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:03 smithi114 ceph-mon[30820]: pgmap v2469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:03 smithi116 ceph-mon[35411]: pgmap v2469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:05 smithi114 ceph-mon[30820]: pgmap v2470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:58:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:05 smithi116 ceph-mon[35411]: pgmap v2470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:58:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:07 smithi114 ceph-mon[30820]: pgmap v2471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:07 smithi116 ceph-mon[35411]: pgmap v2471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:09 smithi114 ceph-mon[30820]: pgmap v2472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:09 smithi116 ceph-mon[35411]: pgmap v2472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:58:12.053 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:11 smithi114 ceph-mon[30820]: pgmap v2473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:11 smithi116 ceph-mon[35411]: pgmap v2473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:58:12.052+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:58:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:58:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:58:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:58:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:58:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:58:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:58:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:58:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:58:12.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:58:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:58:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:58:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:58:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:58:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:58:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:58:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:58:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:58:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:58:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:58:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:58:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:58:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:58:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:58:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:58:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:58:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-mon[30820]: pgmap v2474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-mon[30820]: pgmap v2475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:58:13.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:58:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:12 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:58:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:58:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:58:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:12 smithi116 ceph-mon[35411]: pgmap v2474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:12 smithi116 ceph-mon[35411]: pgmap v2475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:58:13.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:58:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:15 smithi114 ceph-mon[30820]: pgmap v2476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:58:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:15 smithi116 ceph-mon[35411]: pgmap v2476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T07:58:17.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:17 smithi114 ceph-mon[30820]: pgmap v2477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:58:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:17 smithi116 ceph-mon[35411]: pgmap v2477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:58:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:19 smithi114 ceph-mon[30820]: pgmap v2478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:58:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:19 smithi116 ceph-mon[35411]: pgmap v2478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:58:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:21 smithi114 ceph-mon[30820]: pgmap v2479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:58:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:21 smithi116 ceph-mon[35411]: pgmap v2479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T07:58:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:23 smithi114 ceph-mon[30820]: pgmap v2480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:58:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:23 smithi116 ceph-mon[35411]: pgmap v2480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:58:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:25 smithi114 ceph-mon[30820]: pgmap v2481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:25 smithi116 ceph-mon[35411]: pgmap v2481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:27 smithi114 ceph-mon[30820]: pgmap v2482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:27 smithi116 ceph-mon[35411]: pgmap v2482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:29 smithi114 ceph-mon[30820]: pgmap v2483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:29 smithi116 ceph-mon[35411]: pgmap v2483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:31 smithi114 ceph-mon[30820]: pgmap v2484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:58:31.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:31 smithi116 ceph-mon[35411]: pgmap v2484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:58:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:33 smithi114 ceph-mon[30820]: pgmap v2485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:33 smithi116 ceph-mon[35411]: pgmap v2485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:35 smithi114 ceph-mon[30820]: pgmap v2486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:35.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:35 smithi116 ceph-mon[35411]: pgmap v2486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:37 smithi114 ceph-mon[30820]: pgmap v2487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:37.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:37 smithi116 ceph-mon[35411]: pgmap v2487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:39 smithi116 ceph-mon[35411]: pgmap v2488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:39 smithi114 ceph-mon[30820]: pgmap v2488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:40 smithi114 ceph-mon[30820]: pgmap v2489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:58:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:40 smithi116 ceph-mon[35411]: pgmap v2489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:58:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:43 smithi114 ceph-mon[30820]: pgmap v2490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:43.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:43 smithi116 ceph-mon[35411]: pgmap v2490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:45 smithi114 ceph-mon[30820]: pgmap v2491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:45 smithi116 ceph-mon[35411]: pgmap v2491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:47 smithi114 ceph-mon[30820]: pgmap v2492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:47 smithi116 ceph-mon[35411]: pgmap v2492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:49 smithi114 ceph-mon[30820]: pgmap v2493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:49.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:49 smithi116 ceph-mon[35411]: pgmap v2493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:51 smithi114 ceph-mon[30820]: pgmap v2494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:58:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:51 smithi116 ceph-mon[35411]: pgmap v2494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:58:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:53 smithi114 ceph-mon[30820]: pgmap v2495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:53 smithi116 ceph-mon[35411]: pgmap v2495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:58:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:55 smithi114 ceph-mon[30820]: pgmap v2496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:55 smithi116 ceph-mon[35411]: pgmap v2496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:57 smithi114 ceph-mon[30820]: pgmap v2497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:57 smithi116 ceph-mon[35411]: pgmap v2497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:58:59.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:58:59 smithi114 ceph-mon[30820]: pgmap v2498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:58:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:58:59 smithi116 ceph-mon[35411]: pgmap v2498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:01 smithi114 ceph-mon[30820]: pgmap v2499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:01 smithi116 ceph-mon[35411]: pgmap v2499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:03 smithi114 ceph-mon[30820]: pgmap v2500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:03 smithi116 ceph-mon[35411]: pgmap v2500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:05 smithi114 ceph-mon[30820]: pgmap v2501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:05 smithi116 ceph-mon[35411]: pgmap v2501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:07 smithi114 ceph-mon[30820]: pgmap v2502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:59:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:07 smithi116 ceph-mon[35411]: pgmap v2502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T07:59:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:09 smithi116 ceph-mon[35411]: pgmap v2503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:09 smithi114 ceph-mon[30820]: pgmap v2503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:10.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:10 smithi114 ceph-mon[30820]: pgmap v2504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:10 smithi116 ceph-mon[35411]: pgmap v2504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:13 smithi114 ceph-mon[30820]: pgmap v2505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:13 smithi116 ceph-mon[35411]: pgmap v2505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T07:59:15.413 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-mon[30820]: pgmap v2506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:15 smithi116 ceph-mon[35411]: pgmap v2506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:15.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T07:59:15.412+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:59:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:59:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 07:59:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:59:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:59:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T07:59:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T07:59:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T07:59:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:59:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T07:59:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:59:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T07:59:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:17 smithi114 ceph-mon[30820]: pgmap v2507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:17 smithi114 ceph-mon[30820]: pgmap v2508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:59:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:59:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T07:59:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T07:59:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T07:59:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T07:59:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T07:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T07:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T07:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T07:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T07:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T07:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T07:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T07:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:17 smithi116 ceph-mon[35411]: pgmap v2507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T07:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:17 smithi116 ceph-mon[35411]: pgmap v2508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T07:59:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:19 smithi114 ceph-mon[30820]: pgmap v2509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:59:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:19 smithi116 ceph-mon[35411]: pgmap v2509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:59:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:21 smithi114 ceph-mon[30820]: pgmap v2510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:59:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:21 smithi116 ceph-mon[35411]: pgmap v2510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:59:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:23 smithi114 ceph-mon[30820]: pgmap v2511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:59:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:23 smithi116 ceph-mon[35411]: pgmap v2511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:59:25.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:25 smithi114 ceph-mon[30820]: pgmap v2512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:59:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:25 smithi116 ceph-mon[35411]: pgmap v2512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T07:59:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:27 smithi114 ceph-mon[30820]: pgmap v2513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:59:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:27 smithi116 ceph-mon[35411]: pgmap v2513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T07:59:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:29 smithi116 ceph-mon[35411]: pgmap v2514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:59:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:29 smithi114 ceph-mon[30820]: pgmap v2514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:59:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:31 smithi116 ceph-mon[35411]: pgmap v2515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:31 smithi114 ceph-mon[30820]: pgmap v2515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:33 smithi116 ceph-mon[35411]: pgmap v2516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:33 smithi114 ceph-mon[30820]: pgmap v2516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:34 smithi114 ceph-mon[30820]: pgmap v2517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:34 smithi116 ceph-mon[35411]: pgmap v2517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:36 smithi114 ceph-mon[30820]: pgmap v2518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:59:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:36 smithi116 ceph-mon[35411]: pgmap v2518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:59:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:38 smithi114 ceph-mon[30820]: pgmap v2519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:59:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:38 smithi116 ceph-mon[35411]: pgmap v2519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:59:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:40 smithi114 ceph-mon[30820]: pgmap v2520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:40 smithi116 ceph-mon[35411]: pgmap v2520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:42 smithi114 ceph-mon[30820]: pgmap v2521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:42 smithi116 ceph-mon[35411]: pgmap v2521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:44 smithi114 ceph-mon[30820]: pgmap v2522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:44 smithi116 ceph-mon[35411]: pgmap v2522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:46 smithi114 ceph-mon[30820]: pgmap v2523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:59:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:46 smithi116 ceph-mon[35411]: pgmap v2523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:59:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:48 smithi114 ceph-mon[30820]: pgmap v2524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:59:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:48 smithi116 ceph-mon[35411]: pgmap v2524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:59:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:50 smithi114 ceph-mon[30820]: pgmap v2525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:50 smithi116 ceph-mon[35411]: pgmap v2525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T07:59:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:52 smithi114 ceph-mon[30820]: pgmap v2526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:52 smithi116 ceph-mon[35411]: pgmap v2526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T07:59:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:54 smithi114 ceph-mon[30820]: pgmap v2527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:54 smithi116 ceph-mon[35411]: pgmap v2527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T07:59:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:56 smithi114 ceph-mon[30820]: pgmap v2528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:59:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:56 smithi116 ceph-mon[35411]: pgmap v2528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T07:59:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 07:59:58 smithi114 ceph-mon[30820]: pgmap v2529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T07:59:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 07:59:58 smithi116 ceph-mon[35411]: pgmap v2529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:00:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:00 smithi114 ceph-mon[30820]: pgmap v2530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:00:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:00 smithi116 ceph-mon[35411]: pgmap v2530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:00:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:02 smithi114 ceph-mon[30820]: pgmap v2531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:02 smithi116 ceph-mon[35411]: pgmap v2531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:04.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:04 smithi114 ceph-mon[30820]: pgmap v2532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:04 smithi116 ceph-mon[35411]: pgmap v2532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:06 smithi114 ceph-mon[30820]: pgmap v2533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:00:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:06 smithi116 ceph-mon[35411]: pgmap v2533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:00:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:08 smithi114 ceph-mon[30820]: pgmap v2534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:00:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:08 smithi116 ceph-mon[35411]: pgmap v2534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:00:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:10 smithi114 ceph-mon[30820]: pgmap v2535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:10 smithi116 ceph-mon[35411]: pgmap v2535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:12 smithi114 ceph-mon[30820]: pgmap v2536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:12 smithi116 ceph-mon[35411]: pgmap v2536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:14 smithi114 ceph-mon[30820]: pgmap v2537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:14.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:14 smithi116 ceph-mon[35411]: pgmap v2537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:00:16.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:16 smithi114 ceph-mon[30820]: pgmap v2538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:00:16.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:16 smithi116 ceph-mon[35411]: pgmap v2538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:00:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-mon[30820]: pgmap v2539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:00:18.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:00:18.772+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:00:18.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:00:18.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:00:18.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:00:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:00:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:00:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:00:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:00:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:00:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:00:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:00:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:00:18.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:18 smithi116 ceph-mon[35411]: pgmap v2539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:00:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:00:19.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:00:19.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:00:19.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:00:19.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:00:19.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:00:19.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:00:19.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:00:19.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:00:19.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:19.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:19.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:19 smithi114 ceph-mon[30820]: pgmap v2540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:00:19.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:19 smithi114 ceph-mon[30820]: pgmap v2541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:00:19.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:00:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:00:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:00:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:19 smithi116 ceph-mon[35411]: pgmap v2540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:00:19.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:19 smithi116 ceph-mon[35411]: pgmap v2541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:00:19.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:00:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:21 smithi114 ceph-mon[30820]: pgmap v2542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:00:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:21 smithi116 ceph-mon[35411]: pgmap v2542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:00:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:23 smithi114 ceph-mon[30820]: pgmap v2543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:00:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:23 smithi116 ceph-mon[35411]: pgmap v2543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:00:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:25 smithi114 ceph-mon[30820]: pgmap v2544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:00:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:25 smithi116 ceph-mon[35411]: pgmap v2544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:00:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:27 smithi114 ceph-mon[30820]: pgmap v2545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:00:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:27 smithi116 ceph-mon[35411]: pgmap v2545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:00:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:29 smithi114 ceph-mon[30820]: pgmap v2546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:00:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:29 smithi116 ceph-mon[35411]: pgmap v2546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:31 smithi114 ceph-mon[30820]: pgmap v2547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:00:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:31 smithi116 ceph-mon[35411]: pgmap v2547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:00:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:33 smithi114 ceph-mon[30820]: pgmap v2548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:33 smithi116 ceph-mon[35411]: pgmap v2548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:36.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:35 smithi114 ceph-mon[30820]: pgmap v2549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:35 smithi116 ceph-mon[35411]: pgmap v2549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:37 smithi114 ceph-mon[30820]: pgmap v2550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:37 smithi116 ceph-mon[35411]: pgmap v2550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:39 smithi114 ceph-mon[30820]: pgmap v2551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:39 smithi116 ceph-mon[35411]: pgmap v2551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:41 smithi114 ceph-mon[30820]: pgmap v2552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:00:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:41 smithi116 ceph-mon[35411]: pgmap v2552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:00:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:43 smithi114 ceph-mon[30820]: pgmap v2553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:43 smithi116 ceph-mon[35411]: pgmap v2553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:45 smithi114 ceph-mon[30820]: pgmap v2554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:45 smithi116 ceph-mon[35411]: pgmap v2554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:47 smithi114 ceph-mon[30820]: pgmap v2555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:47 smithi116 ceph-mon[35411]: pgmap v2555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:49 smithi114 ceph-mon[30820]: pgmap v2556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:49 smithi116 ceph-mon[35411]: pgmap v2556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:51 smithi114 ceph-mon[30820]: pgmap v2557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:00:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:51 smithi116 ceph-mon[35411]: pgmap v2557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:00:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:53 smithi114 ceph-mon[30820]: pgmap v2558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:53 smithi116 ceph-mon[35411]: pgmap v2558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:00:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:55 smithi114 ceph-mon[30820]: pgmap v2559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:55 smithi116 ceph-mon[35411]: pgmap v2559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:00:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:57 smithi114 ceph-mon[30820]: pgmap v2560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:00:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:57 smithi116 ceph-mon[35411]: pgmap v2560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:00:59 smithi114 ceph-mon[30820]: pgmap v2561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:00:59 smithi116 ceph-mon[35411]: pgmap v2561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:01.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:01 smithi114 ceph-mon[30820]: pgmap v2562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:01 smithi116 ceph-mon[35411]: pgmap v2562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:03 smithi114 ceph-mon[30820]: pgmap v2563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:03 smithi116 ceph-mon[35411]: pgmap v2563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:05 smithi114 ceph-mon[30820]: pgmap v2564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:01:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:05 smithi116 ceph-mon[35411]: pgmap v2564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:01:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:07 smithi114 ceph-mon[30820]: pgmap v2565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:07 smithi116 ceph-mon[35411]: pgmap v2565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:09 smithi114 ceph-mon[30820]: pgmap v2566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:09 smithi116 ceph-mon[35411]: pgmap v2566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:11 smithi114 ceph-mon[30820]: pgmap v2567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:11 smithi116 ceph-mon[35411]: pgmap v2567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:13 smithi114 ceph-mon[30820]: pgmap v2568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:13 smithi116 ceph-mon[35411]: pgmap v2568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:15 smithi114 ceph-mon[30820]: pgmap v2569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:01:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:15 smithi116 ceph-mon[35411]: pgmap v2569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:01:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:17 smithi114 ceph-mon[30820]: pgmap v2570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:18.184 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:17 smithi116 ceph-mon[35411]: pgmap v2570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: pgmap v2571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:01:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: pgmap v2571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:01:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:20 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:01:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:20 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:01:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:20 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:01:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:20 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:01:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:20 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:01:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:20 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:01:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:20 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:01:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:20 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:01:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:21 smithi114 ceph-mon[30820]: pgmap v2572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:01:22.132+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:01:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:01:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:01:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:01:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:01:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:01:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:01:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:01:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:01:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:01:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:01:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:01:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:21 smithi116 ceph-mon[35411]: pgmap v2572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:01:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:01:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:01:23.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:01:23.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-mon[30820]: pgmap v2573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:01:23.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-mon[30820]: pgmap v2574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:01:23.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:22 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:01:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:01:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:01:23.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:01:23.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:01:23.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:01:23.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:01:23.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:22 smithi116 ceph-mon[35411]: pgmap v2573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:22 smithi116 ceph-mon[35411]: pgmap v2574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:01:23.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:01:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:25 smithi114 ceph-mon[30820]: pgmap v2575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:01:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:25 smithi116 ceph-mon[35411]: pgmap v2575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:01:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:27 smithi114 ceph-mon[30820]: pgmap v2576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:01:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:27 smithi116 ceph-mon[35411]: pgmap v2576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:01:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:28 smithi114 ceph-mon[30820]: pgmap v2577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:01:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:28 smithi116 ceph-mon[35411]: pgmap v2577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:01:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:31 smithi114 ceph-mon[30820]: pgmap v2578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T08:01:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:31 smithi116 ceph-mon[35411]: pgmap v2578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T08:01:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:32 smithi114 ceph-mon[30820]: pgmap v2579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:01:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:32 smithi116 ceph-mon[35411]: pgmap v2579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:01:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:35 smithi114 ceph-mon[30820]: pgmap v2580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:01:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:35 smithi116 ceph-mon[35411]: pgmap v2580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:01:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:36 smithi114 ceph-mon[30820]: pgmap v2581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:36 smithi116 ceph-mon[35411]: pgmap v2581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:39 smithi114 ceph-mon[30820]: pgmap v2582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:39 smithi116 ceph-mon[35411]: pgmap v2582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:40 smithi114 ceph-mon[30820]: pgmap v2583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:01:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:40 smithi116 ceph-mon[35411]: pgmap v2583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:01:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:43 smithi114 ceph-mon[30820]: pgmap v2584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:43 smithi116 ceph-mon[35411]: pgmap v2584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:44 smithi114 ceph-mon[30820]: pgmap v2585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:44 smithi116 ceph-mon[35411]: pgmap v2585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:47 smithi114 ceph-mon[30820]: pgmap v2586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:47 smithi116 ceph-mon[35411]: pgmap v2586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:48 smithi114 ceph-mon[30820]: pgmap v2587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:48.724 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:48 smithi116 ceph-mon[35411]: pgmap v2587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:50 smithi114 ceph-mon[30820]: pgmap v2588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:01:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:50 smithi116 ceph-mon[35411]: pgmap v2588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:01:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:52 smithi114 ceph-mon[30820]: pgmap v2589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:52 smithi116 ceph-mon[35411]: pgmap v2589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:01:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:54 smithi114 ceph-mon[30820]: pgmap v2590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:54 smithi116 ceph-mon[35411]: pgmap v2590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:56 smithi114 ceph-mon[30820]: pgmap v2591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:56 smithi116 ceph-mon[35411]: pgmap v2591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:01:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:01:58 smithi114 ceph-mon[30820]: pgmap v2592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:01:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:01:58 smithi116 ceph-mon[35411]: pgmap v2592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:00 smithi114 ceph-mon[30820]: pgmap v2593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:00 smithi116 ceph-mon[35411]: pgmap v2593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:02 smithi114 ceph-mon[30820]: pgmap v2594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:02 smithi116 ceph-mon[35411]: pgmap v2594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:04 smithi114 ceph-mon[30820]: pgmap v2595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:04 smithi116 ceph-mon[35411]: pgmap v2595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:06 smithi114 ceph-mon[30820]: pgmap v2596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:02:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:06 smithi116 ceph-mon[35411]: pgmap v2596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:02:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:08 smithi114 ceph-mon[30820]: pgmap v2597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:08 smithi116 ceph-mon[35411]: pgmap v2597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:10 smithi114 ceph-mon[30820]: pgmap v2598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:10 smithi116 ceph-mon[35411]: pgmap v2598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:12 smithi114 ceph-mon[30820]: pgmap v2599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:12 smithi116 ceph-mon[35411]: pgmap v2599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:14 smithi114 ceph-mon[30820]: pgmap v2600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:14.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:14 smithi116 ceph-mon[35411]: pgmap v2600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:16 smithi114 ceph-mon[30820]: pgmap v2601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:02:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:16 smithi116 ceph-mon[35411]: pgmap v2601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:02:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:18 smithi114 ceph-mon[30820]: pgmap v2602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:18 smithi116 ceph-mon[35411]: pgmap v2602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:20 smithi114 ceph-mon[30820]: pgmap v2603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:20 smithi116 ceph-mon[35411]: pgmap v2603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:22 smithi114 ceph-mon[30820]: pgmap v2604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:22 smithi116 ceph-mon[35411]: pgmap v2604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:02:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:02:24.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:24 smithi114 ceph-mon[30820]: pgmap v2605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:24.579 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:24 smithi116 ceph-mon[35411]: pgmap v2605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:02:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:02:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:02:25.428+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:02:25.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:02:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:02:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:02:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:02:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:02:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:02:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:02:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:02:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:02:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:02:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:02:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:02:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:02:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:02:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:02:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:02:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:02:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:26 smithi114 ceph-mon[30820]: pgmap v2606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T08:02:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:26 smithi114 ceph-mon[30820]: pgmap v2607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:02:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:02:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:02:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:02:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:02:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:02:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:02:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:02:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:02:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:02:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:02:26.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:02:26.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:02:26.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:26 smithi116 ceph-mon[35411]: pgmap v2606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T08:02:26.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:26 smithi116 ceph-mon[35411]: pgmap v2607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:28 smithi114 ceph-mon[30820]: pgmap v2608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:28 smithi116 ceph-mon[35411]: pgmap v2608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:30 smithi114 ceph-mon[30820]: pgmap v2609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:30 smithi116 ceph-mon[35411]: pgmap v2609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:32 smithi114 ceph-mon[30820]: pgmap v2610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:32 smithi116 ceph-mon[35411]: pgmap v2610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:34 smithi114 ceph-mon[30820]: pgmap v2611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:34.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:34 smithi116 ceph-mon[35411]: pgmap v2611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:02:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:36 smithi114 ceph-mon[30820]: pgmap v2612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:02:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:36 smithi116 ceph-mon[35411]: pgmap v2612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:02:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:38 smithi114 ceph-mon[30820]: pgmap v2613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:38 smithi116 ceph-mon[35411]: pgmap v2613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:40.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:40 smithi114 ceph-mon[30820]: pgmap v2614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:40 smithi116 ceph-mon[35411]: pgmap v2614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:42 smithi114 ceph-mon[30820]: pgmap v2615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:42 smithi116 ceph-mon[35411]: pgmap v2615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:44 smithi114 ceph-mon[30820]: pgmap v2616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:44 smithi116 ceph-mon[35411]: pgmap v2616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:46 smithi114 ceph-mon[30820]: pgmap v2617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:02:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:46 smithi116 ceph-mon[35411]: pgmap v2617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:02:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:48 smithi114 ceph-mon[30820]: pgmap v2618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:48 smithi116 ceph-mon[35411]: pgmap v2618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:50 smithi114 ceph-mon[30820]: pgmap v2619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:50 smithi116 ceph-mon[35411]: pgmap v2619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:02:52.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:52 smithi114 ceph-mon[30820]: pgmap v2620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:52 smithi116 ceph-mon[35411]: pgmap v2620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:02:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:54 smithi114 ceph-mon[30820]: pgmap v2621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:54 smithi116 ceph-mon[35411]: pgmap v2621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:56 smithi114 ceph-mon[30820]: pgmap v2622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:02:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:56 smithi116 ceph-mon[35411]: pgmap v2622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:02:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:02:58 smithi114 ceph-mon[30820]: pgmap v2623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:02:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:02:58 smithi116 ceph-mon[35411]: pgmap v2623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:00 smithi114 ceph-mon[30820]: pgmap v2624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:03:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:00 smithi116 ceph-mon[35411]: pgmap v2624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:03:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:02 smithi114 ceph-mon[30820]: pgmap v2625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:02 smithi116 ceph-mon[35411]: pgmap v2625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:04 smithi114 ceph-mon[30820]: pgmap v2626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:04 smithi116 ceph-mon[35411]: pgmap v2626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:06 smithi114 ceph-mon[30820]: pgmap v2627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:06 smithi116 ceph-mon[35411]: pgmap v2627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:08 smithi114 ceph-mon[30820]: pgmap v2628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:08 smithi116 ceph-mon[35411]: pgmap v2628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:10 smithi114 ceph-mon[30820]: pgmap v2629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:03:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:10 smithi116 ceph-mon[35411]: pgmap v2629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:03:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:12 smithi114 ceph-mon[30820]: pgmap v2630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:12.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:12 smithi116 ceph-mon[35411]: pgmap v2630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:14 smithi114 ceph-mon[30820]: pgmap v2631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:14.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:14 smithi116 ceph-mon[35411]: pgmap v2631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:16 smithi114 ceph-mon[30820]: pgmap v2632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:16.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:16 smithi116 ceph-mon[35411]: pgmap v2632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:18 smithi114 ceph-mon[30820]: pgmap v2633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:03:19.044 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:18 smithi116 ceph-mon[35411]: pgmap v2633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:03:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:20 smithi114 ceph-mon[30820]: pgmap v2634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:03:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:20 smithi116 ceph-mon[35411]: pgmap v2634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:03:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:22 smithi114 ceph-mon[30820]: pgmap v2635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:22 smithi116 ceph-mon[35411]: pgmap v2635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:24 smithi114 ceph-mon[30820]: pgmap v2636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:03:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:24 smithi116 ceph-mon[35411]: pgmap v2636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:03:25.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:03:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:26 smithi114 ceph-mon[30820]: pgmap v2637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:26 smithi116 ceph-mon[35411]: pgmap v2637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-mon[30820]: pgmap v2638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:28.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:03:28.788+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:03:28.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:03:28.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:03:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:03:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:03:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:03:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:03:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:03:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:03:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:03:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:03:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:03:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:03:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:28 smithi116 ceph-mon[35411]: pgmap v2638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:29 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:03:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:03:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:03:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:29 smithi114 ceph-mon[30820]: pgmap v2639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:29 smithi114 ceph-mon[30820]: pgmap v2640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:03:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:29 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:03:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:03:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:29 smithi116 ceph-mon[35411]: pgmap v2639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:03:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:29 smithi116 ceph-mon[35411]: pgmap v2640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:03:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:03:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:31 smithi114 ceph-mon[30820]: pgmap v2641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:03:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:31 smithi116 ceph-mon[35411]: pgmap v2641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:03:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:33 smithi114 ceph-mon[30820]: pgmap v2642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:03:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:33 smithi116 ceph-mon[35411]: pgmap v2642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:03:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:35 smithi114 ceph-mon[30820]: pgmap v2643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T08:03:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:35 smithi116 ceph-mon[35411]: pgmap v2643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T08:03:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:37 smithi114 ceph-mon[30820]: pgmap v2644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:03:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:37 smithi116 ceph-mon[35411]: pgmap v2644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:03:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:39 smithi114 ceph-mon[30820]: pgmap v2645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:03:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:39 smithi116 ceph-mon[35411]: pgmap v2645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:41 smithi114 ceph-mon[30820]: pgmap v2646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:41 smithi116 ceph-mon[35411]: pgmap v2646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:43 smithi114 ceph-mon[30820]: pgmap v2647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:43 smithi116 ceph-mon[35411]: pgmap v2647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:45 smithi114 ceph-mon[30820]: pgmap v2648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:03:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:45 smithi116 ceph-mon[35411]: pgmap v2648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:03:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:47 smithi114 ceph-mon[30820]: pgmap v2649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:47 smithi116 ceph-mon[35411]: pgmap v2649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:49 smithi114 ceph-mon[30820]: pgmap v2650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:50.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:49 smithi116 ceph-mon[35411]: pgmap v2650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:51 smithi114 ceph-mon[30820]: pgmap v2651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:51 smithi116 ceph-mon[35411]: pgmap v2651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:03:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:53 smithi114 ceph-mon[30820]: pgmap v2652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:53 smithi116 ceph-mon[35411]: pgmap v2652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:03:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:55 smithi114 ceph-mon[30820]: pgmap v2653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:03:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:55 smithi116 ceph-mon[35411]: pgmap v2653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:03:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:57 smithi114 ceph-mon[30820]: pgmap v2654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:03:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:57 smithi116 ceph-mon[35411]: pgmap v2654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:03:59 smithi114 ceph-mon[30820]: pgmap v2655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:03:59 smithi116 ceph-mon[35411]: pgmap v2655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:01 smithi114 ceph-mon[30820]: pgmap v2656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:01 smithi116 ceph-mon[35411]: pgmap v2656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:03 smithi114 ceph-mon[30820]: pgmap v2657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:03 smithi116 ceph-mon[35411]: pgmap v2657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:05 smithi114 ceph-mon[30820]: pgmap v2658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:04:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:05 smithi116 ceph-mon[35411]: pgmap v2658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:04:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:07 smithi114 ceph-mon[30820]: pgmap v2659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:07 smithi116 ceph-mon[35411]: pgmap v2659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:09 smithi116 ceph-mon[35411]: pgmap v2660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:09 smithi114 ceph-mon[30820]: pgmap v2660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:11 smithi116 ceph-mon[35411]: pgmap v2661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:11 smithi114 ceph-mon[30820]: pgmap v2661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:13 smithi116 ceph-mon[35411]: pgmap v2662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:14.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:13 smithi114 ceph-mon[30820]: pgmap v2662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:15 smithi114 ceph-mon[30820]: pgmap v2663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:04:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:15 smithi116 ceph-mon[35411]: pgmap v2663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:04:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:17 smithi114 ceph-mon[30820]: pgmap v2664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:18.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:17 smithi116 ceph-mon[35411]: pgmap v2664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:19 smithi114 ceph-mon[30820]: pgmap v2665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:20.451 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:19 smithi116 ceph-mon[35411]: pgmap v2665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:21 smithi114 ceph-mon[30820]: pgmap v2666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:22.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:21 smithi116 ceph-mon[35411]: pgmap v2666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:23 smithi114 ceph-mon[30820]: pgmap v2667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:23 smithi116 ceph-mon[35411]: pgmap v2667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:26.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:25 smithi114 ceph-mon[30820]: pgmap v2668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:04:26.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:25 smithi116 ceph-mon[35411]: pgmap v2668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:04:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:27 smithi114 ceph-mon[30820]: pgmap v2669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:27 smithi116 ceph-mon[35411]: pgmap v2669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:29.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:04:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:04:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:29 smithi114 ceph-mon[30820]: pgmap v2670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:29 smithi116 ceph-mon[35411]: pgmap v2670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:31 smithi114 ceph-mon[30820]: pgmap v2671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:31 smithi116 ceph-mon[35411]: pgmap v2671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:04:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:04:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:04:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:04:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-mon[30820]: pgmap v2672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:04:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-mon[30820]: pgmap v2673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:04:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:04:32.148+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:04:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:04:32.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:04:32.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:04:32.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:04:32.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:04:32.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:04:32.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:04:32.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:04:32.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:04:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:32 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:04:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:04:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:04:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:04:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:04:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:04:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:04:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:32 smithi116 ceph-mon[35411]: pgmap v2672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:04:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:32 smithi116 ceph-mon[35411]: pgmap v2673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:04:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:04:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:35 smithi114 ceph-mon[30820]: pgmap v2674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:04:35.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:35 smithi116 ceph-mon[35411]: pgmap v2674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:04:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:37 smithi114 ceph-mon[30820]: pgmap v2675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:04:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:37 smithi116 ceph-mon[35411]: pgmap v2675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:04:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:39 smithi114 ceph-mon[30820]: pgmap v2676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:04:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:39 smithi116 ceph-mon[35411]: pgmap v2676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:41 smithi114 ceph-mon[30820]: pgmap v2677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:04:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:41 smithi116 ceph-mon[35411]: pgmap v2677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:04:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:43 smithi114 ceph-mon[30820]: pgmap v2678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:04:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:43 smithi116 ceph-mon[35411]: pgmap v2678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:04:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:45 smithi114 ceph-mon[30820]: pgmap v2679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:45 smithi116 ceph-mon[35411]: pgmap v2679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:47 smithi114 ceph-mon[30820]: pgmap v2680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:47 smithi116 ceph-mon[35411]: pgmap v2680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:48 smithi114 ceph-mon[30820]: pgmap v2681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:48 smithi116 ceph-mon[35411]: pgmap v2681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:51 smithi114 ceph-mon[30820]: pgmap v2682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:04:51.684 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:51 smithi116 ceph-mon[35411]: pgmap v2682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:04:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:53 smithi114 ceph-mon[30820]: pgmap v2683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:53.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:53 smithi116 ceph-mon[35411]: pgmap v2683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:04:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:55 smithi114 ceph-mon[30820]: pgmap v2684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:55.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:55 smithi116 ceph-mon[35411]: pgmap v2684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:57.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:57 smithi116 ceph-mon[35411]: pgmap v2685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:57.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:57 smithi114 ceph-mon[30820]: pgmap v2685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:04:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:04:59 smithi114 ceph-mon[30820]: pgmap v2686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:04:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:04:59 smithi116 ceph-mon[35411]: pgmap v2686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:01 smithi114 ceph-mon[30820]: pgmap v2687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:01 smithi116 ceph-mon[35411]: pgmap v2687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:02 smithi114 ceph-mon[30820]: pgmap v2688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:02 smithi116 ceph-mon[35411]: pgmap v2688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:05 smithi114 ceph-mon[30820]: pgmap v2689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:05.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:05 smithi116 ceph-mon[35411]: pgmap v2689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:07 smithi114 ceph-mon[30820]: pgmap v2690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:05:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:07 smithi116 ceph-mon[35411]: pgmap v2690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:05:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:09 smithi114 ceph-mon[30820]: pgmap v2691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:09 smithi116 ceph-mon[35411]: pgmap v2691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:11 smithi114 ceph-mon[30820]: pgmap v2692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:11.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:11 smithi116 ceph-mon[35411]: pgmap v2692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:13 smithi114 ceph-mon[30820]: pgmap v2693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:13 smithi116 ceph-mon[35411]: pgmap v2693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:15 smithi114 ceph-mon[30820]: pgmap v2694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:15 smithi116 ceph-mon[35411]: pgmap v2694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:16 smithi114 ceph-mon[30820]: pgmap v2695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:05:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:16 smithi116 ceph-mon[35411]: pgmap v2695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:05:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:19 smithi114 ceph-mon[30820]: pgmap v2696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:19 smithi116 ceph-mon[35411]: pgmap v2696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:21 smithi114 ceph-mon[30820]: pgmap v2697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:21 smithi116 ceph-mon[35411]: pgmap v2697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:23 smithi114 ceph-mon[30820]: pgmap v2698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:23 smithi116 ceph-mon[35411]: pgmap v2698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:25 smithi114 ceph-mon[30820]: pgmap v2699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:25 smithi116 ceph-mon[35411]: pgmap v2699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:26 smithi114 ceph-mon[30820]: pgmap v2700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:05:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:26 smithi116 ceph-mon[35411]: pgmap v2700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:05:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:29 smithi114 ceph-mon[30820]: pgmap v2701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:29 smithi116 ceph-mon[35411]: pgmap v2701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:31 smithi114 ceph-mon[30820]: pgmap v2702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:31 smithi116 ceph-mon[35411]: pgmap v2702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:33 smithi114 ceph-mon[30820]: pgmap v2703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:05:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:33 smithi116 ceph-mon[35411]: pgmap v2703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:05:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:35.510 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-mon[30820]: pgmap v2704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:35.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:35 smithi116 ceph-mon[35411]: pgmap v2704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:05:35.508+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:05:35.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:05:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:05:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:05:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:05:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:05:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:05:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:05:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:05:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:37 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:05:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:05:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:05:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:05:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:05:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:37 smithi114 ceph-mon[30820]: pgmap v2705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T08:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:37 smithi114 ceph-mon[30820]: pgmap v2706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:37 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:05:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:37 smithi116 ceph-mon[35411]: pgmap v2705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T08:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:37 smithi116 ceph-mon[35411]: pgmap v2706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:38 smithi114 ceph-mon[30820]: pgmap v2707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:38 smithi116 ceph-mon[35411]: pgmap v2707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:40 smithi114 ceph-mon[30820]: pgmap v2708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:40 smithi116 ceph-mon[35411]: pgmap v2708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:42 smithi114 ceph-mon[30820]: pgmap v2709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:42 smithi116 ceph-mon[35411]: pgmap v2709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:44 smithi114 ceph-mon[30820]: pgmap v2710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:44 smithi116 ceph-mon[35411]: pgmap v2710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:05:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:46 smithi114 ceph-mon[30820]: pgmap v2711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:05:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:46 smithi116 ceph-mon[35411]: pgmap v2711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:05:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:48 smithi114 ceph-mon[30820]: pgmap v2712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:48 smithi116 ceph-mon[35411]: pgmap v2712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:50 smithi114 ceph-mon[30820]: pgmap v2713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:50 smithi116 ceph-mon[35411]: pgmap v2713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:05:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:52 smithi114 ceph-mon[30820]: pgmap v2714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:52 smithi116 ceph-mon[35411]: pgmap v2714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:05:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:54 smithi114 ceph-mon[30820]: pgmap v2715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:54 smithi116 ceph-mon[35411]: pgmap v2715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:56 smithi114 ceph-mon[30820]: pgmap v2716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:05:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:56 smithi116 ceph-mon[35411]: pgmap v2716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:05:58.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:05:58 smithi114 ceph-mon[30820]: pgmap v2717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:05:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:05:58 smithi116 ceph-mon[35411]: pgmap v2717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:00 smithi114 ceph-mon[30820]: pgmap v2718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:00 smithi116 ceph-mon[35411]: pgmap v2718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:02 smithi114 ceph-mon[30820]: pgmap v2719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:02 smithi116 ceph-mon[35411]: pgmap v2719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:04 smithi114 ceph-mon[30820]: pgmap v2720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:04 smithi116 ceph-mon[35411]: pgmap v2720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:06 smithi114 ceph-mon[30820]: pgmap v2721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:06 smithi116 ceph-mon[35411]: pgmap v2721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:08 smithi114 ceph-mon[30820]: pgmap v2722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:08 smithi116 ceph-mon[35411]: pgmap v2722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:10 smithi114 ceph-mon[30820]: pgmap v2723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:10 smithi116 ceph-mon[35411]: pgmap v2723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:12 smithi114 ceph-mon[30820]: pgmap v2724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:12.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:12 smithi116 ceph-mon[35411]: pgmap v2724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:14 smithi114 ceph-mon[30820]: pgmap v2725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:14 smithi116 ceph-mon[35411]: pgmap v2725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:16 smithi114 ceph-mon[30820]: pgmap v2726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:16 smithi116 ceph-mon[35411]: pgmap v2726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:18 smithi114 ceph-mon[30820]: pgmap v2727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:18 smithi116 ceph-mon[35411]: pgmap v2727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:20 smithi114 ceph-mon[30820]: pgmap v2728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:20 smithi116 ceph-mon[35411]: pgmap v2728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:22 smithi114 ceph-mon[30820]: pgmap v2729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:22 smithi116 ceph-mon[35411]: pgmap v2729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:24 smithi114 ceph-mon[30820]: pgmap v2730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:24 smithi116 ceph-mon[35411]: pgmap v2730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:26 smithi114 ceph-mon[30820]: pgmap v2731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:26 smithi116 ceph-mon[35411]: pgmap v2731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:28 smithi114 ceph-mon[30820]: pgmap v2732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:28 smithi116 ceph-mon[35411]: pgmap v2732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:30 smithi116 ceph-mon[35411]: pgmap v2733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:30 smithi114 ceph-mon[30820]: pgmap v2733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:32 smithi116 ceph-mon[35411]: pgmap v2734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:32 smithi114 ceph-mon[30820]: pgmap v2734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:34 smithi116 ceph-mon[35411]: pgmap v2735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:34 smithi114 ceph-mon[30820]: pgmap v2735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:06:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:06:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:36 smithi114 ceph-mon[30820]: pgmap v2736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:36 smithi116 ceph-mon[35411]: pgmap v2736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-mon[30820]: pgmap v2737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:39.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:06:38.868+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:06:39.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:06:39.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:06:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:06:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:06:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:06:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:06:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:06:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:06:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:06:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:06:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:06:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:06:39.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:38 smithi116 ceph-mon[35411]: pgmap v2737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:06:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:06:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:06:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:06:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:06:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:06:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:06:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:06:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:06:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:06:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:06:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:06:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:39 smithi114 ceph-mon[30820]: pgmap v2738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:06:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:39 smithi114 ceph-mon[30820]: pgmap v2739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:06:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:06:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:06:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:39 smithi116 ceph-mon[35411]: pgmap v2738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:06:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:39 smithi116 ceph-mon[35411]: pgmap v2739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:06:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:06:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:41 smithi114 ceph-mon[30820]: pgmap v2740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:06:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:41 smithi116 ceph-mon[35411]: pgmap v2740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:06:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:43 smithi114 ceph-mon[30820]: pgmap v2741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:06:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:43 smithi116 ceph-mon[35411]: pgmap v2741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:06:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:45 smithi114 ceph-mon[30820]: pgmap v2742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:06:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:45 smithi116 ceph-mon[35411]: pgmap v2742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:06:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:47 smithi114 ceph-mon[30820]: pgmap v2743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:06:48.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:47 smithi116 ceph-mon[35411]: pgmap v2743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:06:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:49 smithi114 ceph-mon[30820]: pgmap v2744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:06:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:49 smithi116 ceph-mon[35411]: pgmap v2744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:06:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:51 smithi114 ceph-mon[30820]: pgmap v2745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:51 smithi116 ceph-mon[35411]: pgmap v2745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:06:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:53 smithi114 ceph-mon[30820]: pgmap v2746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:54.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:53 smithi116 ceph-mon[35411]: pgmap v2746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:06:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:55 smithi114 ceph-mon[30820]: pgmap v2747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:55 smithi116 ceph-mon[35411]: pgmap v2747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:06:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:57 smithi114 ceph-mon[30820]: pgmap v2748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:06:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:57 smithi116 ceph-mon[35411]: pgmap v2748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:00.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:06:59 smithi114 ceph-mon[30820]: pgmap v2749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:00.229 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:06:59 smithi116 ceph-mon[35411]: pgmap v2749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:01 smithi114 ceph-mon[30820]: pgmap v2750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:01 smithi116 ceph-mon[35411]: pgmap v2750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:03 smithi114 ceph-mon[30820]: pgmap v2751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:03 smithi116 ceph-mon[35411]: pgmap v2751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:05 smithi114 ceph-mon[30820]: pgmap v2752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:05 smithi116 ceph-mon[35411]: pgmap v2752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:07 smithi114 ceph-mon[30820]: pgmap v2753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:07 smithi116 ceph-mon[35411]: pgmap v2753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:09 smithi114 ceph-mon[30820]: pgmap v2754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:09 smithi116 ceph-mon[35411]: pgmap v2754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:11 smithi114 ceph-mon[30820]: pgmap v2755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:11 smithi116 ceph-mon[35411]: pgmap v2755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:13 smithi114 ceph-mon[30820]: pgmap v2756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:13 smithi116 ceph-mon[35411]: pgmap v2756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:15 smithi114 ceph-mon[30820]: pgmap v2757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:15 smithi116 ceph-mon[35411]: pgmap v2757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:17 smithi114 ceph-mon[30820]: pgmap v2758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:17 smithi116 ceph-mon[35411]: pgmap v2758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:19 smithi114 ceph-mon[30820]: pgmap v2759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:19 smithi116 ceph-mon[35411]: pgmap v2759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:22 smithi114 ceph-mon[30820]: pgmap v2760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:22 smithi116 ceph-mon[35411]: pgmap v2760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:24 smithi114 ceph-mon[30820]: pgmap v2761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:24 smithi116 ceph-mon[35411]: pgmap v2761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:26 smithi114 ceph-mon[30820]: pgmap v2762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:26 smithi116 ceph-mon[35411]: pgmap v2762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:28 smithi114 ceph-mon[30820]: pgmap v2763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:28 smithi116 ceph-mon[35411]: pgmap v2763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:30 smithi114 ceph-mon[30820]: pgmap v2764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:30.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:30 smithi116 ceph-mon[35411]: pgmap v2764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:32 smithi114 ceph-mon[30820]: pgmap v2765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:32.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:32 smithi116 ceph-mon[35411]: pgmap v2765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:34 smithi114 ceph-mon[30820]: pgmap v2766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:34 smithi116 ceph-mon[35411]: pgmap v2766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:36 smithi114 ceph-mon[30820]: pgmap v2767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:36.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:36 smithi116 ceph-mon[35411]: pgmap v2767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:38 smithi114 ceph-mon[30820]: pgmap v2768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:38.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:38 smithi116 ceph-mon[35411]: pgmap v2768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:07:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:07:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:40 smithi114 ceph-mon[30820]: pgmap v2769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:40.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:40 smithi116 ceph-mon[35411]: pgmap v2769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-mon[30820]: pgmap v2770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:42.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:07:42.229+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:07:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:07:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:07:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:07:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:07:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:07:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:07:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:07:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:07:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:07:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:07:42.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:07:42.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:07:42.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:07:42.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:07:42.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:07:42.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:07:42.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:07:42.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:07:42.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:07:42.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:07:42.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:07:42.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:07:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:07:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:42 smithi116 ceph-mon[35411]: pgmap v2770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:07:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:43 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:07:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:07:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:07:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:07:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:07:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:07:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:07:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:07:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:07:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:07:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:07:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:07:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:07:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:43 smithi114 ceph-mon[30820]: pgmap v2771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:07:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:43 smithi114 ceph-mon[30820]: pgmap v2772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:07:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:07:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:43 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:07:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:07:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:07:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:07:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:07:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:07:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:07:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:07:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:07:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:07:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:07:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:07:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:07:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:43 smithi116 ceph-mon[35411]: pgmap v2771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:07:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:43 smithi116 ceph-mon[35411]: pgmap v2772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:07:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:07:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:44 smithi114 ceph-mon[30820]: pgmap v2773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:07:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:44 smithi116 ceph-mon[35411]: pgmap v2773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:07:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:46 smithi114 ceph-mon[30820]: pgmap v2774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:07:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:46 smithi116 ceph-mon[35411]: pgmap v2774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:07:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:48 smithi114 ceph-mon[30820]: pgmap v2775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:07:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:48 smithi116 ceph-mon[35411]: pgmap v2775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:07:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:50 smithi114 ceph-mon[30820]: pgmap v2776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:07:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:50 smithi116 ceph-mon[35411]: pgmap v2776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:07:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:52 smithi114 ceph-mon[30820]: pgmap v2777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:07:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:52 smithi116 ceph-mon[35411]: pgmap v2777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:07:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:54 smithi114 ceph-mon[30820]: pgmap v2778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:54 smithi116 ceph-mon[35411]: pgmap v2778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:07:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:56 smithi114 ceph-mon[30820]: pgmap v2779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:56 smithi116 ceph-mon[35411]: pgmap v2779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:07:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:07:58 smithi114 ceph-mon[30820]: pgmap v2780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:07:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:07:58 smithi116 ceph-mon[35411]: pgmap v2780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:00 smithi114 ceph-mon[30820]: pgmap v2781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:00 smithi116 ceph-mon[35411]: pgmap v2781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:02 smithi114 ceph-mon[30820]: pgmap v2782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:02 smithi116 ceph-mon[35411]: pgmap v2782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:04 smithi114 ceph-mon[30820]: pgmap v2783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:04 smithi116 ceph-mon[35411]: pgmap v2783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:06 smithi114 ceph-mon[30820]: pgmap v2784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:06 smithi116 ceph-mon[35411]: pgmap v2784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:08 smithi114 ceph-mon[30820]: pgmap v2785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:08 smithi116 ceph-mon[35411]: pgmap v2785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:10.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:10 smithi114 ceph-mon[30820]: pgmap v2786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:10 smithi116 ceph-mon[35411]: pgmap v2786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:12 smithi114 ceph-mon[30820]: pgmap v2787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:12.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:12 smithi116 ceph-mon[35411]: pgmap v2787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:14 smithi114 ceph-mon[30820]: pgmap v2788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:14.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:14 smithi116 ceph-mon[35411]: pgmap v2788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:16 smithi114 ceph-mon[30820]: pgmap v2789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:16 smithi116 ceph-mon[35411]: pgmap v2789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:18 smithi114 ceph-mon[30820]: pgmap v2790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:18.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:18 smithi116 ceph-mon[35411]: pgmap v2790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:20 smithi114 ceph-mon[30820]: pgmap v2791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:20 smithi116 ceph-mon[35411]: pgmap v2791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:22 smithi114 ceph-mon[30820]: pgmap v2792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:22 smithi116 ceph-mon[35411]: pgmap v2792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:24 smithi114 ceph-mon[30820]: pgmap v2793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:24 smithi116 ceph-mon[35411]: pgmap v2793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:26 smithi114 ceph-mon[30820]: pgmap v2794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:26 smithi116 ceph-mon[35411]: pgmap v2794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:28 smithi114 ceph-mon[30820]: pgmap v2795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:28 smithi116 ceph-mon[35411]: pgmap v2795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:30 smithi114 ceph-mon[30820]: pgmap v2796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:30 smithi116 ceph-mon[35411]: pgmap v2796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:32 smithi114 ceph-mon[30820]: pgmap v2797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:32 smithi116 ceph-mon[35411]: pgmap v2797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:34 smithi114 ceph-mon[30820]: pgmap v2798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:34 smithi116 ceph-mon[35411]: pgmap v2798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:36 smithi114 ceph-mon[30820]: pgmap v2799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:36 smithi116 ceph-mon[35411]: pgmap v2799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:38 smithi114 ceph-mon[30820]: pgmap v2800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:38 smithi116 ceph-mon[35411]: pgmap v2800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:40 smithi114 ceph-mon[30820]: pgmap v2801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:40 smithi116 ceph-mon[35411]: pgmap v2801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:42 smithi114 ceph-mon[30820]: pgmap v2802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:42 smithi116 ceph-mon[35411]: pgmap v2802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:08:43.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:08:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:08:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:44 smithi114 ceph-mon[30820]: pgmap v2803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:44 smithi116 ceph-mon[35411]: pgmap v2803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:08:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:08:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:08:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:08:45.589+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:08:45.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:08:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:08:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:08:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:08:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:08:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:08:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:08:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:08:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:08:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:08:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:08:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:08:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:08:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:08:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:08:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:08:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:08:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:08:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:08:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:46 smithi114 ceph-mon[30820]: pgmap v2804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:08:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:08:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:46 smithi116 ceph-mon[35411]: pgmap v2804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:08:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:48 smithi114 ceph-mon[30820]: pgmap v2805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:08:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:48 smithi116 ceph-mon[35411]: pgmap v2805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:08:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:50 smithi114 ceph-mon[30820]: pgmap v2806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:08:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:50 smithi116 ceph-mon[35411]: pgmap v2806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:08:52.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:52 smithi114 ceph-mon[30820]: pgmap v2807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:08:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:52 smithi116 ceph-mon[35411]: pgmap v2807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:08:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:54 smithi114 ceph-mon[30820]: pgmap v2808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:08:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:54 smithi116 ceph-mon[35411]: pgmap v2808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:08:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:56 smithi116 ceph-mon[35411]: pgmap v2809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:08:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:56 smithi114 ceph-mon[30820]: pgmap v2809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:08:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:08:58 smithi116 ceph-mon[35411]: pgmap v2810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:08:59.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:08:58 smithi114 ceph-mon[30820]: pgmap v2810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:00 smithi116 ceph-mon[35411]: pgmap v2811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:00.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:00 smithi114 ceph-mon[30820]: pgmap v2811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:02 smithi114 ceph-mon[30820]: pgmap v2812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:03.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:02 smithi116 ceph-mon[35411]: pgmap v2812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:04 smithi114 ceph-mon[30820]: pgmap v2813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:04 smithi116 ceph-mon[35411]: pgmap v2813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:06 smithi114 ceph-mon[30820]: pgmap v2814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:06 smithi116 ceph-mon[35411]: pgmap v2814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:08 smithi114 ceph-mon[30820]: pgmap v2815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:08 smithi116 ceph-mon[35411]: pgmap v2815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:10 smithi114 ceph-mon[30820]: pgmap v2816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:10 smithi116 ceph-mon[35411]: pgmap v2816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:12 smithi114 ceph-mon[30820]: pgmap v2817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:12 smithi116 ceph-mon[35411]: pgmap v2817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:14 smithi114 ceph-mon[30820]: pgmap v2818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:14 smithi116 ceph-mon[35411]: pgmap v2818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:16 smithi114 ceph-mon[30820]: pgmap v2819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:17.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:16 smithi116 ceph-mon[35411]: pgmap v2819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:18 smithi114 ceph-mon[30820]: pgmap v2820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:19.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:18 smithi116 ceph-mon[35411]: pgmap v2820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:20 smithi114 ceph-mon[30820]: pgmap v2821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:21.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:20 smithi116 ceph-mon[35411]: pgmap v2821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:22 smithi114 ceph-mon[30820]: pgmap v2822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:22 smithi116 ceph-mon[35411]: pgmap v2822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:24 smithi114 ceph-mon[30820]: pgmap v2823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:24 smithi116 ceph-mon[35411]: pgmap v2823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:26 smithi114 ceph-mon[30820]: pgmap v2824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:27.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:26 smithi116 ceph-mon[35411]: pgmap v2824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:28 smithi114 ceph-mon[30820]: pgmap v2825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:29.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:28 smithi116 ceph-mon[35411]: pgmap v2825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:30 smithi114 ceph-mon[30820]: pgmap v2826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:31.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:30 smithi116 ceph-mon[35411]: pgmap v2826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:32 smithi114 ceph-mon[30820]: pgmap v2827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:32 smithi116 ceph-mon[35411]: pgmap v2827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:34 smithi114 ceph-mon[30820]: pgmap v2828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:35.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:34 smithi116 ceph-mon[35411]: pgmap v2828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:36 smithi114 ceph-mon[30820]: pgmap v2829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:37.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:36 smithi116 ceph-mon[35411]: pgmap v2829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:38 smithi114 ceph-mon[30820]: pgmap v2830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:39.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:38 smithi116 ceph-mon[35411]: pgmap v2830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:40 smithi114 ceph-mon[30820]: pgmap v2831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:40 smithi116 ceph-mon[35411]: pgmap v2831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:09:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:42 smithi114 ceph-mon[30820]: pgmap v2832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:42 smithi116 ceph-mon[35411]: pgmap v2832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:09:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:44 smithi114 ceph-mon[30820]: pgmap v2833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:44 smithi116 ceph-mon[35411]: pgmap v2833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:09:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:09:47.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:46 smithi114 ceph-mon[30820]: pgmap v2834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:47.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:46 smithi116 ceph-mon[35411]: pgmap v2834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:09:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-mon[30820]: pgmap v2835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:49.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:09:48.949+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:09:49.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:09:49.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:09:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:09:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:09:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:09:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:09:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:09:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:09:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:09:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:09:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:09:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:48 smithi116 ceph-mon[35411]: pgmap v2835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:09:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:09:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:09:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:09:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:09:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:09:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:09:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:09:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:09:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:09:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:09:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:09:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:09:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:09:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:09:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:09:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:09:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:09:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:09:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:09:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:09:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:09:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:09:50.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:09:50.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:09:50.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:09:50.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:49 smithi114 ceph-mon[30820]: pgmap v2836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:09:50.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:49 smithi114 ceph-mon[30820]: pgmap v2837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:09:50.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:09:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:09:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:09:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:09:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:09:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:09:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:09:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:09:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:09:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:09:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:09:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:09:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:09:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:09:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:09:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:09:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:09:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:09:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:09:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:09:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:09:50.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:09:50.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:09:50.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:09:50.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:09:50.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:09:50.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:49 smithi116 ceph-mon[35411]: pgmap v2836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:09:50.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:49 smithi116 ceph-mon[35411]: pgmap v2837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:09:50.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:09:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:51 smithi114 ceph-mon[30820]: pgmap v2838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:09:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:51 smithi116 ceph-mon[35411]: pgmap v2838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:09:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:54 smithi114 ceph-mon[30820]: pgmap v2839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:09:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:54 smithi116 ceph-mon[35411]: pgmap v2839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:09:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:56 smithi114 ceph-mon[30820]: pgmap v2840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:09:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:56 smithi116 ceph-mon[35411]: pgmap v2840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:09:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:09:58 smithi114 ceph-mon[30820]: pgmap v2841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:09:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:09:58 smithi116 ceph-mon[35411]: pgmap v2841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:10:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:00 smithi114 ceph-mon[30820]: pgmap v2842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:10:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:10:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:00 smithi116 ceph-mon[35411]: pgmap v2842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:10:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:10:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:02 smithi114 ceph-mon[30820]: pgmap v2843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:02 smithi116 ceph-mon[35411]: pgmap v2843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:04 smithi114 ceph-mon[30820]: pgmap v2844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:04 smithi116 ceph-mon[35411]: pgmap v2844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:06.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:06 smithi114 ceph-mon[30820]: pgmap v2845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:06 smithi116 ceph-mon[35411]: pgmap v2845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:08 smithi114 ceph-mon[30820]: pgmap v2846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:08 smithi116 ceph-mon[35411]: pgmap v2846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:10 smithi114 ceph-mon[30820]: pgmap v2847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:10 smithi116 ceph-mon[35411]: pgmap v2847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:12 smithi114 ceph-mon[30820]: pgmap v2848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:12 smithi116 ceph-mon[35411]: pgmap v2848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:14 smithi114 ceph-mon[30820]: pgmap v2849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:14 smithi116 ceph-mon[35411]: pgmap v2849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:16 smithi114 ceph-mon[30820]: pgmap v2850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:16 smithi116 ceph-mon[35411]: pgmap v2850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:18 smithi114 ceph-mon[30820]: pgmap v2851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:18.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:18 smithi116 ceph-mon[35411]: pgmap v2851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:20 smithi114 ceph-mon[30820]: pgmap v2852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:20 smithi116 ceph-mon[35411]: pgmap v2852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:22 smithi114 ceph-mon[30820]: pgmap v2853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:22 smithi116 ceph-mon[35411]: pgmap v2853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:24 smithi114 ceph-mon[30820]: pgmap v2854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:24.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:24 smithi116 ceph-mon[35411]: pgmap v2854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:26 smithi114 ceph-mon[30820]: pgmap v2855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:26.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:26 smithi116 ceph-mon[35411]: pgmap v2855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:28 smithi114 ceph-mon[30820]: pgmap v2856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:28 smithi116 ceph-mon[35411]: pgmap v2856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:30 smithi114 ceph-mon[30820]: pgmap v2857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:30 smithi116 ceph-mon[35411]: pgmap v2857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:32 smithi114 ceph-mon[30820]: pgmap v2858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:32.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:32 smithi116 ceph-mon[35411]: pgmap v2858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:34.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:34 smithi114 ceph-mon[30820]: pgmap v2859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:34.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:34 smithi116 ceph-mon[35411]: pgmap v2859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:36 smithi114 ceph-mon[30820]: pgmap v2860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:36 smithi116 ceph-mon[35411]: pgmap v2860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:38 smithi114 ceph-mon[30820]: pgmap v2861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:38.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:38 smithi116 ceph-mon[35411]: pgmap v2861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:39.282 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() TimeoutError 2024-09-19T08:10:39.285 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() TimeoutError 2024-09-19T08:10:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:40 smithi114 ceph-mon[30820]: pgmap v2862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:40 smithi116 ceph-mon[35411]: pgmap v2862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:42 smithi114 ceph-mon[30820]: pgmap v2863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:42 smithi116 ceph-mon[35411]: pgmap v2863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:44 smithi114 ceph-mon[30820]: pgmap v2864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:44 smithi116 ceph-mon[35411]: pgmap v2864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:46 smithi114 ceph-mon[30820]: pgmap v2865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:46.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:46 smithi116 ceph-mon[35411]: pgmap v2865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:10:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:48 smithi114 ceph-mon[30820]: pgmap v2866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:48.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:48 smithi116 ceph-mon[35411]: pgmap v2866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:10:49.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:10:49.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:10:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:50 smithi114 ceph-mon[30820]: pgmap v2867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:50.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:50 smithi116 ceph-mon[35411]: pgmap v2867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:10:52.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-mon[30820]: pgmap v2868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:52.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:10:52.309+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:10:52.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:10:52.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:10:52.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:10:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:10:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:10:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:10:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:10:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:10:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:10:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:10:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:10:52.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:52 smithi116 ceph-mon[35411]: pgmap v2868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:10:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:10:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:10:53.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:10:53.441 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:10:53.441 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:10:53.442 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:10:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:10:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:10:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:10:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:10:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:10:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:10:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:10:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:10:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:10:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:10:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:54 smithi114 ceph-mon[30820]: pgmap v2869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:10:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:54 smithi114 ceph-mon[30820]: pgmap v2870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:10:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:10:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:10:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:10:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:10:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:10:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:10:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:10:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:10:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:10:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:10:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:54 smithi116 ceph-mon[35411]: pgmap v2869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:10:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:54 smithi116 ceph-mon[35411]: pgmap v2870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:10:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:56 smithi114 ceph-mon[30820]: pgmap v2871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:10:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:56 smithi116 ceph-mon[35411]: pgmap v2871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:10:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:10:58 smithi114 ceph-mon[30820]: pgmap v2872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:10:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:10:58 smithi116 ceph-mon[35411]: pgmap v2872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:11:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:00 smithi114 ceph-mon[30820]: pgmap v2873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:11:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:00 smithi116 ceph-mon[35411]: pgmap v2873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:11:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:02 smithi114 ceph-mon[30820]: pgmap v2874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:11:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:02 smithi116 ceph-mon[35411]: pgmap v2874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:11:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:04 smithi114 ceph-mon[30820]: pgmap v2875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:11:04.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:04 smithi116 ceph-mon[35411]: pgmap v2875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:11:06.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:06 smithi114 ceph-mon[30820]: pgmap v2876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:06 smithi116 ceph-mon[35411]: pgmap v2876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:08 smithi114 ceph-mon[30820]: pgmap v2877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:08 smithi116 ceph-mon[35411]: pgmap v2877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:10 smithi114 ceph-mon[30820]: pgmap v2878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:10 smithi116 ceph-mon[35411]: pgmap v2878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:12 smithi114 ceph-mon[30820]: pgmap v2879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:12 smithi116 ceph-mon[35411]: pgmap v2879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:14 smithi116 ceph-mon[35411]: pgmap v2880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:14 smithi114 ceph-mon[30820]: pgmap v2880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:16 smithi116 ceph-mon[35411]: pgmap v2881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:16 smithi114 ceph-mon[30820]: pgmap v2881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:18 smithi116 ceph-mon[35411]: pgmap v2882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:18 smithi114 ceph-mon[30820]: pgmap v2882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:20 smithi114 ceph-mon[30820]: pgmap v2883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:20 smithi116 ceph-mon[35411]: pgmap v2883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:22 smithi114 ceph-mon[30820]: pgmap v2884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:22 smithi116 ceph-mon[35411]: pgmap v2884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:24 smithi114 ceph-mon[30820]: pgmap v2885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:24 smithi116 ceph-mon[35411]: pgmap v2885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:26 smithi114 ceph-mon[30820]: pgmap v2886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:26 smithi116 ceph-mon[35411]: pgmap v2886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:28 smithi114 ceph-mon[30820]: pgmap v2887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:28 smithi116 ceph-mon[35411]: pgmap v2887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:30.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:30 smithi114 ceph-mon[30820]: pgmap v2888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:30.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:30 smithi116 ceph-mon[35411]: pgmap v2888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:32 smithi114 ceph-mon[30820]: pgmap v2889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:32 smithi116 ceph-mon[35411]: pgmap v2889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:34 smithi114 ceph-mon[30820]: pgmap v2890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:34 smithi116 ceph-mon[35411]: pgmap v2890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:36 smithi114 ceph-mon[30820]: pgmap v2891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:36 smithi116 ceph-mon[35411]: pgmap v2891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:38 smithi114 ceph-mon[30820]: pgmap v2892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:38 smithi116 ceph-mon[35411]: pgmap v2892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:40 smithi114 ceph-mon[30820]: pgmap v2893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:40 smithi116 ceph-mon[35411]: pgmap v2893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:42 smithi114 ceph-mon[30820]: pgmap v2894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:42 smithi116 ceph-mon[35411]: pgmap v2894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:44 smithi114 ceph-mon[30820]: pgmap v2895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:44 smithi116 ceph-mon[35411]: pgmap v2895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:46 smithi114 ceph-mon[30820]: pgmap v2896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:46 smithi116 ceph-mon[35411]: pgmap v2896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:48.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:48 smithi114 ceph-mon[30820]: pgmap v2897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:48 smithi116 ceph-mon[35411]: pgmap v2897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:11:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:50 smithi114 ceph-mon[30820]: pgmap v2898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:50 smithi116 ceph-mon[35411]: pgmap v2898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:11:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:52 smithi114 ceph-mon[30820]: pgmap v2899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:52 smithi116 ceph-mon[35411]: pgmap v2899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:11:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:54 smithi114 ceph-mon[30820]: pgmap v2900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:54 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:11:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:54 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:11:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:54 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:11:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:54 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:11:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:54 smithi116 ceph-mon[35411]: pgmap v2900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:11:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:54 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:11:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:54 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:11:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:54 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:11:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:54 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:11:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:11:55.669+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:11:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:11:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:11:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:11:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:56 smithi114 ceph-mon[30820]: pgmap v2901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:11:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:11:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:11:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:56 smithi116 ceph-mon[35411]: pgmap v2901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:11:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:11:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:11:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:57 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:11:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:11:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:11:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:11:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:11:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:57 smithi114 ceph-mon[30820]: pgmap v2902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:11:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:57 smithi114 ceph-mon[30820]: pgmap v2903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:57 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:11:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:11:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:11:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:11:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:11:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:11:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:57 smithi116 ceph-mon[35411]: pgmap v2902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:11:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:57 smithi116 ceph-mon[35411]: pgmap v2903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:11:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:11:59 smithi114 ceph-mon[30820]: pgmap v2904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:11:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:11:59 smithi116 ceph-mon[35411]: pgmap v2904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:12:01.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:01 smithi114 ceph-mon[30820]: pgmap v2905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:12:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:01 smithi116 ceph-mon[35411]: pgmap v2905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:12:03.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:03 smithi114 ceph-mon[30820]: pgmap v2906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:12:03.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:03 smithi116 ceph-mon[35411]: pgmap v2906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:12:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:05 smithi114 ceph-mon[30820]: pgmap v2907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:12:05.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:05 smithi116 ceph-mon[35411]: pgmap v2907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:12:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:07 smithi114 ceph-mon[30820]: pgmap v2908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:12:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:07 smithi116 ceph-mon[35411]: pgmap v2908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:12:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:09 smithi114 ceph-mon[30820]: pgmap v2909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:09 smithi116 ceph-mon[35411]: pgmap v2909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:11 smithi114 ceph-mon[30820]: pgmap v2910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:11.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:11 smithi116 ceph-mon[35411]: pgmap v2910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:13 smithi114 ceph-mon[30820]: pgmap v2911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:13 smithi116 ceph-mon[35411]: pgmap v2911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:15 smithi114 ceph-mon[30820]: pgmap v2912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:15 smithi116 ceph-mon[35411]: pgmap v2912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:17.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:17 smithi114 ceph-mon[30820]: pgmap v2913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:17.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:17 smithi116 ceph-mon[35411]: pgmap v2913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:19 smithi114 ceph-mon[30820]: pgmap v2914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:19 smithi116 ceph-mon[35411]: pgmap v2914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:21 smithi114 ceph-mon[30820]: pgmap v2915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:21 smithi116 ceph-mon[35411]: pgmap v2915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:23 smithi114 ceph-mon[30820]: pgmap v2916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:23 smithi116 ceph-mon[35411]: pgmap v2916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:25.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:25 smithi114 ceph-mon[30820]: pgmap v2917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:25 smithi116 ceph-mon[35411]: pgmap v2917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:27 smithi114 ceph-mon[30820]: pgmap v2918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:27 smithi116 ceph-mon[35411]: pgmap v2918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:28 smithi114 ceph-mon[30820]: pgmap v2919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:28 smithi116 ceph-mon[35411]: pgmap v2919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:30 smithi114 ceph-mon[30820]: pgmap v2920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:31.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:30 smithi116 ceph-mon[35411]: pgmap v2920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:32 smithi114 ceph-mon[30820]: pgmap v2921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:32 smithi116 ceph-mon[35411]: pgmap v2921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:35.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:34 smithi114 ceph-mon[30820]: pgmap v2922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:34 smithi116 ceph-mon[35411]: pgmap v2922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:36 smithi114 ceph-mon[30820]: pgmap v2923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:36 smithi116 ceph-mon[35411]: pgmap v2923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:38 smithi114 ceph-mon[30820]: pgmap v2924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:39.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:38 smithi116 ceph-mon[35411]: pgmap v2924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:41.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:40 smithi114 ceph-mon[30820]: pgmap v2925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:40 smithi116 ceph-mon[35411]: pgmap v2925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:42 smithi114 ceph-mon[30820]: pgmap v2926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:42 smithi116 ceph-mon[35411]: pgmap v2926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:44 smithi114 ceph-mon[30820]: pgmap v2927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:44 smithi116 ceph-mon[35411]: pgmap v2927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:46 smithi114 ceph-mon[30820]: pgmap v2928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:46 smithi116 ceph-mon[35411]: pgmap v2928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:48 smithi114 ceph-mon[30820]: pgmap v2929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:48 smithi116 ceph-mon[35411]: pgmap v2929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:50 smithi114 ceph-mon[30820]: pgmap v2930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:50 smithi116 ceph-mon[35411]: pgmap v2930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:12:53.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:52 smithi114 ceph-mon[30820]: pgmap v2931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:52 smithi116 ceph-mon[35411]: pgmap v2931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:12:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:54 smithi114 ceph-mon[30820]: pgmap v2932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:54 smithi116 ceph-mon[35411]: pgmap v2932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:57.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:56 smithi114 ceph-mon[30820]: pgmap v2933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:12:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:56 smithi116 ceph-mon[35411]: pgmap v2933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:12:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:12:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:12:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:12:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:12:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:12:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:58 smithi114 ceph-mon[30820]: pgmap v2934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:12:59.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:12:59.028+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:12:59.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:12:59.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:12:59.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:12:59.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:12:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:12:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:12:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:12:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:12:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:12:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:12:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:12:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:12:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:58 smithi116 ceph-mon[35411]: pgmap v2934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:13:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:13:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:13:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:13:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:13:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:13:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:13:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:13:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:13:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:13:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:13:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:13:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:13:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:13:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:13:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:13:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:13:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:13:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-mon[30820]: pgmap v2935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:13:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-mon[30820]: pgmap v2936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:13:00.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:12:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:59 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:13:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:13:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:13:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:13:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:13:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:59 smithi116 ceph-mon[35411]: pgmap v2935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:13:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:59 smithi116 ceph-mon[35411]: pgmap v2936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:13:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:12:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:13:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:01 smithi114 ceph-mon[30820]: pgmap v2937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:13:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:01 smithi116 ceph-mon[35411]: pgmap v2937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:13:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:04 smithi114 ceph-mon[30820]: pgmap v2938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:13:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:04 smithi116 ceph-mon[35411]: pgmap v2938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:13:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:06 smithi114 ceph-mon[30820]: pgmap v2939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:13:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:06 smithi116 ceph-mon[35411]: pgmap v2939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:13:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:08 smithi114 ceph-mon[30820]: pgmap v2940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:13:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:08 smithi116 ceph-mon[35411]: pgmap v2940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:13:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:10 smithi114 ceph-mon[30820]: pgmap v2941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:13:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:10 smithi116 ceph-mon[35411]: pgmap v2941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:13:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:12 smithi114 ceph-mon[30820]: pgmap v2942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:12 smithi116 ceph-mon[35411]: pgmap v2942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:14.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:14 smithi114 ceph-mon[30820]: pgmap v2943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:14 smithi116 ceph-mon[35411]: pgmap v2943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:16.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:16 smithi114 ceph-mon[30820]: pgmap v2944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:13:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:16 smithi116 ceph-mon[35411]: pgmap v2944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:13:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:18 smithi114 ceph-mon[30820]: pgmap v2945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:18 smithi116 ceph-mon[35411]: pgmap v2945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:20 smithi114 ceph-mon[30820]: pgmap v2946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:20.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:20 smithi116 ceph-mon[35411]: pgmap v2946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:22.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:22 smithi114 ceph-mon[30820]: pgmap v2947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:22.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:22 smithi116 ceph-mon[35411]: pgmap v2947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:24 smithi114 ceph-mon[30820]: pgmap v2948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:13:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:24 smithi116 ceph-mon[35411]: pgmap v2948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:13:26.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:26 smithi114 ceph-mon[30820]: pgmap v2949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:13:26.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:26 smithi116 ceph-mon[35411]: pgmap v2949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:13:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:28 smithi114 ceph-mon[30820]: pgmap v2950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:28 smithi116 ceph-mon[35411]: pgmap v2950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:30 smithi114 ceph-mon[30820]: pgmap v2951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:13:30.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:30 smithi116 ceph-mon[35411]: pgmap v2951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:13:32.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:32 smithi114 ceph-mon[30820]: pgmap v2952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:32.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:32 smithi116 ceph-mon[35411]: pgmap v2952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:34 smithi114 ceph-mon[30820]: pgmap v2953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:34 smithi116 ceph-mon[35411]: pgmap v2953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:36 smithi114 ceph-mon[30820]: pgmap v2954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:13:36.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:36 smithi116 ceph-mon[35411]: pgmap v2954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:13:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:38 smithi114 ceph-mon[30820]: pgmap v2955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:38.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:38 smithi116 ceph-mon[35411]: pgmap v2955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:40 smithi114 ceph-mon[30820]: pgmap v2956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:40 smithi116 ceph-mon[35411]: pgmap v2956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:41 smithi114 ceph-mon[30820]: pgmap v2957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:41 smithi116 ceph-mon[35411]: pgmap v2957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:44.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:44 smithi114 ceph-mon[30820]: pgmap v2958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:44.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:44 smithi116 ceph-mon[35411]: pgmap v2958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:46 smithi114 ceph-mon[30820]: pgmap v2959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:13:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:46 smithi116 ceph-mon[35411]: pgmap v2959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:13:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:48 smithi114 ceph-mon[30820]: pgmap v2960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:48.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:48 smithi116 ceph-mon[35411]: pgmap v2960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:50 smithi114 ceph-mon[30820]: pgmap v2961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:50 smithi116 ceph-mon[35411]: pgmap v2961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:52.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:52 smithi114 ceph-mon[30820]: pgmap v2962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:52 smithi116 ceph-mon[35411]: pgmap v2962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:13:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:54 smithi114 ceph-mon[30820]: pgmap v2963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:54 smithi116 ceph-mon[35411]: pgmap v2963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:13:56.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:56 smithi114 ceph-mon[30820]: pgmap v2964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:13:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:56 smithi116 ceph-mon[35411]: pgmap v2964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:13:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:13:58 smithi114 ceph-mon[30820]: pgmap v2965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:13:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:13:58 smithi116 ceph-mon[35411]: pgmap v2965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:00 smithi114 ceph-mon[30820]: pgmap v2966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:14:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:14:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:14:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:14:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:00 smithi116 ceph-mon[35411]: pgmap v2966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:14:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:14:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:14:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:14:02.389 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-mon[30820]: pgmap v2967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:14:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:02 smithi116 ceph-mon[35411]: pgmap v2967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:14:02.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:14:02.388+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:14:02.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:14:02.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:14:02.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:14:02.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:14:02.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:14:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:14:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:14:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:14:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:14:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:14:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:14:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:14:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:14:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:14:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:14:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:14:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:14:02.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:14:02.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:14:02.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:14:02.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:14:02.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:14:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:14:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:14:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:14:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:14:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:14:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:14:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:14:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:14:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:14:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:14:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:04 smithi114 ceph-mon[30820]: pgmap v2968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:14:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:04 smithi114 ceph-mon[30820]: pgmap v2969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:14:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:04 smithi116 ceph-mon[35411]: pgmap v2968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:04 smithi116 ceph-mon[35411]: pgmap v2969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:06.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:06 smithi114 ceph-mon[30820]: pgmap v2970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:06 smithi116 ceph-mon[35411]: pgmap v2970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:08.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:08 smithi114 ceph-mon[30820]: pgmap v2971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:08 smithi116 ceph-mon[35411]: pgmap v2971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:10 smithi114 ceph-mon[30820]: pgmap v2972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:10 smithi116 ceph-mon[35411]: pgmap v2972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:12 smithi114 ceph-mon[30820]: pgmap v2973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:14:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:12 smithi116 ceph-mon[35411]: pgmap v2973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:14:14.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:14 smithi114 ceph-mon[30820]: pgmap v2974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:14:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:14 smithi116 ceph-mon[35411]: pgmap v2974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:14:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:16 smithi116 ceph-mon[35411]: pgmap v2975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:14:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:16 smithi114 ceph-mon[30820]: pgmap v2975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:14:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:18 smithi116 ceph-mon[35411]: pgmap v2976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:18 smithi114 ceph-mon[30820]: pgmap v2976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:20 smithi116 ceph-mon[35411]: pgmap v2977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:14:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:20 smithi114 ceph-mon[30820]: pgmap v2977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:14:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:22 smithi114 ceph-mon[30820]: pgmap v2978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:14:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:22 smithi116 ceph-mon[35411]: pgmap v2978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:14:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:24 smithi114 ceph-mon[30820]: pgmap v2979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:14:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:24 smithi116 ceph-mon[35411]: pgmap v2979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:14:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:26 smithi114 ceph-mon[30820]: pgmap v2980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:14:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:26 smithi116 ceph-mon[35411]: pgmap v2980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:14:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:28 smithi114 ceph-mon[30820]: pgmap v2981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:28 smithi116 ceph-mon[35411]: pgmap v2981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:30 smithi114 ceph-mon[30820]: pgmap v2982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:14:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:30 smithi116 ceph-mon[35411]: pgmap v2982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:14:32.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:32 smithi114 ceph-mon[30820]: pgmap v2983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:32.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:32 smithi116 ceph-mon[35411]: pgmap v2983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:14:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:34 smithi114 ceph-mon[30820]: pgmap v2984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T08:14:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:34 smithi116 ceph-mon[35411]: pgmap v2984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T08:14:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:36 smithi114 ceph-mon[30820]: pgmap v2985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:14:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:36 smithi116 ceph-mon[35411]: pgmap v2985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:14:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:38 smithi114 ceph-mon[30820]: pgmap v2986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:38 smithi116 ceph-mon[35411]: pgmap v2986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:40 smithi114 ceph-mon[30820]: pgmap v2987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:14:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:40 smithi116 ceph-mon[35411]: pgmap v2987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:14:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:42 smithi114 ceph-mon[30820]: pgmap v2988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:14:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:42 smithi116 ceph-mon[35411]: pgmap v2988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:14:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:44 smithi114 ceph-mon[30820]: pgmap v2989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:14:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:44 smithi116 ceph-mon[35411]: pgmap v2989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:14:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:46 smithi114 ceph-mon[30820]: pgmap v2990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:14:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:46 smithi116 ceph-mon[35411]: pgmap v2990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:14:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:48 smithi114 ceph-mon[30820]: pgmap v2991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:48 smithi116 ceph-mon[35411]: pgmap v2991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:50 smithi114 ceph-mon[30820]: pgmap v2992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:14:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:50 smithi116 ceph-mon[35411]: pgmap v2992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:14:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:52 smithi114 ceph-mon[30820]: pgmap v2993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:14:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:52 smithi116 ceph-mon[35411]: pgmap v2993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:14:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:54 smithi114 ceph-mon[30820]: pgmap v2994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:14:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:54 smithi116 ceph-mon[35411]: pgmap v2994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:14:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:56 smithi114 ceph-mon[30820]: pgmap v2995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:14:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:56 smithi116 ceph-mon[35411]: pgmap v2995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:14:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:14:58 smithi114 ceph-mon[30820]: pgmap v2996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:14:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:14:58 smithi116 ceph-mon[35411]: pgmap v2996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:00 smithi114 ceph-mon[30820]: pgmap v2997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:00 smithi116 ceph-mon[35411]: pgmap v2997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:02.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:02 smithi114 ceph-mon[30820]: pgmap v2998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:15:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:02 smithi116 ceph-mon[35411]: pgmap v2998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:15:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:15:03.610 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:15:04.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:04 smithi114 ceph-mon[30820]: pgmap v2999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:15:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:04 smithi116 ceph-mon[35411]: pgmap v2999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:15:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:05 smithi116 ceph-mon[35411]: pgmap v3000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-mon[30820]: pgmap v3000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:06.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:15:05.748+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:15:06.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:15:06.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:15:06.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:15:06.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:15:06.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:15:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:15:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:15:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:15:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:15:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:15:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:06 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:15:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:15:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:15:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:15:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:15:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:15:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:06 smithi116 ceph-mon[35411]: pgmap v3001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:06 smithi116 ceph-mon[35411]: pgmap v3002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:15:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:06 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:15:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:15:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:15:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:15:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:15:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:06 smithi114 ceph-mon[30820]: pgmap v3001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:15:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:06 smithi114 ceph-mon[30820]: pgmap v3002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:15:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:15:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:08 smithi114 ceph-mon[30820]: pgmap v3003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:15:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:08 smithi116 ceph-mon[35411]: pgmap v3003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:15:11.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:10 smithi114 ceph-mon[30820]: pgmap v3004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:15:11.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:10 smithi116 ceph-mon[35411]: pgmap v3004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:15:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:12 smithi114 ceph-mon[30820]: pgmap v3005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:15:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:12 smithi116 ceph-mon[35411]: pgmap v3005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:15:15.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:14 smithi114 ceph-mon[30820]: pgmap v3006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:15:15.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:14 smithi116 ceph-mon[35411]: pgmap v3006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:15:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:16 smithi114 ceph-mon[30820]: pgmap v3007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:15:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:16 smithi116 ceph-mon[35411]: pgmap v3007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:15:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:18 smithi114 ceph-mon[30820]: pgmap v3008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:18 smithi116 ceph-mon[35411]: pgmap v3008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:20 smithi114 ceph-mon[30820]: pgmap v3009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:20 smithi116 ceph-mon[35411]: pgmap v3009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:22 smithi114 ceph-mon[30820]: pgmap v3010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:23.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:22 smithi116 ceph-mon[35411]: pgmap v3010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:24 smithi114 ceph-mon[30820]: pgmap v3011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:24 smithi116 ceph-mon[35411]: pgmap v3011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:26 smithi114 ceph-mon[30820]: pgmap v3012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:15:27.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:26 smithi116 ceph-mon[35411]: pgmap v3012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:15:29.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:28 smithi114 ceph-mon[30820]: pgmap v3013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:28 smithi116 ceph-mon[35411]: pgmap v3013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:30 smithi114 ceph-mon[30820]: pgmap v3014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:30 smithi116 ceph-mon[35411]: pgmap v3014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:33.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:32 smithi114 ceph-mon[30820]: pgmap v3015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:32 smithi116 ceph-mon[35411]: pgmap v3015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:34.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:34 smithi114 ceph-mon[30820]: pgmap v3016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:35.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:34 smithi116 ceph-mon[35411]: pgmap v3016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:37.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:36 smithi114 ceph-mon[30820]: pgmap v3017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:15:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:36 smithi116 ceph-mon[35411]: pgmap v3017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:15:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:38 smithi114 ceph-mon[30820]: pgmap v3018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:38 smithi116 ceph-mon[35411]: pgmap v3018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:40 smithi114 ceph-mon[30820]: pgmap v3019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:41.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:40 smithi116 ceph-mon[35411]: pgmap v3019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:42 smithi114 ceph-mon[30820]: pgmap v3020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:42 smithi116 ceph-mon[35411]: pgmap v3020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:45.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:44 smithi114 ceph-mon[30820]: pgmap v3021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:44 smithi116 ceph-mon[35411]: pgmap v3021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:46 smithi114 ceph-mon[30820]: pgmap v3022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:15:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:46 smithi116 ceph-mon[35411]: pgmap v3022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:15:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:48 smithi114 ceph-mon[30820]: pgmap v3023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:48 smithi116 ceph-mon[35411]: pgmap v3023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:51.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:50 smithi114 ceph-mon[30820]: pgmap v3024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:51.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:50 smithi116 ceph-mon[35411]: pgmap v3024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:15:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:52 smithi114 ceph-mon[30820]: pgmap v3025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:52 smithi116 ceph-mon[35411]: pgmap v3025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:15:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:54 smithi114 ceph-mon[30820]: pgmap v3026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:55.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:54 smithi116 ceph-mon[35411]: pgmap v3026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:57.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:56 smithi114 ceph-mon[30820]: pgmap v3027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:15:57.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:56 smithi116 ceph-mon[35411]: pgmap v3027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:15:59.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:15:58 smithi114 ceph-mon[30820]: pgmap v3028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:15:59.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:15:58 smithi116 ceph-mon[35411]: pgmap v3028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:01.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:00 smithi114 ceph-mon[30820]: pgmap v3029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:00 smithi116 ceph-mon[35411]: pgmap v3029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:02 smithi114 ceph-mon[30820]: pgmap v3030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:03.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:02 smithi116 ceph-mon[35411]: pgmap v3030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:05.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:04 smithi114 ceph-mon[30820]: pgmap v3031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:04 smithi116 ceph-mon[35411]: pgmap v3031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:16:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:16:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:07 smithi114 ceph-mon[30820]: pgmap v3032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:07 smithi116 ceph-mon[35411]: pgmap v3032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:09.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-mon[30820]: pgmap v3033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:16:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:16:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:09.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:16:09.044+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:16:09.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:16:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:16:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:09 smithi116 ceph-mon[35411]: pgmap v3033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:16:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:16:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:16:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:10 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:16:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:16:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:16:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:16:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:16:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:16:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:16:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:16:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:16:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:16:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:16:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:10 smithi114 ceph-mon[30820]: pgmap v3034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:16:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:10 smithi114 ceph-mon[30820]: pgmap v3035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:16:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:10 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:16:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:16:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:16:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:16:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:16:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:16:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:16:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:16:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:16:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:10 smithi116 ceph-mon[35411]: pgmap v3034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:16:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:10 smithi116 ceph-mon[35411]: pgmap v3035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:16:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:12 smithi114 ceph-mon[30820]: pgmap v3036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:16:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:12 smithi116 ceph-mon[35411]: pgmap v3036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:16:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:14 smithi114 ceph-mon[30820]: pgmap v3037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:16:14.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:14 smithi116 ceph-mon[35411]: pgmap v3037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:16:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:16 smithi114 ceph-mon[30820]: pgmap v3038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:16:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:16 smithi116 ceph-mon[35411]: pgmap v3038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:16:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:18 smithi114 ceph-mon[30820]: pgmap v3039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:16:18.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:18 smithi116 ceph-mon[35411]: pgmap v3039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:16:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:20 smithi114 ceph-mon[30820]: pgmap v3040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:16:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:20 smithi116 ceph-mon[35411]: pgmap v3040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:16:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:22 smithi114 ceph-mon[30820]: pgmap v3041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:22 smithi116 ceph-mon[35411]: pgmap v3041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:24 smithi114 ceph-mon[30820]: pgmap v3042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:24 smithi116 ceph-mon[35411]: pgmap v3042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:26 smithi114 ceph-mon[30820]: pgmap v3043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:26 smithi116 ceph-mon[35411]: pgmap v3043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:28 smithi114 ceph-mon[30820]: pgmap v3044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:28 smithi116 ceph-mon[35411]: pgmap v3044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:30 smithi114 ceph-mon[30820]: pgmap v3045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:30 smithi116 ceph-mon[35411]: pgmap v3045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:32.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:32 smithi116 ceph-mon[35411]: pgmap v3046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:32 smithi114 ceph-mon[30820]: pgmap v3046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:34 smithi114 ceph-mon[30820]: pgmap v3047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:34 smithi116 ceph-mon[35411]: pgmap v3047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:36 smithi114 ceph-mon[30820]: pgmap v3048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:36 smithi116 ceph-mon[35411]: pgmap v3048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:38 smithi114 ceph-mon[30820]: pgmap v3049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:38 smithi116 ceph-mon[35411]: pgmap v3049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:40 smithi114 ceph-mon[30820]: pgmap v3050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:40 smithi116 ceph-mon[35411]: pgmap v3050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:42 smithi114 ceph-mon[30820]: pgmap v3051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:42 smithi116 ceph-mon[35411]: pgmap v3051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:44 smithi114 ceph-mon[30820]: pgmap v3052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:44 smithi116 ceph-mon[35411]: pgmap v3052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:46 smithi114 ceph-mon[30820]: pgmap v3053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:46 smithi116 ceph-mon[35411]: pgmap v3053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:48 smithi114 ceph-mon[30820]: pgmap v3054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:48 smithi116 ceph-mon[35411]: pgmap v3054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:50 smithi114 ceph-mon[30820]: pgmap v3055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:50 smithi116 ceph-mon[35411]: pgmap v3055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:52.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:52 smithi114 ceph-mon[30820]: pgmap v3056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:52 smithi116 ceph-mon[35411]: pgmap v3056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:16:54.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:54 smithi114 ceph-mon[30820]: pgmap v3057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:54 smithi116 ceph-mon[35411]: pgmap v3057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:16:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:56 smithi114 ceph-mon[30820]: pgmap v3058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:56 smithi116 ceph-mon[35411]: pgmap v3058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:16:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:16:58 smithi114 ceph-mon[30820]: pgmap v3059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:16:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:16:58 smithi116 ceph-mon[35411]: pgmap v3059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:00 smithi114 ceph-mon[30820]: pgmap v3060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:17:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:00 smithi116 ceph-mon[35411]: pgmap v3060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:17:01.539 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:01 smithi114 ceph-mon[30820]: pgmap v3061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:01 smithi116 ceph-mon[35411]: pgmap v3061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:04.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:04 smithi114 ceph-mon[30820]: pgmap v3062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:17:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:04 smithi116 ceph-mon[35411]: pgmap v3062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:17:06.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:06 smithi114 ceph-mon[30820]: pgmap v3063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:17:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:06 smithi116 ceph-mon[35411]: pgmap v3063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:17:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:08 smithi114 ceph-mon[30820]: pgmap v3064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:08.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:08 smithi116 ceph-mon[35411]: pgmap v3064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:10 smithi114 ceph-mon[30820]: pgmap v3065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:17:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:10 smithi116 ceph-mon[35411]: pgmap v3065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:17:12.406 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-mon[30820]: pgmap v3066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:12 smithi116 ceph-mon[35411]: pgmap v3066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:17:12.404+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:17:12.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:17:12.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:17:12.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:17:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:17:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:17:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:17:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:17:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:17:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:17:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:17:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:17:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:17:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:17:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:17:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:17:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:17:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:17:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:17:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:17:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:17:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:17:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:14 smithi114 ceph-mon[30820]: pgmap v3067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:17:14.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:14 smithi114 ceph-mon[30820]: pgmap v3068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:17:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:17:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:17:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:17:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:17:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:17:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:17:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:17:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:17:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:17:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:17:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:17:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:14 smithi116 ceph-mon[35411]: pgmap v3067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:17:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:14 smithi116 ceph-mon[35411]: pgmap v3068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:17:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:16 smithi114 ceph-mon[30820]: pgmap v3069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:17:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:16 smithi116 ceph-mon[35411]: pgmap v3069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:17:18.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:18 smithi114 ceph-mon[30820]: pgmap v3070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:17:18.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:18 smithi116 ceph-mon[35411]: pgmap v3070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:17:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:20 smithi114 ceph-mon[30820]: pgmap v3071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:17:20.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:20 smithi116 ceph-mon[35411]: pgmap v3071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:17:22.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:22 smithi114 ceph-mon[30820]: pgmap v3072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:17:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:22 smithi116 ceph-mon[35411]: pgmap v3072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:17:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:24 smithi116 ceph-mon[35411]: pgmap v3073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:17:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:24 smithi114 ceph-mon[30820]: pgmap v3073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:17:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:26 smithi116 ceph-mon[35411]: pgmap v3074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:17:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:26 smithi114 ceph-mon[30820]: pgmap v3074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:17:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:28 smithi116 ceph-mon[35411]: pgmap v3075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:28 smithi114 ceph-mon[30820]: pgmap v3075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:30 smithi114 ceph-mon[30820]: pgmap v3076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:30 smithi116 ceph-mon[35411]: pgmap v3076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:32 smithi114 ceph-mon[30820]: pgmap v3077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:32 smithi116 ceph-mon[35411]: pgmap v3077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:34.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:34 smithi114 ceph-mon[30820]: pgmap v3078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:17:34.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:34 smithi116 ceph-mon[35411]: pgmap v3078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:17:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:36 smithi114 ceph-mon[30820]: pgmap v3079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:17:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:36 smithi116 ceph-mon[35411]: pgmap v3079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:17:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:38 smithi114 ceph-mon[30820]: pgmap v3080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:38 smithi116 ceph-mon[35411]: pgmap v3080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:40 smithi114 ceph-mon[30820]: pgmap v3081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:40 smithi116 ceph-mon[35411]: pgmap v3081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:42 smithi114 ceph-mon[30820]: pgmap v3082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:42 smithi116 ceph-mon[35411]: pgmap v3082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:44 smithi114 ceph-mon[30820]: pgmap v3083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:44 smithi116 ceph-mon[35411]: pgmap v3083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:46 smithi114 ceph-mon[30820]: pgmap v3084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:17:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:46 smithi116 ceph-mon[35411]: pgmap v3084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:17:48.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:48 smithi114 ceph-mon[30820]: pgmap v3085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:48 smithi116 ceph-mon[35411]: pgmap v3085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:50 smithi114 ceph-mon[30820]: pgmap v3086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:50 smithi116 ceph-mon[35411]: pgmap v3086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:52 smithi114 ceph-mon[30820]: pgmap v3087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:52 smithi116 ceph-mon[35411]: pgmap v3087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:17:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:54 smithi114 ceph-mon[30820]: pgmap v3088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:54 smithi116 ceph-mon[35411]: pgmap v3088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:17:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:56 smithi114 ceph-mon[30820]: pgmap v3089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:17:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:56 smithi116 ceph-mon[35411]: pgmap v3089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:17:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:17:58 smithi114 ceph-mon[30820]: pgmap v3090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:17:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:17:58 smithi116 ceph-mon[35411]: pgmap v3090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:00 smithi114 ceph-mon[30820]: pgmap v3091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:00 smithi116 ceph-mon[35411]: pgmap v3091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:02.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:02 smithi114 ceph-mon[30820]: pgmap v3092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:02 smithi116 ceph-mon[35411]: pgmap v3092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:04.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:04 smithi114 ceph-mon[30820]: pgmap v3093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:04 smithi116 ceph-mon[35411]: pgmap v3093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:06.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:06 smithi114 ceph-mon[30820]: pgmap v3094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:06 smithi116 ceph-mon[35411]: pgmap v3094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:08 smithi114 ceph-mon[30820]: pgmap v3095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:08 smithi116 ceph-mon[35411]: pgmap v3095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:10 smithi114 ceph-mon[30820]: pgmap v3096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:10 smithi116 ceph-mon[35411]: pgmap v3096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:12.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:12 smithi114 ceph-mon[30820]: pgmap v3097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:12 smithi116 ceph-mon[35411]: pgmap v3097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:13.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:18:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:18:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:14 smithi114 ceph-mon[30820]: pgmap v3098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:14 smithi116 ceph-mon[35411]: pgmap v3098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:16.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:18:15.764+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:18:16.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:18:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:18:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:18:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:16 smithi114 ceph-mon[30820]: pgmap v3099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:18:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:18:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:18:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:16 smithi116 ceph-mon[35411]: pgmap v3099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:18:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:18:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:18:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:18:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:18:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:18:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:18:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:18:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:18:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:18:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:18:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:18:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:18:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:18:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:18:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:18:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:18:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:18:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:18:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:18:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:18:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:18:17.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:18:17.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:18:17.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:18:17.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:18:17.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:17 smithi114 ceph-mon[30820]: pgmap v3100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:18:17.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:17 smithi114 ceph-mon[30820]: pgmap v3101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:18:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:18:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:18:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:18:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:18:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:18:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:18:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:18:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:18:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:18:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:18:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:18:17.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:18:17.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:17 smithi116 ceph-mon[35411]: pgmap v3100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:18:17.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:17 smithi116 ceph-mon[35411]: pgmap v3101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:18:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:19 smithi114 ceph-mon[30820]: pgmap v3102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:18:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:19 smithi116 ceph-mon[35411]: pgmap v3102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:18:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:21 smithi114 ceph-mon[30820]: pgmap v3103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:18:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:21 smithi116 ceph-mon[35411]: pgmap v3103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:18:23.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:23 smithi114 ceph-mon[30820]: pgmap v3104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:18:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:23 smithi116 ceph-mon[35411]: pgmap v3104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:18:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:25 smithi114 ceph-mon[30820]: pgmap v3105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:18:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:25 smithi116 ceph-mon[35411]: pgmap v3105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:18:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:27 smithi114 ceph-mon[30820]: pgmap v3106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:18:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:27 smithi116 ceph-mon[35411]: pgmap v3106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:18:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:29 smithi114 ceph-mon[30820]: pgmap v3107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:29 smithi116 ceph-mon[35411]: pgmap v3107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:31 smithi114 ceph-mon[30820]: pgmap v3108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:31 smithi116 ceph-mon[35411]: pgmap v3108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:33 smithi114 ceph-mon[30820]: pgmap v3109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:33 smithi116 ceph-mon[35411]: pgmap v3109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:35.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:35 smithi114 ceph-mon[30820]: pgmap v3110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:35 smithi116 ceph-mon[35411]: pgmap v3110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:37 smithi114 ceph-mon[30820]: pgmap v3111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:37 smithi116 ceph-mon[35411]: pgmap v3111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:39 smithi114 ceph-mon[30820]: pgmap v3112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:39 smithi116 ceph-mon[35411]: pgmap v3112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:41.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:41 smithi114 ceph-mon[30820]: pgmap v3113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:41 smithi116 ceph-mon[35411]: pgmap v3113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:43.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:43 smithi114 ceph-mon[30820]: pgmap v3114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:43 smithi116 ceph-mon[35411]: pgmap v3114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:45.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:45 smithi114 ceph-mon[30820]: pgmap v3115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:45 smithi116 ceph-mon[35411]: pgmap v3115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:47.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:47 smithi114 ceph-mon[30820]: pgmap v3116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:47 smithi116 ceph-mon[35411]: pgmap v3116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:49 smithi114 ceph-mon[30820]: pgmap v3117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:49 smithi116 ceph-mon[35411]: pgmap v3117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:51.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:51 smithi114 ceph-mon[30820]: pgmap v3118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:51.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:51 smithi116 ceph-mon[35411]: pgmap v3118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:18:53.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:53 smithi114 ceph-mon[30820]: pgmap v3119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:53 smithi116 ceph-mon[35411]: pgmap v3119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:18:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:55 smithi114 ceph-mon[30820]: pgmap v3120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:55.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:55 smithi116 ceph-mon[35411]: pgmap v3120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:57.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:57 smithi114 ceph-mon[30820]: pgmap v3121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:57 smithi116 ceph-mon[35411]: pgmap v3121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:18:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:18:59 smithi114 ceph-mon[30820]: pgmap v3122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:18:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:18:59 smithi116 ceph-mon[35411]: pgmap v3122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:01 smithi114 ceph-mon[30820]: pgmap v3123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:01 smithi116 ceph-mon[35411]: pgmap v3123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:03 smithi114 ceph-mon[30820]: pgmap v3124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:03.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:03 smithi116 ceph-mon[35411]: pgmap v3124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:05 smithi114 ceph-mon[30820]: pgmap v3125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:05.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:05 smithi116 ceph-mon[35411]: pgmap v3125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:07.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:07 smithi114 ceph-mon[30820]: pgmap v3126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:19:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:07 smithi116 ceph-mon[35411]: pgmap v3126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:19:09.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:09 smithi114 ceph-mon[30820]: pgmap v3127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:09 smithi116 ceph-mon[35411]: pgmap v3127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:11.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:11 smithi114 ceph-mon[30820]: pgmap v3128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:11.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:11 smithi116 ceph-mon[35411]: pgmap v3128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:13 smithi114 ceph-mon[30820]: pgmap v3129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:13 smithi116 ceph-mon[35411]: pgmap v3129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:15 smithi114 ceph-mon[30820]: pgmap v3130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:15.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:15 smithi116 ceph-mon[35411]: pgmap v3130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:19:16.666 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:19:17.480 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:17 smithi114 ceph-mon[30820]: pgmap v3131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:19:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:17 smithi116 ceph-mon[35411]: pgmap v3131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:19:18.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:18 smithi114 ceph-mon[30820]: pgmap v3132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:18.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:18 smithi116 ceph-mon[35411]: pgmap v3132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:19.399 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:19:19.124+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:19:19.399 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:19:19.400 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:19:19.400 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:19:19.400 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:19:19.400 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:19:19.400 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:19:19.401 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:19:19.401 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:19:19.401 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:19:19.401 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:19:19.401 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:19:19.401 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:19:19.402 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:19:19.402 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:19:19.402 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:19:19.402 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:19:19.402 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:19:19.402 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:19:19.403 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:19:19.403 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:19:19.403 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:19:19.403 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:19:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:19:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:19:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:19:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:19:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:19:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:19:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:20 smithi114 ceph-mon[30820]: pgmap v3133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:20 smithi116 ceph-mon[35411]: pgmap v3133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:22.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:19:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:19:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:21 smithi114 ceph-mon[30820]: pgmap v3134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 6 op/s 2024-09-19T08:19:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:21 smithi114 ceph-mon[30820]: pgmap v3135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:19:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:19:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:19:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:21 smithi116 ceph-mon[35411]: pgmap v3134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 6 op/s 2024-09-19T08:19:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:21 smithi116 ceph-mon[35411]: pgmap v3135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:19:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:19:24.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:23 smithi114 ceph-mon[30820]: pgmap v3136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T08:19:24.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:23 smithi116 ceph-mon[35411]: pgmap v3136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T08:19:26.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:25 smithi114 ceph-mon[30820]: pgmap v3137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-09-19T08:19:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:25 smithi116 ceph-mon[35411]: pgmap v3137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-09-19T08:19:28.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:27 smithi114 ceph-mon[30820]: pgmap v3138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T08:19:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:27 smithi116 ceph-mon[35411]: pgmap v3138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 6 op/s 2024-09-19T08:19:30.096 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:29 smithi116 ceph-mon[35411]: pgmap v3139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 5 op/s 2024-09-19T08:19:30.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:29 smithi114 ceph-mon[30820]: pgmap v3139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 5 op/s 2024-09-19T08:19:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:31 smithi114 ceph-mon[30820]: pgmap v3140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:19:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:31 smithi116 ceph-mon[35411]: pgmap v3140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:19:34.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:33 smithi114 ceph-mon[30820]: pgmap v3141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:33 smithi116 ceph-mon[35411]: pgmap v3141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:35 smithi114 ceph-mon[30820]: pgmap v3142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:35 smithi116 ceph-mon[35411]: pgmap v3142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:37 smithi114 ceph-mon[30820]: pgmap v3143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:37 smithi116 ceph-mon[35411]: pgmap v3143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:40.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:39 smithi114 ceph-mon[30820]: pgmap v3144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:39 smithi116 ceph-mon[35411]: pgmap v3144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:41 smithi114 ceph-mon[30820]: pgmap v3145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:19:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:41 smithi116 ceph-mon[35411]: pgmap v3145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:19:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:43 smithi114 ceph-mon[30820]: pgmap v3146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:43 smithi116 ceph-mon[35411]: pgmap v3146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:45 smithi114 ceph-mon[30820]: pgmap v3147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:45 smithi116 ceph-mon[35411]: pgmap v3147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:48.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:47 smithi114 ceph-mon[30820]: pgmap v3148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:47 smithi116 ceph-mon[35411]: pgmap v3148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:49 smithi114 ceph-mon[30820]: pgmap v3149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:49 smithi116 ceph-mon[35411]: pgmap v3149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:51 smithi114 ceph-mon[30820]: pgmap v3150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:19:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:51 smithi116 ceph-mon[35411]: pgmap v3150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:19:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:53 smithi114 ceph-mon[30820]: pgmap v3151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:53 smithi116 ceph-mon[35411]: pgmap v3151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:19:56.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:55 smithi114 ceph-mon[30820]: pgmap v3152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:55 smithi116 ceph-mon[35411]: pgmap v3152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:19:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:57 smithi114 ceph-mon[30820]: pgmap v3153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:19:58.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:57 smithi116 ceph-mon[35411]: pgmap v3153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:19:59 smithi114 ceph-mon[30820]: pgmap v3154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:19:59 smithi116 ceph-mon[35411]: pgmap v3154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:01.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:20:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:20:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:01 smithi114 ceph-mon[30820]: pgmap v3155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:01 smithi116 ceph-mon[35411]: pgmap v3155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:03 smithi114 ceph-mon[30820]: pgmap v3156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:03 smithi116 ceph-mon[35411]: pgmap v3156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:05 smithi114 ceph-mon[30820]: pgmap v3157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:05 smithi116 ceph-mon[35411]: pgmap v3157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:07 smithi114 ceph-mon[30820]: pgmap v3158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:07 smithi116 ceph-mon[35411]: pgmap v3158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:10.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:09 smithi114 ceph-mon[30820]: pgmap v3159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:09 smithi116 ceph-mon[35411]: pgmap v3159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:11 smithi114 ceph-mon[30820]: pgmap v3160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:11 smithi116 ceph-mon[35411]: pgmap v3160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:13 smithi114 ceph-mon[30820]: pgmap v3161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:13 smithi116 ceph-mon[35411]: pgmap v3161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:16.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:15 smithi114 ceph-mon[30820]: pgmap v3162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:15 smithi116 ceph-mon[35411]: pgmap v3162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:18.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:17 smithi114 ceph-mon[30820]: pgmap v3163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:18.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:17 smithi116 ceph-mon[35411]: pgmap v3163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:19 smithi114 ceph-mon[30820]: pgmap v3164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:20.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:19 smithi116 ceph-mon[35411]: pgmap v3164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:21.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:20:21.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:20:22.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:21 smithi114 ceph-mon[30820]: pgmap v3165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:21 smithi116 ceph-mon[35411]: pgmap v3165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:22.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:20:22.549+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:20:22.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:20:22.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:20:22.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:20:22.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:20:22.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:20:22.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:20:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:20:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:20:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:20:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:20:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:20:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:20:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:20:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:20:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:20:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:20:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:20:22.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:20:22.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:20:22.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:20:22.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:20:22.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:20:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:20:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:20:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:20:23.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:20:23.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:20:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:23 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:20:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:23 smithi116 ceph-mon[35411]: pgmap v3166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T08:20:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:23 smithi116 ceph-mon[35411]: pgmap v3167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:24.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:23 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:20:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:20:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:23 smithi114 ceph-mon[30820]: pgmap v3166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T08:20:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:23 smithi114 ceph-mon[30820]: pgmap v3167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:25 smithi116 ceph-mon[35411]: pgmap v3168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:25 smithi114 ceph-mon[30820]: pgmap v3168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:28.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:27 smithi114 ceph-mon[30820]: pgmap v3169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:27 smithi116 ceph-mon[35411]: pgmap v3169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:30.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:29 smithi114 ceph-mon[30820]: pgmap v3170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:30.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:29 smithi116 ceph-mon[35411]: pgmap v3170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:32.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:31 smithi114 ceph-mon[30820]: pgmap v3171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:31 smithi116 ceph-mon[35411]: pgmap v3171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:20:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:33 smithi114 ceph-mon[30820]: pgmap v3172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:20:34.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:33 smithi116 ceph-mon[35411]: pgmap v3172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:20:36.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:35 smithi114 ceph-mon[30820]: pgmap v3173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:36.229 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:35 smithi116 ceph-mon[35411]: pgmap v3173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:38.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:37 smithi114 ceph-mon[30820]: pgmap v3174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:37 smithi116 ceph-mon[35411]: pgmap v3174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:39 smithi114 ceph-mon[30820]: pgmap v3175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:40.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:39 smithi116 ceph-mon[35411]: pgmap v3175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:41 smithi114 ceph-mon[30820]: pgmap v3176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:42.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:41 smithi116 ceph-mon[35411]: pgmap v3176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:44.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:43 smithi114 ceph-mon[30820]: pgmap v3177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:44.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:43 smithi116 ceph-mon[35411]: pgmap v3177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:45 smithi114 ceph-mon[30820]: pgmap v3178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:46.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:45 smithi116 ceph-mon[35411]: pgmap v3178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:47 smithi114 ceph-mon[30820]: pgmap v3179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:48.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:47 smithi116 ceph-mon[35411]: pgmap v3179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:50.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:49 smithi114 ceph-mon[30820]: pgmap v3180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:49 smithi116 ceph-mon[35411]: pgmap v3180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:52.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:51 smithi114 ceph-mon[30820]: pgmap v3181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:52.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:51 smithi116 ceph-mon[35411]: pgmap v3181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:20:54.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:53 smithi114 ceph-mon[30820]: pgmap v3182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:54.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:53 smithi116 ceph-mon[35411]: pgmap v3182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:20:56.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:55 smithi114 ceph-mon[30820]: pgmap v3183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:55 smithi116 ceph-mon[35411]: pgmap v3183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:20:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:57 smithi114 ceph-mon[30820]: pgmap v3184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:20:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:57 smithi116 ceph-mon[35411]: pgmap v3184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:20:59 smithi114 ceph-mon[30820]: pgmap v3185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:20:59 smithi116 ceph-mon[35411]: pgmap v3185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:01 smithi114 ceph-mon[30820]: pgmap v3186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:01 smithi116 ceph-mon[35411]: pgmap v3186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:04.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:03 smithi114 ceph-mon[30820]: pgmap v3187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:04.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:03 smithi116 ceph-mon[35411]: pgmap v3187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:06.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:06 smithi114 ceph-mon[30820]: pgmap v3188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:06 smithi116 ceph-mon[35411]: pgmap v3188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:08.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:08 smithi114 ceph-mon[30820]: pgmap v3189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:08.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:08 smithi116 ceph-mon[35411]: pgmap v3189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:10 smithi114 ceph-mon[30820]: pgmap v3190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:10 smithi116 ceph-mon[35411]: pgmap v3190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:12 smithi114 ceph-mon[30820]: pgmap v3191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:12 smithi116 ceph-mon[35411]: pgmap v3191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:14.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:14 smithi114 ceph-mon[30820]: pgmap v3192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:14 smithi116 ceph-mon[35411]: pgmap v3192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:16.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:16 smithi114 ceph-mon[30820]: pgmap v3193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:16 smithi116 ceph-mon[35411]: pgmap v3193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:18.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:18 smithi114 ceph-mon[30820]: pgmap v3194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:18 smithi116 ceph-mon[35411]: pgmap v3194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:20.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:20 smithi114 ceph-mon[30820]: pgmap v3195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:20.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:20 smithi116 ceph-mon[35411]: pgmap v3195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:22.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:22 smithi114 ceph-mon[30820]: pgmap v3196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:22.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:22 smithi116 ceph-mon[35411]: pgmap v3196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:23.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:21:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:21:24.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:24 smithi114 ceph-mon[30820]: pgmap v3197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:24.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:24 smithi116 ceph-mon[35411]: pgmap v3197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:26 smithi114 ceph-mon[30820]: pgmap v3198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:21:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:21:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:21:25.909+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:21:26.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:21:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:21:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:21:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:21:26.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:26 smithi116 ceph-mon[35411]: pgmap v3198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:21:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:21:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:21:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:27 smithi114 ceph-mon[30820]: pgmap v3199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:27 smithi114 ceph-mon[30820]: pgmap v3200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:21:27.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:21:27.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:21:27.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:21:27.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:27 smithi116 ceph-mon[35411]: pgmap v3199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:21:27.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:27 smithi116 ceph-mon[35411]: pgmap v3200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:21:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:29 smithi114 ceph-mon[30820]: pgmap v3201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:21:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:29 smithi116 ceph-mon[35411]: pgmap v3201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:21:31.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:31 smithi114 ceph-mon[30820]: pgmap v3202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:21:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:31 smithi116 ceph-mon[35411]: pgmap v3202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:21:33.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:33 smithi114 ceph-mon[30820]: pgmap v3203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:21:33.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:33 smithi116 ceph-mon[35411]: pgmap v3203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:21:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:35 smithi114 ceph-mon[30820]: pgmap v3204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:21:35.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:35 smithi116 ceph-mon[35411]: pgmap v3204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:21:37.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:37 smithi114 ceph-mon[30820]: pgmap v3205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:21:37.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:37 smithi116 ceph-mon[35411]: pgmap v3205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:21:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:39 smithi114 ceph-mon[30820]: pgmap v3206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:39.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:39 smithi116 ceph-mon[35411]: pgmap v3206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:41.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:41 smithi114 ceph-mon[30820]: pgmap v3207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:41.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:41 smithi116 ceph-mon[35411]: pgmap v3207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:43.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:43 smithi114 ceph-mon[30820]: pgmap v3208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:43.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:43 smithi116 ceph-mon[35411]: pgmap v3208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:45.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:45 smithi114 ceph-mon[30820]: pgmap v3209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:45.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:45 smithi116 ceph-mon[35411]: pgmap v3209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:47 smithi114 ceph-mon[30820]: pgmap v3210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:47.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:47 smithi116 ceph-mon[35411]: pgmap v3210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:49.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:49 smithi114 ceph-mon[30820]: pgmap v3211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:49.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:49 smithi116 ceph-mon[35411]: pgmap v3211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:51 smithi114 ceph-mon[30820]: pgmap v3212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:51 smithi116 ceph-mon[35411]: pgmap v3212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:21:53.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:53 smithi114 ceph-mon[30820]: pgmap v3213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:53 smithi116 ceph-mon[35411]: pgmap v3213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:21:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:55 smithi114 ceph-mon[30820]: pgmap v3214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:55.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:55 smithi116 ceph-mon[35411]: pgmap v3214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:56 smithi114 ceph-mon[30820]: pgmap v3215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:56 smithi116 ceph-mon[35411]: pgmap v3215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:21:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:21:58 smithi114 ceph-mon[30820]: pgmap v3216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:21:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:21:58 smithi116 ceph-mon[35411]: pgmap v3216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:01.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:00 smithi114 ceph-mon[30820]: pgmap v3217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:22:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:00 smithi116 ceph-mon[35411]: pgmap v3217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:22:03.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:02 smithi114 ceph-mon[30820]: pgmap v3218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:02 smithi116 ceph-mon[35411]: pgmap v3218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:04 smithi114 ceph-mon[30820]: pgmap v3219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:04 smithi116 ceph-mon[35411]: pgmap v3219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:07.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:06 smithi114 ceph-mon[30820]: pgmap v3220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:22:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:06 smithi116 ceph-mon[35411]: pgmap v3220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:22:09.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:08 smithi114 ceph-mon[30820]: pgmap v3221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:08 smithi116 ceph-mon[35411]: pgmap v3221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:11 smithi114 ceph-mon[30820]: pgmap v3222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:22:11.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:11 smithi116 ceph-mon[35411]: pgmap v3222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:22:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:13 smithi114 ceph-mon[30820]: pgmap v3223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:13 smithi116 ceph-mon[35411]: pgmap v3223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:15.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:15 smithi114 ceph-mon[30820]: pgmap v3224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:15 smithi116 ceph-mon[35411]: pgmap v3224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:17.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:17 smithi114 ceph-mon[30820]: pgmap v3225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:22:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:17 smithi116 ceph-mon[35411]: pgmap v3225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:22:19.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:19 smithi114 ceph-mon[30820]: pgmap v3226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:19 smithi116 ceph-mon[35411]: pgmap v3226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:21.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:21 smithi114 ceph-mon[30820]: pgmap v3227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:22:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:21 smithi116 ceph-mon[35411]: pgmap v3227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:22:23.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:23 smithi114 ceph-mon[30820]: pgmap v3228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:23 smithi116 ceph-mon[35411]: pgmap v3228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:25.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:25 smithi114 ceph-mon[30820]: pgmap v3229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:25 smithi116 ceph-mon[35411]: pgmap v3229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: pgmap v3230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:22:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:22:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: pgmap v3230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:22:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:28 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:22:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:28 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:22:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:28 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:22:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:28 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:22:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:28 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:22:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:28 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:22:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:28 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:22:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:28 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:22:29.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-mon[30820]: pgmap v3231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:22:29.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:22:29.205+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:22:29.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:22:29.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:22:29.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:22:29.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:22:29.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:22:29.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:22:29.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:22:29.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:22:29.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:22:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:22:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:22:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:22:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:22:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:22:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:22:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:29 smithi116 ceph-mon[35411]: pgmap v3231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:30 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:22:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:30 smithi114 ceph-mon[30820]: pgmap v3232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:30 smithi114 ceph-mon[30820]: pgmap v3233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:30 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:22:30.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:22:30.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:22:30.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:22:30.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:22:30.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:22:30.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:30 smithi116 ceph-mon[35411]: pgmap v3232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:22:30.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:30 smithi116 ceph-mon[35411]: pgmap v3233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:22:30.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:22:32.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:32 smithi114 ceph-mon[30820]: pgmap v3234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:22:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:32 smithi116 ceph-mon[35411]: pgmap v3234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:22:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:34 smithi114 ceph-mon[30820]: pgmap v3235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:22:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:34 smithi116 ceph-mon[35411]: pgmap v3235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:22:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:36 smithi114 ceph-mon[30820]: pgmap v3236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:22:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:36 smithi116 ceph-mon[35411]: pgmap v3236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:22:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:38 smithi114 ceph-mon[30820]: pgmap v3237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:22:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:38 smithi116 ceph-mon[35411]: pgmap v3237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:22:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:40 smithi114 ceph-mon[30820]: pgmap v3238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:22:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:40 smithi116 ceph-mon[35411]: pgmap v3238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:22:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:42 smithi114 ceph-mon[30820]: pgmap v3239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:42 smithi116 ceph-mon[35411]: pgmap v3239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:44 smithi114 ceph-mon[30820]: pgmap v3240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:44 smithi116 ceph-mon[35411]: pgmap v3240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:46 smithi114 ceph-mon[30820]: pgmap v3241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:22:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:46 smithi116 ceph-mon[35411]: pgmap v3241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:22:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:48 smithi114 ceph-mon[30820]: pgmap v3242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:22:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:48 smithi116 ceph-mon[35411]: pgmap v3242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:22:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:50 smithi114 ceph-mon[30820]: pgmap v3243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:22:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:50 smithi116 ceph-mon[35411]: pgmap v3243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:22:52.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:52 smithi114 ceph-mon[30820]: pgmap v3244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:52 smithi116 ceph-mon[35411]: pgmap v3244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:22:54.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:54 smithi114 ceph-mon[30820]: pgmap v3245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:54 smithi116 ceph-mon[35411]: pgmap v3245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:22:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:56 smithi114 ceph-mon[30820]: pgmap v3246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:22:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:56 smithi116 ceph-mon[35411]: pgmap v3246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:22:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:22:58 smithi114 ceph-mon[30820]: pgmap v3247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:22:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:22:58 smithi116 ceph-mon[35411]: pgmap v3247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:23:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:00 smithi114 ceph-mon[30820]: pgmap v3248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:00 smithi116 ceph-mon[35411]: pgmap v3248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:02.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:02 smithi114 ceph-mon[30820]: pgmap v3249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:02 smithi116 ceph-mon[35411]: pgmap v3249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:04.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:04 smithi114 ceph-mon[30820]: pgmap v3250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:04 smithi116 ceph-mon[35411]: pgmap v3250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:06 smithi114 ceph-mon[30820]: pgmap v3251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:23:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:06 smithi116 ceph-mon[35411]: pgmap v3251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:23:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:08 smithi114 ceph-mon[30820]: pgmap v3252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:23:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:08 smithi116 ceph-mon[35411]: pgmap v3252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:23:10.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:10 smithi114 ceph-mon[30820]: pgmap v3253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:10 smithi116 ceph-mon[35411]: pgmap v3253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:12.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:12 smithi114 ceph-mon[30820]: pgmap v3254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:12 smithi116 ceph-mon[35411]: pgmap v3254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:14 smithi114 ceph-mon[30820]: pgmap v3255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:14.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:14 smithi116 ceph-mon[35411]: pgmap v3255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:16 smithi114 ceph-mon[30820]: pgmap v3256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:23:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:16 smithi116 ceph-mon[35411]: pgmap v3256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:23:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:18 smithi114 ceph-mon[30820]: pgmap v3257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:23:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:18 smithi116 ceph-mon[35411]: pgmap v3257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:23:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:20 smithi114 ceph-mon[30820]: pgmap v3258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:20 smithi116 ceph-mon[35411]: pgmap v3258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:22.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:22 smithi114 ceph-mon[30820]: pgmap v3259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:22 smithi116 ceph-mon[35411]: pgmap v3259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:24 smithi114 ceph-mon[30820]: pgmap v3260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:24 smithi116 ceph-mon[35411]: pgmap v3260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:26 smithi114 ceph-mon[30820]: pgmap v3261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:23:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:26 smithi116 ceph-mon[35411]: pgmap v3261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:23:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:28 smithi114 ceph-mon[30820]: pgmap v3262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:23:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:28 smithi116 ceph-mon[35411]: pgmap v3262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:23:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:23:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:23:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:30 smithi114 ceph-mon[30820]: pgmap v3263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:30.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:30 smithi116 ceph-mon[35411]: pgmap v3263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-mon[30820]: pgmap v3264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:32.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:23:32.565+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:23:32.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:23:32.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:23:32.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:23:32.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:23:32.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:23:32.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:23:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:23:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:23:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:23:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:23:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:23:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:32 smithi116 ceph-mon[35411]: pgmap v3264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:23:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:23:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:23:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:23:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:23:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:23:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:34 smithi114 ceph-mon[30820]: pgmap v3265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:34 smithi114 ceph-mon[30820]: pgmap v3266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:23:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:34 smithi116 ceph-mon[35411]: pgmap v3265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:34 smithi116 ceph-mon[35411]: pgmap v3266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:23:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:36 smithi114 ceph-mon[30820]: pgmap v3267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:23:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:36 smithi116 ceph-mon[35411]: pgmap v3267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:23:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:38 smithi114 ceph-mon[30820]: pgmap v3268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:23:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:38 smithi116 ceph-mon[35411]: pgmap v3268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:23:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:40 smithi114 ceph-mon[30820]: pgmap v3269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:23:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:40 smithi116 ceph-mon[35411]: pgmap v3269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:23:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:42 smithi114 ceph-mon[30820]: pgmap v3270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:23:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:42 smithi116 ceph-mon[35411]: pgmap v3270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:23:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:44 smithi114 ceph-mon[30820]: pgmap v3271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:23:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:44 smithi116 ceph-mon[35411]: pgmap v3271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:23:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:46 smithi114 ceph-mon[30820]: pgmap v3272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:46 smithi116 ceph-mon[35411]: pgmap v3272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:48.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:48 smithi114 ceph-mon[30820]: pgmap v3273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:48 smithi116 ceph-mon[35411]: pgmap v3273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:50 smithi114 ceph-mon[30820]: pgmap v3274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:50 smithi116 ceph-mon[35411]: pgmap v3274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:52 smithi116 ceph-mon[35411]: pgmap v3275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:23:52.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:52 smithi114 ceph-mon[30820]: pgmap v3275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:23:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:54 smithi116 ceph-mon[35411]: pgmap v3276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:54 smithi114 ceph-mon[30820]: pgmap v3276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:23:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:56 smithi116 ceph-mon[35411]: pgmap v3277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:56 smithi114 ceph-mon[30820]: pgmap v3277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:23:58.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:23:58 smithi114 ceph-mon[30820]: pgmap v3278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:23:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:23:58 smithi116 ceph-mon[35411]: pgmap v3278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:00 smithi114 ceph-mon[30820]: pgmap v3279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:00 smithi116 ceph-mon[35411]: pgmap v3279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:01.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:01 smithi114 ceph-mon[30820]: pgmap v3280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:01 smithi116 ceph-mon[35411]: pgmap v3280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:03 smithi114 ceph-mon[30820]: pgmap v3281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:03 smithi116 ceph-mon[35411]: pgmap v3281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:05 smithi114 ceph-mon[30820]: pgmap v3282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:24:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:05 smithi116 ceph-mon[35411]: pgmap v3282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:24:07.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:07 smithi114 ceph-mon[30820]: pgmap v3283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:07 smithi116 ceph-mon[35411]: pgmap v3283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:09 smithi114 ceph-mon[30820]: pgmap v3284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:09 smithi116 ceph-mon[35411]: pgmap v3284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:11 smithi114 ceph-mon[30820]: pgmap v3285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:11 smithi116 ceph-mon[35411]: pgmap v3285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:13 smithi114 ceph-mon[30820]: pgmap v3286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:13 smithi116 ceph-mon[35411]: pgmap v3286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:15 smithi114 ceph-mon[30820]: pgmap v3287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:24:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:15 smithi116 ceph-mon[35411]: pgmap v3287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:24:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:17 smithi114 ceph-mon[30820]: pgmap v3288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:17 smithi116 ceph-mon[35411]: pgmap v3288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:19 smithi116 ceph-mon[35411]: pgmap v3289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:19 smithi114 ceph-mon[30820]: pgmap v3289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:21 smithi116 ceph-mon[35411]: pgmap v3290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:22.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:21 smithi114 ceph-mon[30820]: pgmap v3290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:23 smithi116 ceph-mon[35411]: pgmap v3291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:24.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:23 smithi114 ceph-mon[30820]: pgmap v3291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:25 smithi114 ceph-mon[30820]: pgmap v3292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:24:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:25 smithi116 ceph-mon[35411]: pgmap v3292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:24:28.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:27 smithi114 ceph-mon[30820]: pgmap v3293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:27 smithi116 ceph-mon[35411]: pgmap v3293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:30.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:29 smithi114 ceph-mon[30820]: pgmap v3294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:29 smithi116 ceph-mon[35411]: pgmap v3294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:31 smithi114 ceph-mon[30820]: pgmap v3295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:31 smithi116 ceph-mon[35411]: pgmap v3295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:34.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:33 smithi114 ceph-mon[30820]: pgmap v3296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:24:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:33 smithi116 ceph-mon[35411]: pgmap v3296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:24:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-mon[30820]: pgmap v3297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:24:36.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:24:35.924+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:24:36.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:24:36.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:24:36.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:24:36.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:24:36.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:24:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:24:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:24:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:24:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:24:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:24:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:24:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:24:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:35 smithi116 ceph-mon[35411]: pgmap v3297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:24:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:24:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:24:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:24:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:24:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:24:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:24:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:24:37.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:24:37.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:24:37.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:24:37.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:24:37.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:24:37.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:24:37.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:24:37.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:24:37.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:24:37.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:36 smithi114 ceph-mon[30820]: pgmap v3298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:24:37.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:36 smithi114 ceph-mon[30820]: pgmap v3299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:24:37.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:24:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:24:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:36 smithi116 ceph-mon[35411]: pgmap v3298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:24:37.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:36 smithi116 ceph-mon[35411]: pgmap v3299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:24:37.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:24:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:38 smithi114 ceph-mon[30820]: pgmap v3300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:24:39.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:38 smithi116 ceph-mon[35411]: pgmap v3300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:24:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:40 smithi114 ceph-mon[30820]: pgmap v3301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:24:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:40 smithi116 ceph-mon[35411]: pgmap v3301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:24:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:42 smithi114 ceph-mon[30820]: pgmap v3302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:24:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:42 smithi116 ceph-mon[35411]: pgmap v3302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:24:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:44 smithi114 ceph-mon[30820]: pgmap v3303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:24:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:44 smithi116 ceph-mon[35411]: pgmap v3303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:24:47.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:46 smithi114 ceph-mon[30820]: pgmap v3304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:24:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:47 smithi116 ceph-mon[35411]: pgmap v3304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:24:49.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:49 smithi114 ceph-mon[30820]: pgmap v3305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:24:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:49 smithi116 ceph-mon[35411]: pgmap v3305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:24:51.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:51 smithi114 ceph-mon[30820]: pgmap v3306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:24:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:51 smithi116 ceph-mon[35411]: pgmap v3306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:24:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:53 smithi114 ceph-mon[30820]: pgmap v3307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:53 smithi116 ceph-mon[35411]: pgmap v3307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:24:55.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:55 smithi114 ceph-mon[30820]: pgmap v3308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:24:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:55 smithi116 ceph-mon[35411]: pgmap v3308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:24:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:57 smithi114 ceph-mon[30820]: pgmap v3309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:57 smithi116 ceph-mon[35411]: pgmap v3309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:24:59.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:24:59 smithi114 ceph-mon[30820]: pgmap v3310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:24:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:24:59 smithi116 ceph-mon[35411]: pgmap v3310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:01 smithi114 ceph-mon[30820]: pgmap v3311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:25:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:01 smithi116 ceph-mon[35411]: pgmap v3311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:25:03.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:03 smithi114 ceph-mon[30820]: pgmap v3312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:03 smithi116 ceph-mon[35411]: pgmap v3312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:05.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:05 smithi114 ceph-mon[30820]: pgmap v3313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:05 smithi116 ceph-mon[35411]: pgmap v3313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:07.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:07 smithi114 ceph-mon[30820]: pgmap v3314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:25:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:07 smithi116 ceph-mon[35411]: pgmap v3314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:25:09.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:09 smithi114 ceph-mon[30820]: pgmap v3315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:09 smithi116 ceph-mon[35411]: pgmap v3315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:11 smithi114 ceph-mon[30820]: pgmap v3316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:25:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:11 smithi116 ceph-mon[35411]: pgmap v3316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:25:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:13 smithi114 ceph-mon[30820]: pgmap v3317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:13 smithi116 ceph-mon[35411]: pgmap v3317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:15 smithi114 ceph-mon[30820]: pgmap v3318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:15 smithi116 ceph-mon[35411]: pgmap v3318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:17 smithi114 ceph-mon[30820]: pgmap v3319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:25:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:17 smithi116 ceph-mon[35411]: pgmap v3319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:25:19.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:19 smithi114 ceph-mon[30820]: pgmap v3320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:19 smithi116 ceph-mon[35411]: pgmap v3320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:21 smithi114 ceph-mon[30820]: pgmap v3321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:25:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:21 smithi116 ceph-mon[35411]: pgmap v3321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:25:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:23 smithi114 ceph-mon[30820]: pgmap v3322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:23.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:23 smithi116 ceph-mon[35411]: pgmap v3322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:25.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:25 smithi114 ceph-mon[30820]: pgmap v3323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:25 smithi116 ceph-mon[35411]: pgmap v3323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:27 smithi114 ceph-mon[30820]: pgmap v3324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:25:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:27 smithi116 ceph-mon[35411]: pgmap v3324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:25:29.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:29 smithi114 ceph-mon[30820]: pgmap v3325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:29 smithi116 ceph-mon[35411]: pgmap v3325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:31.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:31 smithi114 ceph-mon[30820]: pgmap v3326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:25:31.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:31 smithi116 ceph-mon[35411]: pgmap v3326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:25:33.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:33 smithi114 ceph-mon[30820]: pgmap v3327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:33.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:33 smithi116 ceph-mon[35411]: pgmap v3327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:35.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:35 smithi114 ceph-mon[30820]: pgmap v3328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:35 smithi116 ceph-mon[35411]: pgmap v3328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:36.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:25:36.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:25:37.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:37 smithi114 ceph-mon[30820]: pgmap v3329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:25:37.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:37 smithi116 ceph-mon[35411]: pgmap v3329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:25:38.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:38 smithi114 ceph-mon[30820]: pgmap v3330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:38 smithi116 ceph-mon[35411]: pgmap v3330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:39.626 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:25:39.284+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:25:39.626 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:25:39.626 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:25:39.626 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:25:39.626 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:25:39.627 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:25:39.628 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:25:39.628 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:25:39.628 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:25:39.628 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:25:39.628 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:25:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:25:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:25:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-mon[30820]: pgmap v3331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:25:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-mon[30820]: pgmap v3332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:25:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:25:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:25:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:25:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:25:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:25:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:25:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:39 smithi116 ceph-mon[35411]: pgmap v3331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:39 smithi116 ceph-mon[35411]: pgmap v3332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:25:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:25:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:42 smithi114 ceph-mon[30820]: pgmap v3333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:25:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:42 smithi116 ceph-mon[35411]: pgmap v3333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:25:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:44 smithi114 ceph-mon[30820]: pgmap v3334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:25:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:44 smithi116 ceph-mon[35411]: pgmap v3334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:25:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:46 smithi114 ceph-mon[30820]: pgmap v3335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:25:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:46 smithi116 ceph-mon[35411]: pgmap v3335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:25:48.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:48 smithi114 ceph-mon[30820]: pgmap v3336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:25:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:48 smithi116 ceph-mon[35411]: pgmap v3336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:25:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:50 smithi114 ceph-mon[30820]: pgmap v3337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:25:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:50 smithi116 ceph-mon[35411]: pgmap v3337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:25:52.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:52 smithi114 ceph-mon[30820]: pgmap v3338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:52 smithi116 ceph-mon[35411]: pgmap v3338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:25:54.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:54 smithi114 ceph-mon[30820]: pgmap v3339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:54 smithi116 ceph-mon[35411]: pgmap v3339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:25:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:56 smithi114 ceph-mon[30820]: pgmap v3340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:25:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:56 smithi116 ceph-mon[35411]: pgmap v3340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:25:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:25:58 smithi114 ceph-mon[30820]: pgmap v3341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:25:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:25:58 smithi116 ceph-mon[35411]: pgmap v3341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:00 smithi114 ceph-mon[30820]: pgmap v3342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:00 smithi116 ceph-mon[35411]: pgmap v3342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:02.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:02 smithi114 ceph-mon[30820]: pgmap v3343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:02 smithi116 ceph-mon[35411]: pgmap v3343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:04 smithi114 ceph-mon[30820]: pgmap v3344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:26:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:04 smithi116 ceph-mon[35411]: pgmap v3344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:26:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:06 smithi114 ceph-mon[30820]: pgmap v3345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:26:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:06 smithi116 ceph-mon[35411]: pgmap v3345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:26:08.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:08 smithi114 ceph-mon[30820]: pgmap v3346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:08 smithi116 ceph-mon[35411]: pgmap v3346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:10 smithi114 ceph-mon[30820]: pgmap v3347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:10 smithi116 ceph-mon[35411]: pgmap v3347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:12 smithi114 ceph-mon[30820]: pgmap v3348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:12.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:12 smithi116 ceph-mon[35411]: pgmap v3348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:14 smithi114 ceph-mon[30820]: pgmap v3349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:26:14.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:14 smithi116 ceph-mon[35411]: pgmap v3349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:26:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:16 smithi114 ceph-mon[30820]: pgmap v3350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:26:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:16 smithi116 ceph-mon[35411]: pgmap v3350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:26:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:18 smithi114 ceph-mon[30820]: pgmap v3351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:18 smithi116 ceph-mon[35411]: pgmap v3351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:20 smithi114 ceph-mon[30820]: pgmap v3352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:20 smithi116 ceph-mon[35411]: pgmap v3352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:22 smithi116 ceph-mon[35411]: pgmap v3353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:22.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:22 smithi114 ceph-mon[30820]: pgmap v3353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:24 smithi116 ceph-mon[35411]: pgmap v3354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:26:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:24 smithi114 ceph-mon[30820]: pgmap v3354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:26:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:26 smithi116 ceph-mon[35411]: pgmap v3355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:26:26.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:26 smithi114 ceph-mon[30820]: pgmap v3355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:26:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:28 smithi114 ceph-mon[30820]: pgmap v3356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:28 smithi116 ceph-mon[35411]: pgmap v3356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:30.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:30 smithi114 ceph-mon[30820]: pgmap v3357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:30 smithi116 ceph-mon[35411]: pgmap v3357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:32 smithi114 ceph-mon[30820]: pgmap v3358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:32 smithi116 ceph-mon[35411]: pgmap v3358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:34 smithi114 ceph-mon[30820]: pgmap v3359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:26:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:34 smithi116 ceph-mon[35411]: pgmap v3359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:26:36.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:36 smithi114 ceph-mon[30820]: pgmap v3360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:26:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:36 smithi116 ceph-mon[35411]: pgmap v3360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:26:38.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:38 smithi114 ceph-mon[30820]: pgmap v3361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:38 smithi116 ceph-mon[35411]: pgmap v3361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:26:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:26:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:26:40.829 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:40 smithi116 ceph-mon[35411]: pgmap v3362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:40.829 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:40.829 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:40.829 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:40.829 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:40.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:40 smithi114 ceph-mon[30820]: pgmap v3362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:40.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:40.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:40.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:40.831 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:42.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-mon[30820]: pgmap v3363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:42.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:26:42.580+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:26:42.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:26:42.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:26:42.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:26:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:26:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:26:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:26:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:26:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:26:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:26:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:26:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:26:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:26:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:26:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:42 smithi116 ceph-mon[35411]: pgmap v3363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:43.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:26:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:26:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:26:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:26:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:44 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:26:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:26:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:26:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:44 smithi114 ceph-mon[30820]: pgmap v3364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:26:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:44 smithi114 ceph-mon[30820]: pgmap v3365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:44 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:26:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:44 smithi116 ceph-mon[35411]: pgmap v3364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:26:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:44 smithi116 ceph-mon[35411]: pgmap v3365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:26:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:45 smithi114 ceph-mon[30820]: pgmap v3366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T08:26:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:45 smithi116 ceph-mon[35411]: pgmap v3366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T08:26:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:47 smithi114 ceph-mon[30820]: pgmap v3367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:26:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:47 smithi116 ceph-mon[35411]: pgmap v3367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:26:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:49 smithi114 ceph-mon[30820]: pgmap v3368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:26:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:49 smithi116 ceph-mon[35411]: pgmap v3368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:26:51.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:51 smithi114 ceph-mon[30820]: pgmap v3369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:26:51.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:51 smithi116 ceph-mon[35411]: pgmap v3369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:26:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:53 smithi114 ceph-mon[30820]: pgmap v3370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:26:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:53 smithi116 ceph-mon[35411]: pgmap v3370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:26:55.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:55 smithi114 ceph-mon[30820]: pgmap v3371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:55 smithi116 ceph-mon[35411]: pgmap v3371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:26:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:57 smithi114 ceph-mon[30820]: pgmap v3372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:57 smithi116 ceph-mon[35411]: pgmap v3372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:26:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:26:59 smithi116 ceph-mon[35411]: pgmap v3373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:00.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:26:59 smithi114 ceph-mon[30820]: pgmap v3373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:01 smithi116 ceph-mon[35411]: pgmap v3374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:01 smithi114 ceph-mon[30820]: pgmap v3374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:03 smithi116 ceph-mon[35411]: pgmap v3375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:03 smithi114 ceph-mon[30820]: pgmap v3375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:05 smithi114 ceph-mon[30820]: pgmap v3376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:05 smithi116 ceph-mon[35411]: pgmap v3376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:07 smithi114 ceph-mon[30820]: pgmap v3377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:27:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:07 smithi116 ceph-mon[35411]: pgmap v3377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:27:10.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:09 smithi114 ceph-mon[30820]: pgmap v3378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:09 smithi116 ceph-mon[35411]: pgmap v3378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:11.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:11 smithi114 ceph-mon[30820]: pgmap v3379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:11.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:11 smithi116 ceph-mon[35411]: pgmap v3379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:14.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:13 smithi114 ceph-mon[30820]: pgmap v3380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:13 smithi116 ceph-mon[35411]: pgmap v3380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:16.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:15 smithi114 ceph-mon[30820]: pgmap v3381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:15 smithi116 ceph-mon[35411]: pgmap v3381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:17 smithi114 ceph-mon[30820]: pgmap v3382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:27:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:17 smithi116 ceph-mon[35411]: pgmap v3382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:27:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:19 smithi114 ceph-mon[30820]: pgmap v3383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:19 smithi116 ceph-mon[35411]: pgmap v3383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:21 smithi114 ceph-mon[30820]: pgmap v3384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:21 smithi116 ceph-mon[35411]: pgmap v3384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:24.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:23 smithi114 ceph-mon[30820]: pgmap v3385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:23 smithi116 ceph-mon[35411]: pgmap v3385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:25 smithi114 ceph-mon[30820]: pgmap v3386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:25 smithi116 ceph-mon[35411]: pgmap v3386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:27 smithi114 ceph-mon[30820]: pgmap v3387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:27:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:27 smithi116 ceph-mon[35411]: pgmap v3387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:27:30.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:29 smithi114 ceph-mon[30820]: pgmap v3388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:29 smithi116 ceph-mon[35411]: pgmap v3388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:31 smithi114 ceph-mon[30820]: pgmap v3389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:31 smithi116 ceph-mon[35411]: pgmap v3389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:34.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:33 smithi114 ceph-mon[30820]: pgmap v3390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:33 smithi116 ceph-mon[35411]: pgmap v3390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:36.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:35 smithi114 ceph-mon[30820]: pgmap v3391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:35 smithi116 ceph-mon[35411]: pgmap v3391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:38.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:37 smithi114 ceph-mon[30820]: pgmap v3392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:27:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:37 smithi116 ceph-mon[35411]: pgmap v3392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:27:40.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:39 smithi114 ceph-mon[30820]: pgmap v3393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:39 smithi116 ceph-mon[35411]: pgmap v3393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:41 smithi114 ceph-mon[30820]: pgmap v3394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:41 smithi116 ceph-mon[35411]: pgmap v3394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:27:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:43 smithi114 ceph-mon[30820]: pgmap v3395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:27:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:43 smithi116 ceph-mon[35411]: pgmap v3395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:27:46.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:45 smithi114 ceph-mon[30820]: pgmap v3396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:45 smithi116 ceph-mon[35411]: pgmap v3396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:27:46.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:27:46.069+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:27:46.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:27:46.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:27:46.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:27:46.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:27:46.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:27:46.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:27:46.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:27:46.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:27:46.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:27:46.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:27:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:27:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-mon[30820]: pgmap v3397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T08:27:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-mon[30820]: pgmap v3398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:27:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:27:47.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:46 smithi116 ceph-mon[35411]: pgmap v3397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T08:27:47.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:46 smithi116 ceph-mon[35411]: pgmap v3398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:27:47.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:27:49.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:49 smithi114 ceph-mon[30820]: pgmap v3399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:27:49.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:49 smithi116 ceph-mon[35411]: pgmap v3399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:27:51.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:51 smithi114 ceph-mon[30820]: pgmap v3400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:27:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:51 smithi116 ceph-mon[35411]: pgmap v3400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:27:53.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:53 smithi114 ceph-mon[30820]: pgmap v3401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:27:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:53 smithi116 ceph-mon[35411]: pgmap v3401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:27:55.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:55 smithi114 ceph-mon[30820]: pgmap v3402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:27:55.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:55 smithi116 ceph-mon[35411]: pgmap v3402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:27:57.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:57 smithi114 ceph-mon[30820]: pgmap v3403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:27:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:57 smithi116 ceph-mon[35411]: pgmap v3403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:27:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:27:59 smithi114 ceph-mon[30820]: pgmap v3404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:27:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:27:59 smithi116 ceph-mon[35411]: pgmap v3404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:01 smithi114 ceph-mon[30820]: pgmap v3405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:01 smithi116 ceph-mon[35411]: pgmap v3405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:03.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:03 smithi114 ceph-mon[30820]: pgmap v3406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:03 smithi116 ceph-mon[35411]: pgmap v3406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:05.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:05 smithi114 ceph-mon[30820]: pgmap v3407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:05 smithi116 ceph-mon[35411]: pgmap v3407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:07.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:07 smithi114 ceph-mon[30820]: pgmap v3408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:07 smithi116 ceph-mon[35411]: pgmap v3408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:09 smithi116 ceph-mon[35411]: pgmap v3409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:09.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:09 smithi114 ceph-mon[30820]: pgmap v3409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:11 smithi116 ceph-mon[35411]: pgmap v3410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:11.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:11 smithi114 ceph-mon[30820]: pgmap v3410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:13 smithi116 ceph-mon[35411]: pgmap v3411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:13.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:13 smithi114 ceph-mon[30820]: pgmap v3411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:15.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:15 smithi114 ceph-mon[30820]: pgmap v3412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:15.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:15 smithi116 ceph-mon[35411]: pgmap v3412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:17 smithi114 ceph-mon[30820]: pgmap v3413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:17.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:17 smithi116 ceph-mon[35411]: pgmap v3413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:19 smithi114 ceph-mon[30820]: pgmap v3414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:19 smithi116 ceph-mon[35411]: pgmap v3414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:21 smithi114 ceph-mon[30820]: pgmap v3415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:21 smithi116 ceph-mon[35411]: pgmap v3415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:23.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:23 smithi114 ceph-mon[30820]: pgmap v3416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:23 smithi116 ceph-mon[35411]: pgmap v3416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:25.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:25 smithi114 ceph-mon[30820]: pgmap v3417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:25 smithi116 ceph-mon[35411]: pgmap v3417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:27.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:27 smithi114 ceph-mon[30820]: pgmap v3418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:27 smithi116 ceph-mon[35411]: pgmap v3418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:29 smithi114 ceph-mon[30820]: pgmap v3419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:29 smithi116 ceph-mon[35411]: pgmap v3419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:31.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:31 smithi114 ceph-mon[30820]: pgmap v3420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:31 smithi116 ceph-mon[35411]: pgmap v3420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:33 smithi114 ceph-mon[30820]: pgmap v3421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:33.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:33 smithi116 ceph-mon[35411]: pgmap v3421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:35.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:35 smithi114 ceph-mon[30820]: pgmap v3422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:35.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:35 smithi116 ceph-mon[35411]: pgmap v3422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:37.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:37 smithi114 ceph-mon[30820]: pgmap v3423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:37 smithi116 ceph-mon[35411]: pgmap v3423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:39 smithi114 ceph-mon[30820]: pgmap v3424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:39 smithi116 ceph-mon[35411]: pgmap v3424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:41.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:41 smithi116 ceph-mon[35411]: pgmap v3425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:41.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:41 smithi114 ceph-mon[30820]: pgmap v3425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:28:43.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:43 smithi114 ceph-mon[30820]: pgmap v3426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:43 smithi116 ceph-mon[35411]: pgmap v3426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:28:45.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:45 smithi114 ceph-mon[30820]: pgmap v3427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:45 smithi116 ceph-mon[35411]: pgmap v3427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:28:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:28:47.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:47 smithi114 ceph-mon[30820]: pgmap v3428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:47 smithi116 ceph-mon[35411]: pgmap v3428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:28:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-mon[30820]: pgmap v3429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:49.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:28:49.365+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:28:49.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:28:49.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:28:49.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:28:49.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:28:49.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:28:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:28:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:28:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:28:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:28:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:28:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:28:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:28:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:28:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:28:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:28:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:28:49.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:28:49.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:28:49.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:28:49.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:28:49.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:28:49.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:28:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:28:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:49 smithi116 ceph-mon[35411]: pgmap v3429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:50 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:28:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:28:50.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:28:50.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:28:50.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:50 smithi114 ceph-mon[30820]: pgmap v3430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T08:28:50.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:50 smithi114 ceph-mon[30820]: pgmap v3431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:28:50.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:28:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:50 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:28:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:28:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:28:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:50 smithi116 ceph-mon[35411]: pgmap v3430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T08:28:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:50 smithi116 ceph-mon[35411]: pgmap v3431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:28:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:28:52.697 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:52 smithi116 ceph-mon[35411]: pgmap v3432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:28:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:52 smithi114 ceph-mon[30820]: pgmap v3432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:28:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:54 smithi114 ceph-mon[30820]: pgmap v3433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:28:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:54 smithi116 ceph-mon[35411]: pgmap v3433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:28:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:56 smithi114 ceph-mon[30820]: pgmap v3434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:28:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:56 smithi116 ceph-mon[35411]: pgmap v3434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:28:58.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:28:58 smithi114 ceph-mon[30820]: pgmap v3435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:28:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:28:58 smithi116 ceph-mon[35411]: pgmap v3435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:29:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:00 smithi114 ceph-mon[30820]: pgmap v3436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:29:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:00 smithi116 ceph-mon[35411]: pgmap v3436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:29:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:01 smithi114 ceph-mon[30820]: pgmap v3437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:01 smithi116 ceph-mon[35411]: pgmap v3437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:04.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:04 smithi114 ceph-mon[30820]: pgmap v3438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:29:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:04 smithi116 ceph-mon[35411]: pgmap v3438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:29:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:06 smithi114 ceph-mon[30820]: pgmap v3439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:29:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:06 smithi116 ceph-mon[35411]: pgmap v3439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:29:08.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:08 smithi114 ceph-mon[30820]: pgmap v3440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:29:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:08 smithi116 ceph-mon[35411]: pgmap v3440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:29:10.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:10 smithi114 ceph-mon[30820]: pgmap v3441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:29:10.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:10 smithi116 ceph-mon[35411]: pgmap v3441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:29:12.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:12 smithi114 ceph-mon[30820]: pgmap v3442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:12.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:12 smithi116 ceph-mon[35411]: pgmap v3442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:14 smithi114 ceph-mon[30820]: pgmap v3443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:29:14.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:14 smithi116 ceph-mon[35411]: pgmap v3443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:29:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:16 smithi114 ceph-mon[30820]: pgmap v3444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:29:16.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:16 smithi116 ceph-mon[35411]: pgmap v3444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:29:18.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:18 smithi114 ceph-mon[30820]: pgmap v3445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:29:18.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:18 smithi116 ceph-mon[35411]: pgmap v3445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:29:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:20 smithi114 ceph-mon[30820]: pgmap v3446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:29:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:20 smithi116 ceph-mon[35411]: pgmap v3446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:29:22.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:22 smithi114 ceph-mon[30820]: pgmap v3447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:22 smithi116 ceph-mon[35411]: pgmap v3447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:24 smithi114 ceph-mon[30820]: pgmap v3448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:29:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:24 smithi116 ceph-mon[35411]: pgmap v3448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:29:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:26 smithi114 ceph-mon[30820]: pgmap v3449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:29:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:26 smithi116 ceph-mon[35411]: pgmap v3449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:29:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:28 smithi114 ceph-mon[30820]: pgmap v3450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T08:29:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:28 smithi116 ceph-mon[35411]: pgmap v3450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T08:29:30.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:30 smithi114 ceph-mon[30820]: pgmap v3451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:29:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:30 smithi116 ceph-mon[35411]: pgmap v3451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:29:32.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:32 smithi114 ceph-mon[30820]: pgmap v3452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:32 smithi116 ceph-mon[35411]: pgmap v3452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:33 smithi114 ceph-mon[30820]: pgmap v3453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:29:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:33 smithi116 ceph-mon[35411]: pgmap v3453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:29:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:36 smithi114 ceph-mon[30820]: pgmap v3454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:29:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:36 smithi116 ceph-mon[35411]: pgmap v3454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:29:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:38 smithi114 ceph-mon[30820]: pgmap v3455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:29:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:38 smithi116 ceph-mon[35411]: pgmap v3455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:29:40.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:40 smithi114 ceph-mon[30820]: pgmap v3456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:29:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:40 smithi116 ceph-mon[35411]: pgmap v3456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:29:42.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:42 smithi114 ceph-mon[30820]: pgmap v3457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:42.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:42 smithi116 ceph-mon[35411]: pgmap v3457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:44 smithi114 ceph-mon[30820]: pgmap v3458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:29:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:44 smithi116 ceph-mon[35411]: pgmap v3458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:29:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:46 smithi114 ceph-mon[30820]: pgmap v3459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:29:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:46 smithi116 ceph-mon[35411]: pgmap v3459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:29:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:48 smithi114 ceph-mon[30820]: pgmap v3460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:29:48.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:48 smithi116 ceph-mon[35411]: pgmap v3460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:29:50.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:50 smithi114 ceph-mon[30820]: pgmap v3461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:29:50.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:29:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:50 smithi116 ceph-mon[35411]: pgmap v3461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:29:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:29:52.725 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-mon[30820]: pgmap v3462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:52 smithi116 ceph-mon[35411]: pgmap v3462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:29:53.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:29:52.724+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:29:53.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:29:53.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:29:53.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:29:53.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:29:53.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:29:53.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:29:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:29:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:29:53.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:29:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:29:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:29:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:29:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:29:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:29:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:29:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:29:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:29:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:54 smithi114 ceph-mon[30820]: pgmap v3463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:54 smithi114 ceph-mon[30820]: pgmap v3464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:29:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:54 smithi116 ceph-mon[35411]: pgmap v3463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:54 smithi116 ceph-mon[35411]: pgmap v3464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:29:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:56 smithi114 ceph-mon[30820]: pgmap v3465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:29:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:56 smithi116 ceph-mon[35411]: pgmap v3465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:29:58.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:29:58 smithi114 ceph-mon[30820]: pgmap v3466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:29:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:29:58 smithi116 ceph-mon[35411]: pgmap v3466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:30:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:00 smithi114 ceph-mon[30820]: pgmap v3467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:30:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:30:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:00 smithi116 ceph-mon[35411]: pgmap v3467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:30:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:30:02.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:02 smithi114 ceph-mon[30820]: pgmap v3468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:30:02.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:02 smithi116 ceph-mon[35411]: pgmap v3468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:30:03.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:03 smithi114 ceph-mon[30820]: pgmap v3469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:30:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:03 smithi116 ceph-mon[35411]: pgmap v3469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:30:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:05 smithi114 ceph-mon[30820]: pgmap v3470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:05 smithi116 ceph-mon[35411]: pgmap v3470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:07 smithi114 ceph-mon[30820]: pgmap v3471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:07 smithi116 ceph-mon[35411]: pgmap v3471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:10.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:09 smithi114 ceph-mon[30820]: pgmap v3472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:09 smithi116 ceph-mon[35411]: pgmap v3472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:11 smithi114 ceph-mon[30820]: pgmap v3473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:11 smithi116 ceph-mon[35411]: pgmap v3473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:14.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:13 smithi114 ceph-mon[30820]: pgmap v3474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:13 smithi116 ceph-mon[35411]: pgmap v3474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:16.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:15 smithi114 ceph-mon[30820]: pgmap v3475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:15 smithi116 ceph-mon[35411]: pgmap v3475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:18.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:17 smithi114 ceph-mon[30820]: pgmap v3476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:18.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:17 smithi116 ceph-mon[35411]: pgmap v3476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:19 smithi114 ceph-mon[30820]: pgmap v3477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:20.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:19 smithi116 ceph-mon[35411]: pgmap v3477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:22.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:21 smithi114 ceph-mon[30820]: pgmap v3478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:21 smithi116 ceph-mon[35411]: pgmap v3478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:24.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:23 smithi114 ceph-mon[30820]: pgmap v3479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:23 smithi116 ceph-mon[35411]: pgmap v3479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:26.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:25 smithi114 ceph-mon[30820]: pgmap v3480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:25 smithi116 ceph-mon[35411]: pgmap v3480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:27 smithi114 ceph-mon[30820]: pgmap v3481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:27 smithi116 ceph-mon[35411]: pgmap v3481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:29 smithi114 ceph-mon[30820]: pgmap v3482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:29 smithi116 ceph-mon[35411]: pgmap v3482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:31 smithi114 ceph-mon[30820]: pgmap v3483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:31 smithi116 ceph-mon[35411]: pgmap v3483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:34.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:33 smithi114 ceph-mon[30820]: pgmap v3484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:33 smithi116 ceph-mon[35411]: pgmap v3484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:36.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:35 smithi114 ceph-mon[30820]: pgmap v3485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:35 smithi116 ceph-mon[35411]: pgmap v3485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:38.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:37 smithi114 ceph-mon[30820]: pgmap v3486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:37 smithi116 ceph-mon[35411]: pgmap v3486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:40.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:39 smithi114 ceph-mon[30820]: pgmap v3487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:39 smithi116 ceph-mon[35411]: pgmap v3487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:41 smithi114 ceph-mon[30820]: pgmap v3488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:41 smithi116 ceph-mon[35411]: pgmap v3488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:43 smithi114 ceph-mon[30820]: pgmap v3489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:43 smithi116 ceph-mon[35411]: pgmap v3489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:45 smithi114 ceph-mon[30820]: pgmap v3490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:45 smithi116 ceph-mon[35411]: pgmap v3490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:48.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:47 smithi114 ceph-mon[30820]: pgmap v3491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:47 smithi116 ceph-mon[35411]: pgmap v3491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:30:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:49 smithi114 ceph-mon[30820]: pgmap v3492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:50.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:49 smithi116 ceph-mon[35411]: pgmap v3492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:51 smithi114 ceph-mon[30820]: pgmap v3493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:51 smithi116 ceph-mon[35411]: pgmap v3493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:30:53.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:30:53.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:30:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:53 smithi114 ceph-mon[30820]: pgmap v3494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:53 smithi116 ceph-mon[35411]: pgmap v3494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:30:56.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:55 smithi114 ceph-mon[30820]: pgmap v3495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:30:56.085+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:30:56.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:30:56.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:30:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:55 smithi116 ceph-mon[35411]: pgmap v3495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:30:57.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:30:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:30:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:30:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:30:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-mon[30820]: pgmap v3496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-mon[30820]: pgmap v3497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:30:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:30:57.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:56 smithi116 ceph-mon[35411]: pgmap v3496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:56 smithi116 ceph-mon[35411]: pgmap v3497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:30:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:30:59.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:30:59 smithi114 ceph-mon[30820]: pgmap v3498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:30:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:30:59 smithi116 ceph-mon[35411]: pgmap v3498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:31:01.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:01 smithi114 ceph-mon[30820]: pgmap v3499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:31:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:01 smithi116 ceph-mon[35411]: pgmap v3499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:31:03.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:03 smithi114 ceph-mon[30820]: pgmap v3500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:31:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:03 smithi116 ceph-mon[35411]: pgmap v3500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:31:05.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:05 smithi114 ceph-mon[30820]: pgmap v3501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:31:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:05 smithi116 ceph-mon[35411]: pgmap v3501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:31:07.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:07 smithi114 ceph-mon[30820]: pgmap v3502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:31:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:07 smithi116 ceph-mon[35411]: pgmap v3502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:31:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:09 smithi114 ceph-mon[30820]: pgmap v3503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:09 smithi116 ceph-mon[35411]: pgmap v3503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:11 smithi116 ceph-mon[35411]: pgmap v3504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:11.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:11 smithi114 ceph-mon[30820]: pgmap v3504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:13 smithi116 ceph-mon[35411]: pgmap v3505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:13.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:13 smithi114 ceph-mon[30820]: pgmap v3505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:15 smithi116 ceph-mon[35411]: pgmap v3506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:15.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:15 smithi114 ceph-mon[30820]: pgmap v3506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:17 smithi116 ceph-mon[35411]: pgmap v3507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:17.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:17 smithi114 ceph-mon[30820]: pgmap v3507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:19 smithi114 ceph-mon[30820]: pgmap v3508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:19 smithi116 ceph-mon[35411]: pgmap v3508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:21 smithi114 ceph-mon[30820]: pgmap v3509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:21 smithi116 ceph-mon[35411]: pgmap v3509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:23 smithi114 ceph-mon[30820]: pgmap v3510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:23 smithi116 ceph-mon[35411]: pgmap v3510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:25.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:25 smithi114 ceph-mon[30820]: pgmap v3511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:25 smithi116 ceph-mon[35411]: pgmap v3511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:27.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:27 smithi114 ceph-mon[30820]: pgmap v3512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:27.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:27 smithi116 ceph-mon[35411]: pgmap v3512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:29 smithi114 ceph-mon[30820]: pgmap v3513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:29 smithi116 ceph-mon[35411]: pgmap v3513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:31 smithi114 ceph-mon[30820]: pgmap v3514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:31 smithi116 ceph-mon[35411]: pgmap v3514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:33 smithi114 ceph-mon[30820]: pgmap v3515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:33.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:33 smithi116 ceph-mon[35411]: pgmap v3515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:35.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:35 smithi114 ceph-mon[30820]: pgmap v3516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:35.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:35 smithi116 ceph-mon[35411]: pgmap v3516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:37.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:37 smithi114 ceph-mon[30820]: pgmap v3517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:37 smithi116 ceph-mon[35411]: pgmap v3517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:39.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:39 smithi114 ceph-mon[30820]: pgmap v3518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:39 smithi116 ceph-mon[35411]: pgmap v3518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:41.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:41 smithi114 ceph-mon[30820]: pgmap v3519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:41 smithi116 ceph-mon[35411]: pgmap v3519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:43.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:43 smithi116 ceph-mon[35411]: pgmap v3520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:43.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:43 smithi114 ceph-mon[30820]: pgmap v3520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:45.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:45 smithi114 ceph-mon[30820]: pgmap v3521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:45 smithi116 ceph-mon[35411]: pgmap v3521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:47.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:47 smithi114 ceph-mon[30820]: pgmap v3522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:47 smithi116 ceph-mon[35411]: pgmap v3522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:49.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:49 smithi114 ceph-mon[30820]: pgmap v3523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:49 smithi116 ceph-mon[35411]: pgmap v3523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:51 smithi114 ceph-mon[30820]: pgmap v3524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:51.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:51 smithi116 ceph-mon[35411]: pgmap v3524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:31:53.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:53 smithi114 ceph-mon[30820]: pgmap v3525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:53.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:53 smithi116 ceph-mon[35411]: pgmap v3525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:31:55.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:55 smithi114 ceph-mon[30820]: pgmap v3526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:55.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:55 smithi116 ceph-mon[35411]: pgmap v3526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:31:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:31:57.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:57 smithi114 ceph-mon[30820]: pgmap v3527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:57.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:57 smithi116 ceph-mon[35411]: pgmap v3527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:31:59.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-mon[30820]: pgmap v3528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:31:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:31:59.444+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:31:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:31:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:31:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:31:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:31:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:31:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:31:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:31:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:31:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:31:59.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:31:59 smithi116 ceph-mon[35411]: pgmap v3528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:00 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:32:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:32:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:32:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:32:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:00 smithi114 ceph-mon[30820]: pgmap v3529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:00 smithi114 ceph-mon[30820]: pgmap v3530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:00 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:32:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:32:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:32:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:32:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:32:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:32:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:32:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:00 smithi116 ceph-mon[35411]: pgmap v3529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:32:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:00 smithi116 ceph-mon[35411]: pgmap v3530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:32:02.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:02 smithi114 ceph-mon[30820]: pgmap v3531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:02.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:02 smithi116 ceph-mon[35411]: pgmap v3531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:04.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:04 smithi114 ceph-mon[30820]: pgmap v3532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:04 smithi116 ceph-mon[35411]: pgmap v3532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:06.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:06 smithi114 ceph-mon[30820]: pgmap v3533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:06 smithi116 ceph-mon[35411]: pgmap v3533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:08.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:08 smithi114 ceph-mon[30820]: pgmap v3534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:08 smithi116 ceph-mon[35411]: pgmap v3534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:32:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:09 smithi114 ceph-mon[30820]: pgmap v3535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:32:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:09 smithi116 ceph-mon[35411]: pgmap v3535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:32:12.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:12 smithi114 ceph-mon[30820]: pgmap v3536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:12.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:12 smithi116 ceph-mon[35411]: pgmap v3536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:14 smithi114 ceph-mon[30820]: pgmap v3537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:14 smithi116 ceph-mon[35411]: pgmap v3537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:16.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:16 smithi114 ceph-mon[30820]: pgmap v3538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:16.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:16 smithi116 ceph-mon[35411]: pgmap v3538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:17 smithi114 ceph-mon[30820]: pgmap v3539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:17 smithi116 ceph-mon[35411]: pgmap v3539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:20 smithi114 ceph-mon[30820]: pgmap v3540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:32:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:20 smithi116 ceph-mon[35411]: pgmap v3540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:32:22.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:22 smithi114 ceph-mon[30820]: pgmap v3541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:22 smithi116 ceph-mon[35411]: pgmap v3541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:24.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:24 smithi114 ceph-mon[30820]: pgmap v3542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:24 smithi116 ceph-mon[35411]: pgmap v3542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:25.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:25 smithi114 ceph-mon[30820]: pgmap v3543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:25 smithi116 ceph-mon[35411]: pgmap v3543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:28 smithi114 ceph-mon[30820]: pgmap v3544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:28 smithi116 ceph-mon[35411]: pgmap v3544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:30.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:30 smithi114 ceph-mon[30820]: pgmap v3545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:32:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:30 smithi116 ceph-mon[35411]: pgmap v3545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:32:31.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:31 smithi114 ceph-mon[30820]: pgmap v3546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:31 smithi116 ceph-mon[35411]: pgmap v3546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:34.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:34 smithi114 ceph-mon[30820]: pgmap v3547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:34.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:34 smithi116 ceph-mon[35411]: pgmap v3547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:36.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:36 smithi114 ceph-mon[30820]: pgmap v3548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:36 smithi116 ceph-mon[35411]: pgmap v3548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:37.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:37 smithi114 ceph-mon[30820]: pgmap v3549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:37 smithi116 ceph-mon[35411]: pgmap v3549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:40.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:40 smithi114 ceph-mon[30820]: pgmap v3550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:32:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:40 smithi116 ceph-mon[35411]: pgmap v3550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:32:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:41 smithi114 ceph-mon[30820]: pgmap v3551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:41 smithi116 ceph-mon[35411]: pgmap v3551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:44 smithi114 ceph-mon[30820]: pgmap v3552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:44 smithi116 ceph-mon[35411]: pgmap v3552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:45 smithi114 ceph-mon[30820]: pgmap v3553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:45 smithi116 ceph-mon[35411]: pgmap v3553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:48 smithi114 ceph-mon[30820]: pgmap v3554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:48.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:48 smithi116 ceph-mon[35411]: pgmap v3554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:32:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:49 smithi114 ceph-mon[30820]: pgmap v3555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:32:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:49 smithi116 ceph-mon[35411]: pgmap v3555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:32:52.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:52 smithi114 ceph-mon[30820]: pgmap v3556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:52 smithi116 ceph-mon[35411]: pgmap v3556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:32:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:53 smithi114 ceph-mon[30820]: pgmap v3557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:32:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:53 smithi116 ceph-mon[35411]: pgmap v3557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:32:55.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:55 smithi114 ceph-mon[30820]: pgmap v3558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:55 smithi116 ceph-mon[35411]: pgmap v3558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:32:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:57 smithi114 ceph-mon[30820]: pgmap v3559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:32:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:57 smithi116 ceph-mon[35411]: pgmap v3559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:32:59.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:32:59 smithi114 ceph-mon[30820]: pgmap v3560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 5 op/s 2024-09-19T08:32:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:32:59 smithi116 ceph-mon[35411]: pgmap v3560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 5 op/s 2024-09-19T08:33:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:33:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:00 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:33:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:33:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:33:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:00 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:33:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:01 smithi114 ceph-mon[30820]: pgmap v3561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 5 op/s 2024-09-19T08:33:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:01 smithi116 ceph-mon[35411]: pgmap v3561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 5 op/s 2024-09-19T08:33:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:33:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:33:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:33:03.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:33:02.804+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:33:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:33:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:33:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:33:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:33:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:33:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:33:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:33:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:33:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:33:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:33:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:33:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:33:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:33:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:33:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:33:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:33:03.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:33:03.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:33:03.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:33:03.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:33:03.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:33:03.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:33:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:33:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:33:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:33:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:33:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:33:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:33:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:33:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:33:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:33:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:33:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:33:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:33:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:33:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:33:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:33:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:33:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:33:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:33:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:33:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:03 smithi114 ceph-mon[30820]: pgmap v3562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:33:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:03 smithi114 ceph-mon[30820]: pgmap v3563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:33:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:33:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:33:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:33:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:33:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:33:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:33:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:33:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:33:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:33:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:33:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:33:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:33:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:33:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:33:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:33:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:03 smithi116 ceph-mon[35411]: pgmap v3562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:33:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:03 smithi116 ceph-mon[35411]: pgmap v3563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:33:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:05 smithi114 ceph-mon[30820]: pgmap v3564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:33:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:05 smithi116 ceph-mon[35411]: pgmap v3564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:33:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:07 smithi114 ceph-mon[30820]: pgmap v3565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:33:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:07 smithi116 ceph-mon[35411]: pgmap v3565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:33:10.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:09 smithi114 ceph-mon[30820]: pgmap v3566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:33:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:09 smithi116 ceph-mon[35411]: pgmap v3566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:33:12.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:11 smithi114 ceph-mon[30820]: pgmap v3567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:33:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:11 smithi116 ceph-mon[35411]: pgmap v3567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:33:14.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:13 smithi114 ceph-mon[30820]: pgmap v3568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:33:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:13 smithi116 ceph-mon[35411]: pgmap v3568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:33:16.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:15 smithi114 ceph-mon[30820]: pgmap v3569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:15 smithi116 ceph-mon[35411]: pgmap v3569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:18.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:17 smithi114 ceph-mon[30820]: pgmap v3570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:18.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:17 smithi116 ceph-mon[35411]: pgmap v3570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:19 smithi114 ceph-mon[30820]: pgmap v3571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:19 smithi116 ceph-mon[35411]: pgmap v3571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:22.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:21 smithi114 ceph-mon[30820]: pgmap v3572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:33:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:21 smithi116 ceph-mon[35411]: pgmap v3572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:33:24.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:23 smithi114 ceph-mon[30820]: pgmap v3573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:23 smithi116 ceph-mon[35411]: pgmap v3573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:25 smithi116 ceph-mon[35411]: pgmap v3574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:26.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:25 smithi114 ceph-mon[30820]: pgmap v3574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:27 smithi116 ceph-mon[35411]: pgmap v3575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:28.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:27 smithi114 ceph-mon[30820]: pgmap v3575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:29 smithi116 ceph-mon[35411]: pgmap v3576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:30.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:29 smithi114 ceph-mon[30820]: pgmap v3576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:32.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:31 smithi114 ceph-mon[30820]: pgmap v3577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:33:32.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:31 smithi116 ceph-mon[35411]: pgmap v3577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:33:34.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:33 smithi114 ceph-mon[30820]: pgmap v3578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:34.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:33 smithi116 ceph-mon[35411]: pgmap v3578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:36.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:35 smithi114 ceph-mon[30820]: pgmap v3579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:36.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:35 smithi116 ceph-mon[35411]: pgmap v3579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:38.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:37 smithi114 ceph-mon[30820]: pgmap v3580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:38.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:37 smithi116 ceph-mon[35411]: pgmap v3580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:40.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:39 smithi114 ceph-mon[30820]: pgmap v3581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:40.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:39 smithi116 ceph-mon[35411]: pgmap v3581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:42.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:41 smithi114 ceph-mon[30820]: pgmap v3582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:33:42.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:41 smithi116 ceph-mon[35411]: pgmap v3582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:33:44.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:43 smithi114 ceph-mon[30820]: pgmap v3583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:44.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:43 smithi116 ceph-mon[35411]: pgmap v3583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:46.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:45 smithi114 ceph-mon[30820]: pgmap v3584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:46.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:45 smithi116 ceph-mon[35411]: pgmap v3584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:47 smithi114 ceph-mon[30820]: pgmap v3585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:48.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:47 smithi116 ceph-mon[35411]: pgmap v3585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:50.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:49 smithi114 ceph-mon[30820]: pgmap v3586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:50.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:49 smithi116 ceph-mon[35411]: pgmap v3586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:52.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:51 smithi114 ceph-mon[30820]: pgmap v3587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:33:52.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:51 smithi116 ceph-mon[35411]: pgmap v3587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:33:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:53 smithi114 ceph-mon[30820]: pgmap v3588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:54.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:53 smithi116 ceph-mon[35411]: pgmap v3588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:33:56.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:55 smithi114 ceph-mon[30820]: pgmap v3589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:55 smithi116 ceph-mon[35411]: pgmap v3589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:33:58.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:57 smithi114 ceph-mon[30820]: pgmap v3590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:33:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:57 smithi116 ceph-mon[35411]: pgmap v3590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:34:00.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:33:59 smithi114 ceph-mon[30820]: pgmap v3591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:33:59 smithi116 ceph-mon[35411]: pgmap v3591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:02.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:01 smithi114 ceph-mon[30820]: pgmap v3592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:34:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:01 smithi116 ceph-mon[35411]: pgmap v3592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:34:03.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:34:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:34:04.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:03 smithi114 ceph-mon[30820]: pgmap v3593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:04.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:03 smithi116 ceph-mon[35411]: pgmap v3593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:06.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:05 smithi114 ceph-mon[30820]: pgmap v3594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:34:06.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:34:06.164+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:34:06.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:34:06.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:34:06.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:34:06.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:34:06.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:34:06.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:34:06.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:34:06.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:34:06.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:34:06.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:34:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:05 smithi116 ceph-mon[35411]: pgmap v3594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:34:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:34:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-mon[30820]: pgmap v3595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:34:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-mon[30820]: pgmap v3596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:34:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:34:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:06 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:34:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:06 smithi116 ceph-mon[35411]: pgmap v3595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:06 smithi116 ceph-mon[35411]: pgmap v3596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:34:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:34:09.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:09 smithi114 ceph-mon[30820]: pgmap v3597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:34:09.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:09 smithi116 ceph-mon[35411]: pgmap v3597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:34:11.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:11 smithi114 ceph-mon[30820]: pgmap v3598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:34:11.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:11 smithi116 ceph-mon[35411]: pgmap v3598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:34:13.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:13 smithi114 ceph-mon[30820]: pgmap v3599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:34:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:13 smithi116 ceph-mon[35411]: pgmap v3599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:34:15.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:15 smithi114 ceph-mon[30820]: pgmap v3600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:34:15.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:15 smithi116 ceph-mon[35411]: pgmap v3600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:34:17.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:17 smithi114 ceph-mon[30820]: pgmap v3601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:34:17.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:17 smithi116 ceph-mon[35411]: pgmap v3601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:34:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:19 smithi114 ceph-mon[30820]: pgmap v3602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:19 smithi116 ceph-mon[35411]: pgmap v3602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:21 smithi114 ceph-mon[30820]: pgmap v3603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:34:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:21 smithi116 ceph-mon[35411]: pgmap v3603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:34:23.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:23 smithi114 ceph-mon[30820]: pgmap v3604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:34:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:23 smithi116 ceph-mon[35411]: pgmap v3604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:34:25.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:25 smithi114 ceph-mon[30820]: pgmap v3605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:25 smithi116 ceph-mon[35411]: pgmap v3605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:27 smithi114 ceph-mon[30820]: pgmap v3606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:34:27.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:27 smithi116 ceph-mon[35411]: pgmap v3606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:34:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:29 smithi114 ceph-mon[30820]: pgmap v3607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:29 smithi116 ceph-mon[35411]: pgmap v3607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:31.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:31 smithi114 ceph-mon[30820]: pgmap v3608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:34:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:31 smithi116 ceph-mon[35411]: pgmap v3608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:34:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:33 smithi114 ceph-mon[30820]: pgmap v3609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:34:33.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:33 smithi116 ceph-mon[35411]: pgmap v3609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:34:35.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:35 smithi114 ceph-mon[30820]: pgmap v3610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:34:35.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:35 smithi116 ceph-mon[35411]: pgmap v3610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:34:37.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:37 smithi114 ceph-mon[30820]: pgmap v3611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:34:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:37 smithi116 ceph-mon[35411]: pgmap v3611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:34:39.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:39 smithi114 ceph-mon[30820]: pgmap v3612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:39 smithi116 ceph-mon[35411]: pgmap v3612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:41.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:41 smithi114 ceph-mon[30820]: pgmap v3613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:34:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:41 smithi116 ceph-mon[35411]: pgmap v3613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:34:43.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:43 smithi114 ceph-mon[30820]: pgmap v3614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:34:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:43 smithi116 ceph-mon[35411]: pgmap v3614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:34:45.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:45 smithi114 ceph-mon[30820]: pgmap v3615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:34:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:45 smithi116 ceph-mon[35411]: pgmap v3615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:34:47.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:47 smithi114 ceph-mon[30820]: pgmap v3616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:34:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:47 smithi116 ceph-mon[35411]: pgmap v3616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:34:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:49 smithi114 ceph-mon[30820]: pgmap v3617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:34:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:49 smithi116 ceph-mon[35411]: pgmap v3617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:34:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:51 smithi114 ceph-mon[30820]: pgmap v3618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:34:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:51 smithi116 ceph-mon[35411]: pgmap v3618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:34:53.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:53 smithi114 ceph-mon[30820]: pgmap v3619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T08:34:53.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:53 smithi116 ceph-mon[35411]: pgmap v3619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T08:34:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:55 smithi114 ceph-mon[30820]: pgmap v3620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:34:55.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:55 smithi116 ceph-mon[35411]: pgmap v3620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:34:57.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:57 smithi114 ceph-mon[30820]: pgmap v3621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:34:57.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:57 smithi116 ceph-mon[35411]: pgmap v3621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:34:59.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:34:59 smithi114 ceph-mon[30820]: pgmap v3622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:34:59.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:34:59 smithi116 ceph-mon[35411]: pgmap v3622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:01.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:01 smithi114 ceph-mon[30820]: pgmap v3623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:35:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:01 smithi116 ceph-mon[35411]: pgmap v3623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:35:03.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:03 smithi114 ceph-mon[30820]: pgmap v3624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:35:03.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:03 smithi116 ceph-mon[35411]: pgmap v3624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:35:05.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:05 smithi114 ceph-mon[30820]: pgmap v3625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:05.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:05 smithi116 ceph-mon[35411]: pgmap v3625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:06.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:35:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:35:07.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:07 smithi114 ceph-mon[30820]: pgmap v3626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:07 smithi116 ceph-mon[35411]: pgmap v3626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:09.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-mon[30820]: pgmap v3627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:09.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:35:09.524+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:35:09.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:35:09.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:35:09.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:35:09.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:35:09.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:35:09.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:35:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:35:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:35:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:35:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:35:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:35:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:35:09.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:09 smithi116 ceph-mon[35411]: pgmap v3627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:10 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:35:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:35:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:35:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:35:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:10 smithi114 ceph-mon[30820]: pgmap v3628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:35:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:10 smithi114 ceph-mon[30820]: pgmap v3629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:35:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:10 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:10 smithi116 ceph-mon[35411]: pgmap v3628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:35:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:10 smithi116 ceph-mon[35411]: pgmap v3629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:35:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:35:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:11 smithi114 ceph-mon[30820]: pgmap v3630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:35:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:11 smithi116 ceph-mon[35411]: pgmap v3630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:35:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:13 smithi114 ceph-mon[30820]: pgmap v3631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:35:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:13 smithi116 ceph-mon[35411]: pgmap v3631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:35:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:15 smithi114 ceph-mon[30820]: pgmap v3632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:35:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:15 smithi116 ceph-mon[35411]: pgmap v3632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:35:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:17 smithi114 ceph-mon[30820]: pgmap v3633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:35:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:17 smithi116 ceph-mon[35411]: pgmap v3633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:35:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:19 smithi114 ceph-mon[30820]: pgmap v3634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:35:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:19 smithi116 ceph-mon[35411]: pgmap v3634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:35:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:21 smithi114 ceph-mon[30820]: pgmap v3635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:21 smithi116 ceph-mon[35411]: pgmap v3635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:23 smithi114 ceph-mon[30820]: pgmap v3636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:23 smithi116 ceph-mon[35411]: pgmap v3636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:25.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:25 smithi114 ceph-mon[30820]: pgmap v3637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:35:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:25 smithi116 ceph-mon[35411]: pgmap v3637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:35:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:27 smithi114 ceph-mon[30820]: pgmap v3638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:27 smithi116 ceph-mon[35411]: pgmap v3638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:29 smithi114 ceph-mon[30820]: pgmap v3639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:29 smithi116 ceph-mon[35411]: pgmap v3639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:31 smithi114 ceph-mon[30820]: pgmap v3640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:31 smithi116 ceph-mon[35411]: pgmap v3640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:33 smithi114 ceph-mon[30820]: pgmap v3641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:33 smithi116 ceph-mon[35411]: pgmap v3641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:35.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:35 smithi114 ceph-mon[30820]: pgmap v3642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:35:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:35 smithi116 ceph-mon[35411]: pgmap v3642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:35:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:37 smithi114 ceph-mon[30820]: pgmap v3643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:37 smithi116 ceph-mon[35411]: pgmap v3643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:39 smithi114 ceph-mon[30820]: pgmap v3644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:39 smithi116 ceph-mon[35411]: pgmap v3644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:41 smithi114 ceph-mon[30820]: pgmap v3645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:41 smithi116 ceph-mon[35411]: pgmap v3645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:43.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:43 smithi114 ceph-mon[30820]: pgmap v3646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:43 smithi116 ceph-mon[35411]: pgmap v3646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:45 smithi114 ceph-mon[30820]: pgmap v3647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:35:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:45 smithi116 ceph-mon[35411]: pgmap v3647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:35:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:47 smithi114 ceph-mon[30820]: pgmap v3648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:47 smithi116 ceph-mon[35411]: pgmap v3648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:49 smithi114 ceph-mon[30820]: pgmap v3649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:49 smithi116 ceph-mon[35411]: pgmap v3649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:51.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:51 smithi114 ceph-mon[30820]: pgmap v3650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:51.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:51 smithi116 ceph-mon[35411]: pgmap v3650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:35:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:53 smithi114 ceph-mon[30820]: pgmap v3651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:53 smithi116 ceph-mon[35411]: pgmap v3651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:55 smithi114 ceph-mon[30820]: pgmap v3652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:35:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:55 smithi116 ceph-mon[35411]: pgmap v3652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:35:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:57 smithi114 ceph-mon[30820]: pgmap v3653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:57 smithi116 ceph-mon[35411]: pgmap v3653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:35:59.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:35:59 smithi114 ceph-mon[30820]: pgmap v3654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:35:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:35:59 smithi116 ceph-mon[35411]: pgmap v3654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:01.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:01 smithi114 ceph-mon[30820]: pgmap v3655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:01 smithi116 ceph-mon[35411]: pgmap v3655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:03 smithi114 ceph-mon[30820]: pgmap v3656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:03 smithi116 ceph-mon[35411]: pgmap v3656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:05 smithi114 ceph-mon[30820]: pgmap v3657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:36:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:05 smithi116 ceph-mon[35411]: pgmap v3657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:36:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:07 smithi114 ceph-mon[30820]: pgmap v3658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:07 smithi116 ceph-mon[35411]: pgmap v3658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:09 smithi114 ceph-mon[30820]: pgmap v3659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:09 smithi116 ceph-mon[35411]: pgmap v3659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:10.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:36:10.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:36:12.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-mon[30820]: pgmap v3660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:12 smithi116 ceph-mon[35411]: pgmap v3660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:13.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:36:12.885+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:36:13.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:36:13.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:36:13.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:36:13.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:36:13.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:36:13.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:36:13.164 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:36:13.165 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:36:13.165 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:36:13.165 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:36:13.165 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:36:13.165 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:36:13.165 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:36:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:36:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:36:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:36:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:36:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:36:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:36:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:36:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:14 smithi116 ceph-mon[35411]: pgmap v3661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:36:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:14 smithi116 ceph-mon[35411]: pgmap v3662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:36:14.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:36:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:36:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:36:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:36:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:14 smithi114 ceph-mon[30820]: pgmap v3661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:36:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:14 smithi114 ceph-mon[30820]: pgmap v3662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:36:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:16 smithi116 ceph-mon[35411]: pgmap v3663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:36:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:16 smithi114 ceph-mon[30820]: pgmap v3663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:36:18.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:18 smithi114 ceph-mon[30820]: pgmap v3664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:36:18.509 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:18 smithi116 ceph-mon[35411]: pgmap v3664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:36:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:20 smithi114 ceph-mon[30820]: pgmap v3665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:36:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:20 smithi116 ceph-mon[35411]: pgmap v3665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:36:22.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:22 smithi114 ceph-mon[30820]: pgmap v3666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:36:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:22 smithi116 ceph-mon[35411]: pgmap v3666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:36:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:24 smithi114 ceph-mon[30820]: pgmap v3667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:36:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:24 smithi116 ceph-mon[35411]: pgmap v3667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:36:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:26 smithi114 ceph-mon[30820]: pgmap v3668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:26 smithi116 ceph-mon[35411]: pgmap v3668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:28 smithi114 ceph-mon[30820]: pgmap v3669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:28 smithi116 ceph-mon[35411]: pgmap v3669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:30 smithi114 ceph-mon[30820]: pgmap v3670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:30 smithi116 ceph-mon[35411]: pgmap v3670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:32.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:32 smithi114 ceph-mon[30820]: pgmap v3671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:36:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:32 smithi116 ceph-mon[35411]: pgmap v3671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:36:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:34 smithi114 ceph-mon[30820]: pgmap v3672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:34 smithi116 ceph-mon[35411]: pgmap v3672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:36 smithi114 ceph-mon[30820]: pgmap v3673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:36 smithi116 ceph-mon[35411]: pgmap v3673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:38 smithi114 ceph-mon[30820]: pgmap v3674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:38 smithi116 ceph-mon[35411]: pgmap v3674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:40.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:40 smithi114 ceph-mon[30820]: pgmap v3675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:40 smithi116 ceph-mon[35411]: pgmap v3675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:42 smithi114 ceph-mon[30820]: pgmap v3676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:36:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:42 smithi116 ceph-mon[35411]: pgmap v3676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:36:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:44 smithi114 ceph-mon[30820]: pgmap v3677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:44 smithi116 ceph-mon[35411]: pgmap v3677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:46 smithi114 ceph-mon[30820]: pgmap v3678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:46 smithi116 ceph-mon[35411]: pgmap v3678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:48 smithi114 ceph-mon[30820]: pgmap v3679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:48 smithi116 ceph-mon[35411]: pgmap v3679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:50 smithi114 ceph-mon[30820]: pgmap v3680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:50 smithi116 ceph-mon[35411]: pgmap v3680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:52.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:52 smithi114 ceph-mon[30820]: pgmap v3681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:36:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:52 smithi116 ceph-mon[35411]: pgmap v3681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:36:54.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:54 smithi114 ceph-mon[30820]: pgmap v3682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:54 smithi116 ceph-mon[35411]: pgmap v3682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:36:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:56 smithi114 ceph-mon[30820]: pgmap v3683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:56 smithi116 ceph-mon[35411]: pgmap v3683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:36:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:36:58 smithi114 ceph-mon[30820]: pgmap v3684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:36:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:36:58 smithi116 ceph-mon[35411]: pgmap v3684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:00 smithi114 ceph-mon[30820]: pgmap v3685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:00 smithi116 ceph-mon[35411]: pgmap v3685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:01 smithi114 ceph-mon[30820]: pgmap v3686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:37:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:01 smithi116 ceph-mon[35411]: pgmap v3686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:37:04.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:03 smithi114 ceph-mon[30820]: pgmap v3687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:04.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:03 smithi116 ceph-mon[35411]: pgmap v3687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:06.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:05 smithi114 ceph-mon[30820]: pgmap v3688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:05 smithi116 ceph-mon[35411]: pgmap v3688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:08.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:07 smithi114 ceph-mon[30820]: pgmap v3689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:08.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:07 smithi116 ceph-mon[35411]: pgmap v3689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:09 smithi114 ceph-mon[30820]: pgmap v3690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:09 smithi116 ceph-mon[35411]: pgmap v3690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:12.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:11 smithi114 ceph-mon[30820]: pgmap v3691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:37:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:11 smithi116 ceph-mon[35411]: pgmap v3691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:37:14.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:13 smithi114 ceph-mon[30820]: pgmap v3692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:37:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:13 smithi116 ceph-mon[35411]: pgmap v3692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:37:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:16.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:15 smithi114 ceph-mon[30820]: pgmap v3693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:15 smithi116 ceph-mon[35411]: pgmap v3693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:37:16.244+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:37:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:37:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:37:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:37:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:37:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:37:16.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:37:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:37:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:37:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:37:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:37:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:37:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:37:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:37:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:37:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-mon[30820]: pgmap v3694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:37:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-mon[30820]: pgmap v3695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:37:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:16 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:37:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:37:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:37:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:16 smithi116 ceph-mon[35411]: pgmap v3694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:16 smithi116 ceph-mon[35411]: pgmap v3695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:37:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:19 smithi114 ceph-mon[30820]: pgmap v3696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:37:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:19 smithi116 ceph-mon[35411]: pgmap v3696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:37:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:21 smithi114 ceph-mon[30820]: pgmap v3697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:37:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:21 smithi116 ceph-mon[35411]: pgmap v3697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:37:23.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:23 smithi114 ceph-mon[30820]: pgmap v3698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:37:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:23 smithi116 ceph-mon[35411]: pgmap v3698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:37:25.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:25 smithi114 ceph-mon[30820]: pgmap v3699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:37:25.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:25 smithi116 ceph-mon[35411]: pgmap v3699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:37:27.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:27 smithi114 ceph-mon[30820]: pgmap v3700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:37:27.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:27 smithi116 ceph-mon[35411]: pgmap v3700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:37:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:29 smithi114 ceph-mon[30820]: pgmap v3701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:29 smithi116 ceph-mon[35411]: pgmap v3701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:31.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:31 smithi114 ceph-mon[30820]: pgmap v3702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:37:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:31 smithi116 ceph-mon[35411]: pgmap v3702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:37:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:33 smithi114 ceph-mon[30820]: pgmap v3703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:37:33.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:33 smithi116 ceph-mon[35411]: pgmap v3703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:37:35.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:35 smithi114 ceph-mon[30820]: pgmap v3704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:35.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:35 smithi116 ceph-mon[35411]: pgmap v3704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:37.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:37 smithi114 ceph-mon[30820]: pgmap v3705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:37 smithi116 ceph-mon[35411]: pgmap v3705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:39.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:39 smithi114 ceph-mon[30820]: pgmap v3706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:39 smithi116 ceph-mon[35411]: pgmap v3706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:41.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:41 smithi114 ceph-mon[30820]: pgmap v3707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:37:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:41 smithi116 ceph-mon[35411]: pgmap v3707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:37:43.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:43 smithi114 ceph-mon[30820]: pgmap v3708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:37:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:43 smithi116 ceph-mon[35411]: pgmap v3708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:37:45.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:45 smithi114 ceph-mon[30820]: pgmap v3709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:45 smithi116 ceph-mon[35411]: pgmap v3709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:47.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:47 smithi114 ceph-mon[30820]: pgmap v3710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:47 smithi116 ceph-mon[35411]: pgmap v3710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:49.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:49 smithi114 ceph-mon[30820]: pgmap v3711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:49 smithi116 ceph-mon[35411]: pgmap v3711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:51.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:51 smithi114 ceph-mon[30820]: pgmap v3712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:37:51.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:51 smithi116 ceph-mon[35411]: pgmap v3712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:37:53.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:53 smithi114 ceph-mon[30820]: pgmap v3713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:37:53.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:53 smithi116 ceph-mon[35411]: pgmap v3713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:37:55.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:55 smithi114 ceph-mon[30820]: pgmap v3714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:55.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:55 smithi116 ceph-mon[35411]: pgmap v3714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:37:57.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:57 smithi114 ceph-mon[30820]: pgmap v3715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:57.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:57 smithi116 ceph-mon[35411]: pgmap v3715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:37:59.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:37:59 smithi114 ceph-mon[30820]: pgmap v3716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:37:59.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:37:59 smithi116 ceph-mon[35411]: pgmap v3716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:01.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:01 smithi114 ceph-mon[30820]: pgmap v3717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:38:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:01 smithi116 ceph-mon[35411]: pgmap v3717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:38:03.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:03 smithi114 ceph-mon[30820]: pgmap v3718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:38:03.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:03 smithi116 ceph-mon[35411]: pgmap v3718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:38:05.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:05 smithi114 ceph-mon[30820]: pgmap v3719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:38:05.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:05 smithi116 ceph-mon[35411]: pgmap v3719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:38:07.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:07 smithi114 ceph-mon[30820]: pgmap v3720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:07 smithi116 ceph-mon[35411]: pgmap v3720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:09.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:09 smithi114 ceph-mon[30820]: pgmap v3721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:09.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:09 smithi116 ceph-mon[35411]: pgmap v3721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:11.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:11 smithi114 ceph-mon[30820]: pgmap v3722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:38:11.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:11 smithi116 ceph-mon[35411]: pgmap v3722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:38:13.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:13 smithi114 ceph-mon[30820]: pgmap v3723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:38:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:13 smithi116 ceph-mon[35411]: pgmap v3723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:38:15.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:15 smithi114 ceph-mon[30820]: pgmap v3724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:38:15.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:15 smithi116 ceph-mon[35411]: pgmap v3724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:38:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:38:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:38:17.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:17 smithi114 ceph-mon[30820]: pgmap v3725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:17 smithi116 ceph-mon[35411]: pgmap v3725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:19 smithi116 ceph-mon[35411]: pgmap v3726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-mon[30820]: pgmap v3726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:19.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:38:19.604+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:38:19.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:38:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:38:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:38:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:38:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:38:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:38:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:38:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:38:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:38:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:38:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:38:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:38:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:38:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:38:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:38:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:38:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:38:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:38:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:38:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:38:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:38:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:38:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:38:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:21 smithi114 ceph-mon[30820]: pgmap v3727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:38:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:21 smithi114 ceph-mon[30820]: pgmap v3728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:38:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:38:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:38:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:21 smithi116 ceph-mon[35411]: pgmap v3727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:38:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:21 smithi116 ceph-mon[35411]: pgmap v3728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:38:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:23 smithi114 ceph-mon[30820]: pgmap v3729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:38:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:23 smithi116 ceph-mon[35411]: pgmap v3729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:38:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:25 smithi114 ceph-mon[30820]: pgmap v3730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:38:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:25 smithi116 ceph-mon[35411]: pgmap v3730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:38:27.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:27 smithi114 ceph-mon[30820]: pgmap v3731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:38:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:27 smithi116 ceph-mon[35411]: pgmap v3731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:38:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:29 smithi114 ceph-mon[30820]: pgmap v3732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:38:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:29 smithi116 ceph-mon[35411]: pgmap v3732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:38:31.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:31 smithi114 ceph-mon[30820]: pgmap v3733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:38:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:31 smithi116 ceph-mon[35411]: pgmap v3733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:38:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:33 smithi114 ceph-mon[30820]: pgmap v3734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:33 smithi116 ceph-mon[35411]: pgmap v3734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:35.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:35 smithi114 ceph-mon[30820]: pgmap v3735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:35 smithi116 ceph-mon[35411]: pgmap v3735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:37.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:37 smithi114 ceph-mon[30820]: pgmap v3736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:38:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:37 smithi116 ceph-mon[35411]: pgmap v3736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:38:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:39 smithi114 ceph-mon[30820]: pgmap v3737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:39 smithi116 ceph-mon[35411]: pgmap v3737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:41 smithi114 ceph-mon[30820]: pgmap v3738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:38:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:41 smithi116 ceph-mon[35411]: pgmap v3738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:38:43.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:43 smithi114 ceph-mon[30820]: pgmap v3739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:43 smithi116 ceph-mon[35411]: pgmap v3739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:45 smithi114 ceph-mon[30820]: pgmap v3740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:45 smithi116 ceph-mon[35411]: pgmap v3740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:47 smithi114 ceph-mon[30820]: pgmap v3741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:38:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:47 smithi116 ceph-mon[35411]: pgmap v3741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:38:49.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:49 smithi116 ceph-mon[35411]: pgmap v3742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:49 smithi114 ceph-mon[30820]: pgmap v3742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:51.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:51 smithi114 ceph-mon[30820]: pgmap v3743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:38:51.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:51 smithi116 ceph-mon[35411]: pgmap v3743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:38:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:53 smithi114 ceph-mon[30820]: pgmap v3744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:53 smithi116 ceph-mon[35411]: pgmap v3744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:38:55.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:55 smithi114 ceph-mon[30820]: pgmap v3745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:55 smithi116 ceph-mon[35411]: pgmap v3745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:57 smithi114 ceph-mon[30820]: pgmap v3746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:38:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:57 smithi116 ceph-mon[35411]: pgmap v3746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:38:59.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:38:59 smithi114 ceph-mon[30820]: pgmap v3747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:38:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:38:59 smithi116 ceph-mon[35411]: pgmap v3747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:01.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:01 smithi114 ceph-mon[30820]: pgmap v3748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:01 smithi116 ceph-mon[35411]: pgmap v3748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:03.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:03 smithi114 ceph-mon[30820]: pgmap v3749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:03 smithi116 ceph-mon[35411]: pgmap v3749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:05 smithi114 ceph-mon[30820]: pgmap v3750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:05 smithi116 ceph-mon[35411]: pgmap v3750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:07.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:07 smithi114 ceph-mon[30820]: pgmap v3751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:39:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:07 smithi116 ceph-mon[35411]: pgmap v3751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:39:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:09 smithi114 ceph-mon[30820]: pgmap v3752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:09 smithi116 ceph-mon[35411]: pgmap v3752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:11 smithi114 ceph-mon[30820]: pgmap v3753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:11 smithi116 ceph-mon[35411]: pgmap v3753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:13 smithi114 ceph-mon[30820]: pgmap v3754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:13 smithi116 ceph-mon[35411]: pgmap v3754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:15 smithi114 ceph-mon[30820]: pgmap v3755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:15 smithi116 ceph-mon[35411]: pgmap v3755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:17 smithi114 ceph-mon[30820]: pgmap v3756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:39:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:17 smithi116 ceph-mon[35411]: pgmap v3756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:39:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:19 smithi114 ceph-mon[30820]: pgmap v3757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:19 smithi116 ceph-mon[35411]: pgmap v3757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:39:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:39:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:21 smithi114 ceph-mon[30820]: pgmap v3758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:21 smithi116 ceph-mon[35411]: pgmap v3758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:23.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:39:22.965+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:39:23.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:39:23.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:39:23.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:39:23.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:39:23.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:39:23.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:39:23.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:39:23.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:39:23.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:39:23.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:39:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:39:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:23 smithi114 ceph-mon[30820]: pgmap v3759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:39:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:39:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:39:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:23 smithi116 ceph-mon[35411]: pgmap v3759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:39:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:39:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:39:24.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:39:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:39:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:39:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:39:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:39:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:39:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:39:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:39:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:39:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:39:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:39:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:39:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:24 smithi114 ceph-mon[30820]: pgmap v3760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:39:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:24 smithi114 ceph-mon[30820]: pgmap v3761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:39:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:24 smithi116 ceph-mon[35411]: pgmap v3760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:39:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:24 smithi116 ceph-mon[35411]: pgmap v3761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:39:26.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:26 smithi114 ceph-mon[30820]: pgmap v3762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:39:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:26 smithi116 ceph-mon[35411]: pgmap v3762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:39:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:28 smithi114 ceph-mon[30820]: pgmap v3763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:39:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:28 smithi116 ceph-mon[35411]: pgmap v3763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:39:30.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:30 smithi114 ceph-mon[30820]: pgmap v3764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:39:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:30 smithi116 ceph-mon[35411]: pgmap v3764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:39:32.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:32 smithi114 ceph-mon[30820]: pgmap v3765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:39:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:32 smithi116 ceph-mon[35411]: pgmap v3765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:39:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:34 smithi114 ceph-mon[30820]: pgmap v3766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:39:34.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:34 smithi116 ceph-mon[35411]: pgmap v3766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:39:36.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:36 smithi114 ceph-mon[30820]: pgmap v3767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:36 smithi116 ceph-mon[35411]: pgmap v3767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:38.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:38 smithi114 ceph-mon[30820]: pgmap v3768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:38 smithi116 ceph-mon[35411]: pgmap v3768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:40.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:40 smithi114 ceph-mon[30820]: pgmap v3769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:40 smithi116 ceph-mon[35411]: pgmap v3769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:42.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:42 smithi114 ceph-mon[30820]: pgmap v3770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:39:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:42 smithi116 ceph-mon[35411]: pgmap v3770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:39:44.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:44 smithi114 ceph-mon[30820]: pgmap v3771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:44 smithi116 ceph-mon[35411]: pgmap v3771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:46 smithi114 ceph-mon[30820]: pgmap v3772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:46 smithi116 ceph-mon[35411]: pgmap v3772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:48 smithi114 ceph-mon[30820]: pgmap v3773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:48.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:48 smithi116 ceph-mon[35411]: pgmap v3773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:49 smithi114 ceph-mon[30820]: pgmap v3774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:49 smithi116 ceph-mon[35411]: pgmap v3774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:52.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:52 smithi114 ceph-mon[30820]: pgmap v3775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:39:52.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:52 smithi116 ceph-mon[35411]: pgmap v3775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:39:54.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:54 smithi114 ceph-mon[30820]: pgmap v3776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:54.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:54 smithi116 ceph-mon[35411]: pgmap v3776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:39:56.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:56 smithi114 ceph-mon[30820]: pgmap v3777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:56 smithi116 ceph-mon[35411]: pgmap v3777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:39:58.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:39:58 smithi114 ceph-mon[30820]: pgmap v3778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:39:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:39:58 smithi116 ceph-mon[35411]: pgmap v3778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:00.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:00 smithi114 ceph-mon[30820]: pgmap v3779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:00.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:40:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:00 smithi116 ceph-mon[35411]: pgmap v3779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:40:02.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:02 smithi114 ceph-mon[30820]: pgmap v3780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:40:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:02 smithi116 ceph-mon[35411]: pgmap v3780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:40:04.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:04 smithi114 ceph-mon[30820]: pgmap v3781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:04.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:04 smithi116 ceph-mon[35411]: pgmap v3781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:06 smithi114 ceph-mon[30820]: pgmap v3782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:06 smithi116 ceph-mon[35411]: pgmap v3782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:08.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:08 smithi114 ceph-mon[30820]: pgmap v3783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:08.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:08 smithi116 ceph-mon[35411]: pgmap v3783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:10 smithi114 ceph-mon[30820]: pgmap v3784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:10 smithi116 ceph-mon[35411]: pgmap v3784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:12 smithi114 ceph-mon[30820]: pgmap v3785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:40:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:12 smithi116 ceph-mon[35411]: pgmap v3785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:40:14.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:14 smithi114 ceph-mon[30820]: pgmap v3786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:14 smithi116 ceph-mon[35411]: pgmap v3786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:16.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:16 smithi114 ceph-mon[30820]: pgmap v3787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:16 smithi116 ceph-mon[35411]: pgmap v3787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:18.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:18 smithi114 ceph-mon[30820]: pgmap v3788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:18 smithi116 ceph-mon[35411]: pgmap v3788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:20.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:20 smithi114 ceph-mon[30820]: pgmap v3789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:20.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:20 smithi116 ceph-mon[35411]: pgmap v3789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:22.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:22 smithi114 ceph-mon[30820]: pgmap v3790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:40:22.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:22 smithi116 ceph-mon[35411]: pgmap v3790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:40:23.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:40:23.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:40:24.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:24 smithi114 ceph-mon[30820]: pgmap v3791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:24.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:24 smithi116 ceph-mon[35411]: pgmap v3791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:26.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-mon[30820]: pgmap v3792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:26.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:40:26.324+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:40:26.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:40:26.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:40:26.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:40:26.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:40:26.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:40:26.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:40:26.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:40:26.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:40:26.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:40:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:40:26.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:26 smithi116 ceph-mon[35411]: pgmap v3792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:40:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:40:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:40:27.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:27 smithi114 ceph-mon[30820]: pgmap v3793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:40:27.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:27 smithi114 ceph-mon[30820]: pgmap v3794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:40:27.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:40:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:40:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:40:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:40:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:27 smithi116 ceph-mon[35411]: pgmap v3793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:27 smithi116 ceph-mon[35411]: pgmap v3794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:40:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:29 smithi114 ceph-mon[30820]: pgmap v3795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:40:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:29 smithi116 ceph-mon[35411]: pgmap v3795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:40:31.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:31 smithi114 ceph-mon[30820]: pgmap v3796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:40:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:31 smithi116 ceph-mon[35411]: pgmap v3796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:40:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:33 smithi114 ceph-mon[30820]: pgmap v3797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:40:33.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:33 smithi116 ceph-mon[35411]: pgmap v3797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:40:35.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:35 smithi114 ceph-mon[30820]: pgmap v3798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:40:35.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:35 smithi116 ceph-mon[35411]: pgmap v3798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:40:37.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:37 smithi114 ceph-mon[30820]: pgmap v3799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:40:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:37 smithi116 ceph-mon[35411]: pgmap v3799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:40:39.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:39 smithi114 ceph-mon[30820]: pgmap v3800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:39 smithi116 ceph-mon[35411]: pgmap v3800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:41.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:41 smithi114 ceph-mon[30820]: pgmap v3801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:40:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:41 smithi116 ceph-mon[35411]: pgmap v3801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:40:43.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:43 smithi114 ceph-mon[30820]: pgmap v3802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:40:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:43 smithi116 ceph-mon[35411]: pgmap v3802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:40:45.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:45 smithi114 ceph-mon[30820]: pgmap v3803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:45 smithi116 ceph-mon[35411]: pgmap v3803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:47 smithi116 ceph-mon[35411]: pgmap v3804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:47 smithi114 ceph-mon[30820]: pgmap v3804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:49 smithi116 ceph-mon[35411]: pgmap v3805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:49 smithi114 ceph-mon[30820]: pgmap v3805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:51.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:51 smithi116 ceph-mon[35411]: pgmap v3806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:40:51.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:51 smithi114 ceph-mon[30820]: pgmap v3806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:40:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:53 smithi114 ceph-mon[30820]: pgmap v3807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:40:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:53 smithi116 ceph-mon[35411]: pgmap v3807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:40:55.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:55 smithi114 ceph-mon[30820]: pgmap v3808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:55 smithi116 ceph-mon[35411]: pgmap v3808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:40:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:57 smithi114 ceph-mon[30820]: pgmap v3809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:57 smithi116 ceph-mon[35411]: pgmap v3809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:40:59.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:40:59 smithi114 ceph-mon[30820]: pgmap v3810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:40:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:40:59 smithi116 ceph-mon[35411]: pgmap v3810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:01.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:01 smithi114 ceph-mon[30820]: pgmap v3811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:41:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:01 smithi116 ceph-mon[35411]: pgmap v3811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:41:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:03 smithi114 ceph-mon[30820]: pgmap v3812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:41:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:03 smithi116 ceph-mon[35411]: pgmap v3812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:41:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:05 smithi114 ceph-mon[30820]: pgmap v3813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:41:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:05 smithi116 ceph-mon[35411]: pgmap v3813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:41:07.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:07 smithi114 ceph-mon[30820]: pgmap v3814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:07 smithi116 ceph-mon[35411]: pgmap v3814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:09 smithi114 ceph-mon[30820]: pgmap v3815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:09 smithi116 ceph-mon[35411]: pgmap v3815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:11 smithi114 ceph-mon[30820]: pgmap v3816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:41:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:11 smithi116 ceph-mon[35411]: pgmap v3816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:41:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:13 smithi114 ceph-mon[30820]: pgmap v3817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:41:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:13 smithi116 ceph-mon[35411]: pgmap v3817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:41:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:15 smithi114 ceph-mon[30820]: pgmap v3818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:41:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:15 smithi116 ceph-mon[35411]: pgmap v3818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:41:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:17 smithi114 ceph-mon[30820]: pgmap v3819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:17 smithi116 ceph-mon[35411]: pgmap v3819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:19 smithi114 ceph-mon[30820]: pgmap v3820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:19 smithi116 ceph-mon[35411]: pgmap v3820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:21 smithi114 ceph-mon[30820]: pgmap v3821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:41:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:21 smithi116 ceph-mon[35411]: pgmap v3821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:41:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:23 smithi114 ceph-mon[30820]: pgmap v3822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:41:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:23 smithi116 ceph-mon[35411]: pgmap v3822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:41:25.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:25 smithi114 ceph-mon[30820]: pgmap v3823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:41:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:25 smithi116 ceph-mon[35411]: pgmap v3823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:41:26.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:41:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:41:27.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:27 smithi114 ceph-mon[30820]: pgmap v3824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:27 smithi116 ceph-mon[35411]: pgmap v3824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-mon[30820]: pgmap v3825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:29.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:41:29.684+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:41:29.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:41:29.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:41:29.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:41:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:41:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:41:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:41:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:41:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:41:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:41:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:41:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:41:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:41:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:41:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:29 smithi116 ceph-mon[35411]: pgmap v3825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:30.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:41:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:41:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:41:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:41:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:41:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:41:31.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:41:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:31 smithi114 ceph-mon[30820]: pgmap v3826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:41:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:31 smithi114 ceph-mon[30820]: pgmap v3827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:41:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:41:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:41:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:31 smithi116 ceph-mon[35411]: pgmap v3826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:41:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:31 smithi116 ceph-mon[35411]: pgmap v3827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:41:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:33 smithi114 ceph-mon[30820]: pgmap v3828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:41:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:33 smithi116 ceph-mon[35411]: pgmap v3828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:41:35.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:35 smithi114 ceph-mon[30820]: pgmap v3829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:41:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:35 smithi116 ceph-mon[35411]: pgmap v3829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:41:37.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:37 smithi114 ceph-mon[30820]: pgmap v3830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:41:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:37 smithi116 ceph-mon[35411]: pgmap v3830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:41:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:39 smithi114 ceph-mon[30820]: pgmap v3831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:41:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:39 smithi116 ceph-mon[35411]: pgmap v3831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:41:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:41 smithi114 ceph-mon[30820]: pgmap v3832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:41:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:41 smithi116 ceph-mon[35411]: pgmap v3832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:41:43.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:43 smithi114 ceph-mon[30820]: pgmap v3833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:43 smithi116 ceph-mon[35411]: pgmap v3833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:45 smithi114 ceph-mon[30820]: pgmap v3834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:45 smithi116 ceph-mon[35411]: pgmap v3834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:47 smithi114 ceph-mon[30820]: pgmap v3835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:41:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:47 smithi116 ceph-mon[35411]: pgmap v3835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:41:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:49 smithi114 ceph-mon[30820]: pgmap v3836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:49 smithi116 ceph-mon[35411]: pgmap v3836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:51 smithi114 ceph-mon[30820]: pgmap v3837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:41:51.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:51 smithi116 ceph-mon[35411]: pgmap v3837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:41:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:53 smithi114 ceph-mon[30820]: pgmap v3838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:53 smithi116 ceph-mon[35411]: pgmap v3838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:41:55.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:55 smithi114 ceph-mon[30820]: pgmap v3839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:55 smithi116 ceph-mon[35411]: pgmap v3839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:57 smithi114 ceph-mon[30820]: pgmap v3840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:41:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:57 smithi116 ceph-mon[35411]: pgmap v3840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:41:59.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:41:59 smithi114 ceph-mon[30820]: pgmap v3841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:41:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:41:59 smithi116 ceph-mon[35411]: pgmap v3841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:01.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:01 smithi114 ceph-mon[30820]: pgmap v3842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:01 smithi116 ceph-mon[35411]: pgmap v3842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:03.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:03 smithi114 ceph-mon[30820]: pgmap v3843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:03 smithi116 ceph-mon[35411]: pgmap v3843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:05 smithi114 ceph-mon[30820]: pgmap v3844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:05 smithi116 ceph-mon[35411]: pgmap v3844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:07.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:07 smithi114 ceph-mon[30820]: pgmap v3845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:42:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:07 smithi116 ceph-mon[35411]: pgmap v3845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:42:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:09 smithi114 ceph-mon[30820]: pgmap v3846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:09 smithi116 ceph-mon[35411]: pgmap v3846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:11 smithi114 ceph-mon[30820]: pgmap v3847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:11 smithi116 ceph-mon[35411]: pgmap v3847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:13 smithi114 ceph-mon[30820]: pgmap v3848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:13 smithi116 ceph-mon[35411]: pgmap v3848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:15 smithi114 ceph-mon[30820]: pgmap v3849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:15 smithi116 ceph-mon[35411]: pgmap v3849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:17 smithi114 ceph-mon[30820]: pgmap v3850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:42:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:17 smithi116 ceph-mon[35411]: pgmap v3850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:42:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:19 smithi114 ceph-mon[30820]: pgmap v3851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:19 smithi116 ceph-mon[35411]: pgmap v3851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:21 smithi114 ceph-mon[30820]: pgmap v3852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:21 smithi116 ceph-mon[35411]: pgmap v3852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:23 smithi114 ceph-mon[30820]: pgmap v3853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:23 smithi116 ceph-mon[35411]: pgmap v3853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:25.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:25 smithi114 ceph-mon[30820]: pgmap v3854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:25 smithi116 ceph-mon[35411]: pgmap v3854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:27.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:27 smithi114 ceph-mon[30820]: pgmap v3855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:42:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:27 smithi116 ceph-mon[35411]: pgmap v3855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:42:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:29 smithi114 ceph-mon[30820]: pgmap v3856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:29 smithi116 ceph-mon[35411]: pgmap v3856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:30.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:42:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:42:31.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:31 smithi114 ceph-mon[30820]: pgmap v3857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:31 smithi116 ceph-mon[35411]: pgmap v3857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:32.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:32 smithi114 ceph-mon[30820]: pgmap v3858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:32 smithi116 ceph-mon[35411]: pgmap v3858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:33.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:42:33.044+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:42:33.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:42:33.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:42:33.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:42:33.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:42:33.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:42:33.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:42:33.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:42:33.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:42:33.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:42:33.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:42:33.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:42:33.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:42:33.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:42:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:42:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:42:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:42:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:42:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-mon[30820]: pgmap v3859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:42:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-mon[30820]: pgmap v3860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:42:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:42:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:33 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:42:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:42:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:42:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:33 smithi116 ceph-mon[35411]: pgmap v3859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:33 smithi116 ceph-mon[35411]: pgmap v3860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:42:36.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:36 smithi114 ceph-mon[30820]: pgmap v3861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:42:36.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:36 smithi116 ceph-mon[35411]: pgmap v3861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:42:38.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:38 smithi114 ceph-mon[30820]: pgmap v3862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:42:38.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:38 smithi116 ceph-mon[35411]: pgmap v3862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:42:40.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:40 smithi114 ceph-mon[30820]: pgmap v3863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:42:40.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:40 smithi116 ceph-mon[35411]: pgmap v3863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:42:42.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:42 smithi114 ceph-mon[30820]: pgmap v3864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:42:42.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:42 smithi116 ceph-mon[35411]: pgmap v3864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:42:44.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:44 smithi114 ceph-mon[30820]: pgmap v3865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:42:44.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:44 smithi116 ceph-mon[35411]: pgmap v3865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:42:46.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:46 smithi114 ceph-mon[30820]: pgmap v3866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:46.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:46 smithi116 ceph-mon[35411]: pgmap v3866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:48.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:48 smithi114 ceph-mon[30820]: pgmap v3867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:48.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:48 smithi116 ceph-mon[35411]: pgmap v3867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:50.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:50 smithi114 ceph-mon[30820]: pgmap v3868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:50.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:50 smithi116 ceph-mon[35411]: pgmap v3868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:52.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:52 smithi114 ceph-mon[30820]: pgmap v3869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:42:52.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:52 smithi116 ceph-mon[35411]: pgmap v3869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:42:54.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:54 smithi114 ceph-mon[30820]: pgmap v3870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:54.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:54 smithi116 ceph-mon[35411]: pgmap v3870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:42:56.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:56 smithi114 ceph-mon[30820]: pgmap v3871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:56 smithi116 ceph-mon[35411]: pgmap v3871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:42:58.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:42:58 smithi114 ceph-mon[30820]: pgmap v3872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:42:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:42:58 smithi116 ceph-mon[35411]: pgmap v3872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:43:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:00 smithi114 ceph-mon[30820]: pgmap v3873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:00.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:00 smithi116 ceph-mon[35411]: pgmap v3873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:02.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:02 smithi114 ceph-mon[30820]: pgmap v3874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:02 smithi116 ceph-mon[35411]: pgmap v3874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:04 smithi114 ceph-mon[30820]: pgmap v3875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:04.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:04 smithi116 ceph-mon[35411]: pgmap v3875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:06.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:06 smithi114 ceph-mon[30820]: pgmap v3876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:43:06.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:06 smithi116 ceph-mon[35411]: pgmap v3876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:43:08.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:08 smithi114 ceph-mon[30820]: pgmap v3877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:43:08.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:08 smithi116 ceph-mon[35411]: pgmap v3877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:43:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:10 smithi114 ceph-mon[30820]: pgmap v3878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:10 smithi116 ceph-mon[35411]: pgmap v3878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:12.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:12 smithi114 ceph-mon[30820]: pgmap v3879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:12 smithi116 ceph-mon[35411]: pgmap v3879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:14 smithi116 ceph-mon[35411]: pgmap v3880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:14.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:14 smithi114 ceph-mon[30820]: pgmap v3880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:16.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:16 smithi116 ceph-mon[35411]: pgmap v3881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:43:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:16 smithi114 ceph-mon[30820]: pgmap v3881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:43:18.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:18 smithi116 ceph-mon[35411]: pgmap v3882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:43:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:18 smithi114 ceph-mon[30820]: pgmap v3882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:43:20.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:20 smithi116 ceph-mon[35411]: pgmap v3883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:20 smithi114 ceph-mon[30820]: pgmap v3883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:22.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:22 smithi114 ceph-mon[30820]: pgmap v3884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:22 smithi116 ceph-mon[35411]: pgmap v3884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:24 smithi114 ceph-mon[30820]: pgmap v3885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:24 smithi116 ceph-mon[35411]: pgmap v3885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:26 smithi114 ceph-mon[30820]: pgmap v3886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:43:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:26 smithi116 ceph-mon[35411]: pgmap v3886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:43:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:28 smithi114 ceph-mon[30820]: pgmap v3887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:43:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:28 smithi116 ceph-mon[35411]: pgmap v3887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:43:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:30 smithi114 ceph-mon[30820]: pgmap v3888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:30 smithi116 ceph-mon[35411]: pgmap v3888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:32.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:32 smithi114 ceph-mon[30820]: pgmap v3889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:32 smithi116 ceph-mon[35411]: pgmap v3889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:43:33.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:43:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:34 smithi114 ceph-mon[30820]: pgmap v3890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:34 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:43:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:34 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:43:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:34 smithi116 ceph-mon[35411]: pgmap v3890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:43:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:34 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:43:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:34 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:43:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:43:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:43:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:35 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:43:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:35 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:43:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:43:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:43:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:43:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:35 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:43:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:35 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:43:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-mon[30820]: pgmap v3891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:43:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:43:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:43:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:43:36.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:43:36.404+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:43:36.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:43:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:43:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:43:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:43:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:43:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:43:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:43:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:43:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:43:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:43:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:43:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:36 smithi116 ceph-mon[35411]: pgmap v3891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:43:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:43:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:43:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:43:37.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:37 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:37 smithi114 ceph-mon[30820]: pgmap v3892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:43:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:37 smithi114 ceph-mon[30820]: pgmap v3893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:37 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:43:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:37 smithi116 ceph-mon[35411]: pgmap v3892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:43:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:37 smithi116 ceph-mon[35411]: pgmap v3893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:39 smithi114 ceph-mon[30820]: pgmap v3894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:39 smithi116 ceph-mon[35411]: pgmap v3894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:41 smithi114 ceph-mon[30820]: pgmap v3895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:41 smithi116 ceph-mon[35411]: pgmap v3895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:42 smithi114 ceph-mon[30820]: pgmap v3896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:42 smithi116 ceph-mon[35411]: pgmap v3896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:45 smithi114 ceph-mon[30820]: pgmap v3897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:45 smithi116 ceph-mon[35411]: pgmap v3897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:43:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:47 smithi114 ceph-mon[30820]: pgmap v3898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:43:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:47 smithi116 ceph-mon[35411]: pgmap v3898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:43:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:49 smithi114 ceph-mon[30820]: pgmap v3899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:43:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:49 smithi116 ceph-mon[35411]: pgmap v3899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:43:51.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:51 smithi114 ceph-mon[30820]: pgmap v3900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:51.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:51 smithi116 ceph-mon[35411]: pgmap v3900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:43:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:53 smithi114 ceph-mon[30820]: pgmap v3901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:43:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:53 smithi116 ceph-mon[35411]: pgmap v3901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:43:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:55 smithi114 ceph-mon[30820]: pgmap v3902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:43:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:55 smithi116 ceph-mon[35411]: pgmap v3902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:43:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:57 smithi114 ceph-mon[30820]: pgmap v3903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 6 op/s 2024-09-19T08:43:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:57 smithi116 ceph-mon[35411]: pgmap v3903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 6 op/s 2024-09-19T08:43:59.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:43:59 smithi114 ceph-mon[30820]: pgmap v3904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:43:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:43:59 smithi116 ceph-mon[35411]: pgmap v3904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:44:01.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:01 smithi114 ceph-mon[30820]: pgmap v3905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:01 smithi116 ceph-mon[35411]: pgmap v3905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:03.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:03 smithi114 ceph-mon[30820]: pgmap v3906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:44:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:03 smithi116 ceph-mon[35411]: pgmap v3906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:44:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:05 smithi114 ceph-mon[30820]: pgmap v3907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:44:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:05 smithi116 ceph-mon[35411]: pgmap v3907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:44:07.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:07 smithi114 ceph-mon[30820]: pgmap v3908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:07 smithi116 ceph-mon[35411]: pgmap v3908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:09 smithi114 ceph-mon[30820]: pgmap v3909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:09 smithi116 ceph-mon[35411]: pgmap v3909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:11 smithi114 ceph-mon[30820]: pgmap v3910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:11 smithi116 ceph-mon[35411]: pgmap v3910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:13 smithi114 ceph-mon[30820]: pgmap v3911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:13 smithi116 ceph-mon[35411]: pgmap v3911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:15 smithi114 ceph-mon[30820]: pgmap v3912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:44:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:15 smithi116 ceph-mon[35411]: pgmap v3912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:44:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:17 smithi114 ceph-mon[30820]: pgmap v3913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:17 smithi116 ceph-mon[35411]: pgmap v3913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:19 smithi114 ceph-mon[30820]: pgmap v3914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:19 smithi116 ceph-mon[35411]: pgmap v3914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:21 smithi114 ceph-mon[30820]: pgmap v3915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:21 smithi116 ceph-mon[35411]: pgmap v3915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:23 smithi114 ceph-mon[30820]: pgmap v3916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:23 smithi116 ceph-mon[35411]: pgmap v3916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:25.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:25 smithi114 ceph-mon[30820]: pgmap v3917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:44:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:25 smithi116 ceph-mon[35411]: pgmap v3917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:44:27.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:27 smithi114 ceph-mon[30820]: pgmap v3918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:27 smithi116 ceph-mon[35411]: pgmap v3918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:29 smithi114 ceph-mon[30820]: pgmap v3919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:29 smithi116 ceph-mon[35411]: pgmap v3919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:31.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:31 smithi114 ceph-mon[30820]: pgmap v3920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:31 smithi116 ceph-mon[35411]: pgmap v3920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:33 smithi114 ceph-mon[30820]: pgmap v3921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:33 smithi116 ceph-mon[35411]: pgmap v3921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:35.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:35 smithi114 ceph-mon[30820]: pgmap v3922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:44:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:35 smithi116 ceph-mon[35411]: pgmap v3922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:44:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:44:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:44:37.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:37 smithi114 ceph-mon[30820]: pgmap v3923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:44:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:44:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:37 smithi116 ceph-mon[35411]: pgmap v3923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:44:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:44:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-mon[30820]: pgmap v3924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:44:39.764+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:44:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:44:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:44:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:44:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:44:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:44:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:44:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:44:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:44:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:44:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:44:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:44:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:39 smithi116 ceph-mon[35411]: pgmap v3924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:44:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:44:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:44:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:44:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:44:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:44:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:44:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:44:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:44:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:44:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:44:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:41 smithi114 ceph-mon[30820]: pgmap v3925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:44:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:41 smithi114 ceph-mon[30820]: pgmap v3926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:44:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:44:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:41 smithi116 ceph-mon[35411]: pgmap v3925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:44:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:41 smithi116 ceph-mon[35411]: pgmap v3926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:44:43.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:43 smithi114 ceph-mon[30820]: pgmap v3927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:44:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:43 smithi116 ceph-mon[35411]: pgmap v3927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:44:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:45 smithi114 ceph-mon[30820]: pgmap v3928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:44:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:45 smithi116 ceph-mon[35411]: pgmap v3928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:44:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:47 smithi114 ceph-mon[30820]: pgmap v3929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:44:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:47 smithi116 ceph-mon[35411]: pgmap v3929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:44:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:49 smithi114 ceph-mon[30820]: pgmap v3930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:44:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:49 smithi116 ceph-mon[35411]: pgmap v3930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:44:51.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:51 smithi114 ceph-mon[30820]: pgmap v3931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:44:51.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:51 smithi116 ceph-mon[35411]: pgmap v3931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:44:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:53 smithi114 ceph-mon[30820]: pgmap v3932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:53 smithi116 ceph-mon[35411]: pgmap v3932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:44:55.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:55 smithi114 ceph-mon[30820]: pgmap v3933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:55 smithi116 ceph-mon[35411]: pgmap v3933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:57 smithi114 ceph-mon[30820]: pgmap v3934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:57 smithi116 ceph-mon[35411]: pgmap v3934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:44:59.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:44:59 smithi114 ceph-mon[30820]: pgmap v3935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:44:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:44:59 smithi116 ceph-mon[35411]: pgmap v3935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:01.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:01 smithi114 ceph-mon[30820]: pgmap v3936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:01 smithi116 ceph-mon[35411]: pgmap v3936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:03.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:03 smithi114 ceph-mon[30820]: pgmap v3937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:03 smithi116 ceph-mon[35411]: pgmap v3937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:05 smithi114 ceph-mon[30820]: pgmap v3938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:05 smithi116 ceph-mon[35411]: pgmap v3938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:07.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:07 smithi114 ceph-mon[30820]: pgmap v3939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:45:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:07 smithi116 ceph-mon[35411]: pgmap v3939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:45:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:09 smithi114 ceph-mon[30820]: pgmap v3940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:09 smithi116 ceph-mon[35411]: pgmap v3940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:11 smithi114 ceph-mon[30820]: pgmap v3941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:11 smithi116 ceph-mon[35411]: pgmap v3941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:13 smithi114 ceph-mon[30820]: pgmap v3942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:13 smithi116 ceph-mon[35411]: pgmap v3942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:15 smithi114 ceph-mon[30820]: pgmap v3943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:15 smithi116 ceph-mon[35411]: pgmap v3943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:17 smithi114 ceph-mon[30820]: pgmap v3944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:45:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:17 smithi116 ceph-mon[35411]: pgmap v3944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:45:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:19 smithi114 ceph-mon[30820]: pgmap v3945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:19 smithi116 ceph-mon[35411]: pgmap v3945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:21.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:20 smithi114 ceph-mon[30820]: pgmap v3946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:21.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:20 smithi116 ceph-mon[35411]: pgmap v3946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:22 smithi114 ceph-mon[30820]: pgmap v3947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:23.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:22 smithi116 ceph-mon[35411]: pgmap v3947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:25.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:24 smithi114 ceph-mon[30820]: pgmap v3948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:25.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:24 smithi116 ceph-mon[35411]: pgmap v3948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:27.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:26 smithi114 ceph-mon[30820]: pgmap v3949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:45:27.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:26 smithi116 ceph-mon[35411]: pgmap v3949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:45:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:28 smithi114 ceph-mon[30820]: pgmap v3950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:29.187 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:28 smithi116 ceph-mon[35411]: pgmap v3950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:31.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:30 smithi114 ceph-mon[30820]: pgmap v3951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:31.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:30 smithi116 ceph-mon[35411]: pgmap v3951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:33.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:32 smithi114 ceph-mon[30820]: pgmap v3952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:32 smithi116 ceph-mon[35411]: pgmap v3952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:35.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:34 smithi114 ceph-mon[30820]: pgmap v3953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:35.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:34 smithi116 ceph-mon[35411]: pgmap v3953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:37.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:36 smithi114 ceph-mon[30820]: pgmap v3954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:45:37.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:36 smithi116 ceph-mon[35411]: pgmap v3954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:45:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:38 smithi114 ceph-mon[30820]: pgmap v3955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:39.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:38 smithi116 ceph-mon[35411]: pgmap v3955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:45:41.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:40 smithi114 ceph-mon[30820]: pgmap v3956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:45:41.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:40 smithi116 ceph-mon[35411]: pgmap v3956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:45:43.125 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:42 smithi114 ceph-mon[30820]: pgmap v3957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:43.126 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:45:43.124+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:45:43.126 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:45:43.126 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:45:43.126 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:45:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:42 smithi116 ceph-mon[35411]: pgmap v3957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:43.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:45:43.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:45:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:45:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:45:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:45:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:45:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:45:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:45:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-mon[30820]: pgmap v3958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:45:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-mon[30820]: pgmap v3959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:45:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:45:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:43 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:43 smithi116 ceph-mon[35411]: pgmap v3958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:43 smithi116 ceph-mon[35411]: pgmap v3959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:45:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:46 smithi114 ceph-mon[30820]: pgmap v3960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:45:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:46 smithi116 ceph-mon[35411]: pgmap v3960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:45:48.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:48 smithi114 ceph-mon[30820]: pgmap v3961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:45:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:48 smithi116 ceph-mon[35411]: pgmap v3961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:45:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:50 smithi114 ceph-mon[30820]: pgmap v3962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:45:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:50 smithi116 ceph-mon[35411]: pgmap v3962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:45:52.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:52 smithi114 ceph-mon[30820]: pgmap v3963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:45:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:52 smithi116 ceph-mon[35411]: pgmap v3963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:45:54.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:54 smithi114 ceph-mon[30820]: pgmap v3964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:45:54.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:54 smithi116 ceph-mon[35411]: pgmap v3964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:45:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:56 smithi114 ceph-mon[30820]: pgmap v3965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:56 smithi116 ceph-mon[35411]: pgmap v3965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:45:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:45:58 smithi114 ceph-mon[30820]: pgmap v3966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:45:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:45:58 smithi116 ceph-mon[35411]: pgmap v3966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:00 smithi114 ceph-mon[30820]: pgmap v3967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:00 smithi116 ceph-mon[35411]: pgmap v3967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:02.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:02 smithi114 ceph-mon[30820]: pgmap v3968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:02 smithi116 ceph-mon[35411]: pgmap v3968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:04.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:04 smithi114 ceph-mon[30820]: pgmap v3969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:04 smithi116 ceph-mon[35411]: pgmap v3969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:06.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:06 smithi114 ceph-mon[30820]: pgmap v3970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:46:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:06 smithi116 ceph-mon[35411]: pgmap v3970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:46:08.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:08 smithi114 ceph-mon[30820]: pgmap v3971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:08.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:08 smithi116 ceph-mon[35411]: pgmap v3971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:10.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:10 smithi114 ceph-mon[30820]: pgmap v3972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:46:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:10 smithi116 ceph-mon[35411]: pgmap v3972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:46:12.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:12 smithi114 ceph-mon[30820]: pgmap v3973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:12 smithi116 ceph-mon[35411]: pgmap v3973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:14.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:14 smithi114 ceph-mon[30820]: pgmap v3974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:46:14.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:14 smithi116 ceph-mon[35411]: pgmap v3974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:46:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:16 smithi114 ceph-mon[30820]: pgmap v3975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:46:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:16 smithi116 ceph-mon[35411]: pgmap v3975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:46:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:18 smithi114 ceph-mon[30820]: pgmap v3976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:18.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:18 smithi116 ceph-mon[35411]: pgmap v3976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:20 smithi114 ceph-mon[30820]: pgmap v3977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:20 smithi116 ceph-mon[35411]: pgmap v3977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:22.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:22 smithi114 ceph-mon[30820]: pgmap v3978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:22 smithi116 ceph-mon[35411]: pgmap v3978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:24 smithi114 ceph-mon[30820]: pgmap v3979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:24 smithi116 ceph-mon[35411]: pgmap v3979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:26 smithi114 ceph-mon[30820]: pgmap v3980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:46:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:26 smithi116 ceph-mon[35411]: pgmap v3980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:46:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:28 smithi114 ceph-mon[30820]: pgmap v3981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:28 smithi116 ceph-mon[35411]: pgmap v3981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:30 smithi114 ceph-mon[30820]: pgmap v3982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:30 smithi116 ceph-mon[35411]: pgmap v3982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:32.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:32 smithi114 ceph-mon[30820]: pgmap v3983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:32 smithi116 ceph-mon[35411]: pgmap v3983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:34 smithi114 ceph-mon[30820]: pgmap v3984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:34 smithi116 ceph-mon[35411]: pgmap v3984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:36 smithi114 ceph-mon[30820]: pgmap v3985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:46:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:36 smithi116 ceph-mon[35411]: pgmap v3985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:46:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:38 smithi114 ceph-mon[30820]: pgmap v3986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:38 smithi116 ceph-mon[35411]: pgmap v3986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:46:40.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:40 smithi114 ceph-mon[30820]: pgmap v3987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:40 smithi116 ceph-mon[35411]: pgmap v3987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:41 smithi114 ceph-mon[30820]: pgmap v3988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:41 smithi116 ceph-mon[35411]: pgmap v3988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:46:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:44 smithi114 ceph-mon[30820]: pgmap v3989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:46:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:44 smithi116 ceph-mon[35411]: pgmap v3989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:46:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-mon[30820]: pgmap v3990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:46:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:46:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:46:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:46:46.484+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:46:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:46:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:46:46.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:46:46.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:46:46.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:46:46.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:46:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:46:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:46 smithi116 ceph-mon[35411]: pgmap v3990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:46:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:46:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:46:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:46:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:47 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:46:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:47 smithi114 ceph-mon[30820]: pgmap v3991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:46:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:47 smithi114 ceph-mon[30820]: pgmap v3992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:47 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:46:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:46:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:46:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:46:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:46:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:46:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:46:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:46:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:46:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:46:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:47 smithi116 ceph-mon[35411]: pgmap v3991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T08:46:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:47 smithi116 ceph-mon[35411]: pgmap v3992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:49 smithi116 ceph-mon[35411]: pgmap v3993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:49 smithi114 ceph-mon[30820]: pgmap v3993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:51 smithi116 ceph-mon[35411]: pgmap v3994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:52.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:51 smithi114 ceph-mon[30820]: pgmap v3994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:52 smithi116 ceph-mon[35411]: pgmap v3995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:53.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:52 smithi114 ceph-mon[30820]: pgmap v3995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:55.869 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:55 smithi114 ceph-mon[30820]: pgmap v3996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:55.900 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:55 smithi116 ceph-mon[35411]: pgmap v3996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:46:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:57 smithi114 ceph-mon[30820]: pgmap v3997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:46:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:57 smithi116 ceph-mon[35411]: pgmap v3997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:46:59.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:46:59 smithi114 ceph-mon[30820]: pgmap v3998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:46:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:46:59 smithi116 ceph-mon[35411]: pgmap v3998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:01.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:01 smithi114 ceph-mon[30820]: pgmap v3999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:01 smithi116 ceph-mon[35411]: pgmap v3999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:03.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:03 smithi114 ceph-mon[30820]: pgmap v4000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:03 smithi116 ceph-mon[35411]: pgmap v4000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:05.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:05 smithi114 ceph-mon[30820]: pgmap v4001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:05.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:05 smithi116 ceph-mon[35411]: pgmap v4001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:07.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:07 smithi114 ceph-mon[30820]: pgmap v4002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:07 smithi116 ceph-mon[35411]: pgmap v4002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:09 smithi114 ceph-mon[30820]: pgmap v4003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:09 smithi116 ceph-mon[35411]: pgmap v4003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:11.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:11 smithi114 ceph-mon[30820]: pgmap v4004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:11 smithi116 ceph-mon[35411]: pgmap v4004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:13 smithi114 ceph-mon[30820]: pgmap v4005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:13 smithi116 ceph-mon[35411]: pgmap v4005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:15 smithi114 ceph-mon[30820]: pgmap v4006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:15 smithi116 ceph-mon[35411]: pgmap v4006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:17 smithi114 ceph-mon[30820]: pgmap v4007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:17 smithi116 ceph-mon[35411]: pgmap v4007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:19 smithi114 ceph-mon[30820]: pgmap v4008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:19 smithi116 ceph-mon[35411]: pgmap v4008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:21 smithi114 ceph-mon[30820]: pgmap v4009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:21 smithi116 ceph-mon[35411]: pgmap v4009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:23 smithi114 ceph-mon[30820]: pgmap v4010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:23 smithi116 ceph-mon[35411]: pgmap v4010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:25.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:25 smithi114 ceph-mon[30820]: pgmap v4011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:25 smithi116 ceph-mon[35411]: pgmap v4011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:27.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:27 smithi114 ceph-mon[30820]: pgmap v4012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:27 smithi116 ceph-mon[35411]: pgmap v4012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:29.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:29 smithi114 ceph-mon[30820]: pgmap v4013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:29 smithi116 ceph-mon[35411]: pgmap v4013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:31.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:31 smithi114 ceph-mon[30820]: pgmap v4014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:31 smithi116 ceph-mon[35411]: pgmap v4014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:33 smithi114 ceph-mon[30820]: pgmap v4015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:33 smithi116 ceph-mon[35411]: pgmap v4015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:35.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:35 smithi114 ceph-mon[30820]: pgmap v4016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:35 smithi116 ceph-mon[35411]: pgmap v4016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:37.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:37 smithi114 ceph-mon[30820]: pgmap v4017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:37 smithi116 ceph-mon[35411]: pgmap v4017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:39.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:39 smithi114 ceph-mon[30820]: pgmap v4018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:39 smithi116 ceph-mon[35411]: pgmap v4018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:41.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:41 smithi114 ceph-mon[30820]: pgmap v4019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:41.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:41 smithi116 ceph-mon[35411]: pgmap v4019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:47:43.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:43 smithi114 ceph-mon[30820]: pgmap v4020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:43 smithi116 ceph-mon[35411]: pgmap v4020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:45.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:45 smithi114 ceph-mon[30820]: pgmap v4021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:45 smithi116 ceph-mon[35411]: pgmap v4021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:47:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:47:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:47:47.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:47 smithi114 ceph-mon[30820]: pgmap v4022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:47 smithi116 ceph-mon[35411]: pgmap v4022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:47:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:47.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:49.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-mon[30820]: pgmap v4023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:49.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:47:49.781+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:47:49.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:47:49.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:47:49.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:47:49.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:47:49.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:47:49.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:47:49.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:47:49.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:47:49.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:47:49.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:47:49.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:47:49.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:47:49.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:47:49.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:47:49.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:47:49.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:47:49.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:47:49.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:47:49.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:47:49.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:47:49.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:47:49.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:47:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:47:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:49 smithi116 ceph-mon[35411]: pgmap v4023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:47:50.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:47:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:47:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:47:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:47:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:47:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:47:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:47:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:47:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:47:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:47:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:47:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:47:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:47:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:47:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:47:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:47:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:47:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:51 smithi114 ceph-mon[30820]: pgmap v4024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T08:47:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:51 smithi114 ceph-mon[30820]: pgmap v4025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:47:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:47:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:47:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:47:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:47:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:47:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:51 smithi116 ceph-mon[35411]: pgmap v4024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T08:47:51.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:51 smithi116 ceph-mon[35411]: pgmap v4025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:47:53.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:53 smithi114 ceph-mon[30820]: pgmap v4026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:47:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:53 smithi116 ceph-mon[35411]: pgmap v4026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:47:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:55 smithi116 ceph-mon[35411]: pgmap v4027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:47:56.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:55 smithi114 ceph-mon[30820]: pgmap v4027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:47:57.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:57 smithi114 ceph-mon[30820]: pgmap v4028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:47:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:57 smithi116 ceph-mon[35411]: pgmap v4028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:47:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:47:59 smithi116 ceph-mon[35411]: pgmap v4029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:48:00.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:47:59 smithi114 ceph-mon[30820]: pgmap v4029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:48:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:01 smithi114 ceph-mon[30820]: pgmap v4030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:48:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:01 smithi116 ceph-mon[35411]: pgmap v4030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:48:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:03 smithi114 ceph-mon[30820]: pgmap v4031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:03 smithi116 ceph-mon[35411]: pgmap v4031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:05.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:04 smithi114 ceph-mon[30820]: pgmap v4032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:05.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:04 smithi116 ceph-mon[35411]: pgmap v4032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:07.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:06 smithi114 ceph-mon[30820]: pgmap v4033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:06 smithi116 ceph-mon[35411]: pgmap v4033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:09.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:08 smithi114 ceph-mon[30820]: pgmap v4034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:08 smithi116 ceph-mon[35411]: pgmap v4034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:11.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:10 smithi114 ceph-mon[30820]: pgmap v4035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:11.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:10 smithi116 ceph-mon[35411]: pgmap v4035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:12 smithi114 ceph-mon[30820]: pgmap v4036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:12 smithi116 ceph-mon[35411]: pgmap v4036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:15.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:14 smithi114 ceph-mon[30820]: pgmap v4037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:15.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:14 smithi116 ceph-mon[35411]: pgmap v4037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:17.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:16 smithi114 ceph-mon[30820]: pgmap v4038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:17.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:16 smithi116 ceph-mon[35411]: pgmap v4038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:19.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:18 smithi114 ceph-mon[30820]: pgmap v4039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:19.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:18 smithi116 ceph-mon[35411]: pgmap v4039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:21.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:20 smithi114 ceph-mon[30820]: pgmap v4040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:21.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:20 smithi116 ceph-mon[35411]: pgmap v4040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:23.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:22 smithi114 ceph-mon[30820]: pgmap v4041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:23.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:22 smithi116 ceph-mon[35411]: pgmap v4041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:25.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:24 smithi114 ceph-mon[30820]: pgmap v4042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:25.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:24 smithi116 ceph-mon[35411]: pgmap v4042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:27.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:26 smithi114 ceph-mon[30820]: pgmap v4043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:27.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:26 smithi116 ceph-mon[35411]: pgmap v4043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:29.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:28 smithi114 ceph-mon[30820]: pgmap v4044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:29.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:28 smithi116 ceph-mon[35411]: pgmap v4044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:31.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:30 smithi114 ceph-mon[30820]: pgmap v4045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:31.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:30 smithi116 ceph-mon[35411]: pgmap v4045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:33.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:32 smithi114 ceph-mon[30820]: pgmap v4046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:32 smithi116 ceph-mon[35411]: pgmap v4046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:35.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:34 smithi114 ceph-mon[30820]: pgmap v4047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:35.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:34 smithi116 ceph-mon[35411]: pgmap v4047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:36 smithi114 ceph-mon[30820]: pgmap v4048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:36 smithi116 ceph-mon[35411]: pgmap v4048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:39.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:38 smithi114 ceph-mon[30820]: pgmap v4049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:39.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:38 smithi116 ceph-mon[35411]: pgmap v4049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:41.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:40 smithi114 ceph-mon[30820]: pgmap v4050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:41.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:40 smithi116 ceph-mon[35411]: pgmap v4050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:43.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:42 smithi114 ceph-mon[30820]: pgmap v4051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:42 smithi116 ceph-mon[35411]: pgmap v4051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:45.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:44 smithi116 ceph-mon[35411]: pgmap v4052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:45.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:44 smithi114 ceph-mon[30820]: pgmap v4052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:46 smithi116 ceph-mon[35411]: pgmap v4053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:47.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:46 smithi114 ceph-mon[30820]: pgmap v4053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:48:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:48 smithi116 ceph-mon[35411]: pgmap v4054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:49.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:48 smithi114 ceph-mon[30820]: pgmap v4054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:48:50.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:48:50.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:48:51.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:50 smithi114 ceph-mon[30820]: pgmap v4055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:50 smithi116 ceph-mon[35411]: pgmap v4055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:48:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:52 smithi114 ceph-mon[30820]: pgmap v4056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:53.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:48:53.140+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:48:53.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:48:53.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:48:53.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:48:53.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:48:53.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:48:53.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:48:53.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:48:53.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:48:53.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:48:53.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:48:53.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:48:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:52 smithi116 ceph-mon[35411]: pgmap v4056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:48:54.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:48:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:48:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:48:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:48:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:48:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:48:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-mon[30820]: pgmap v4057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:48:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-mon[30820]: pgmap v4058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:48:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:53 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:48:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:48:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:53 smithi116 ceph-mon[35411]: pgmap v4057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:48:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:53 smithi116 ceph-mon[35411]: pgmap v4058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:48:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:48:56.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:56 smithi114 ceph-mon[30820]: pgmap v4059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:48:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:56 smithi116 ceph-mon[35411]: pgmap v4059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:48:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:48:58 smithi114 ceph-mon[30820]: pgmap v4060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:48:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:48:58 smithi116 ceph-mon[35411]: pgmap v4060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:49:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:00 smithi114 ceph-mon[30820]: pgmap v4061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:49:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:00 smithi116 ceph-mon[35411]: pgmap v4061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:49:02.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:01 smithi114 ceph-mon[30820]: pgmap v4062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:49:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:01 smithi116 ceph-mon[35411]: pgmap v4062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:49:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:04 smithi114 ceph-mon[30820]: pgmap v4063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:49:04.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:04 smithi116 ceph-mon[35411]: pgmap v4063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:49:06.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:06 smithi114 ceph-mon[30820]: pgmap v4064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:06 smithi116 ceph-mon[35411]: pgmap v4064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:08.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:08 smithi114 ceph-mon[30820]: pgmap v4065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:08.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:08 smithi116 ceph-mon[35411]: pgmap v4065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:10.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:10 smithi114 ceph-mon[30820]: pgmap v4066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:10.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:10 smithi116 ceph-mon[35411]: pgmap v4066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:12.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:12 smithi114 ceph-mon[30820]: pgmap v4067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:12.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:12 smithi116 ceph-mon[35411]: pgmap v4067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:14.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:14 smithi114 ceph-mon[30820]: pgmap v4068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:14 smithi116 ceph-mon[35411]: pgmap v4068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:16 smithi114 ceph-mon[30820]: pgmap v4069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:16 smithi116 ceph-mon[35411]: pgmap v4069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:18 smithi114 ceph-mon[30820]: pgmap v4070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:18.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:18 smithi116 ceph-mon[35411]: pgmap v4070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:20 smithi114 ceph-mon[30820]: pgmap v4071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:20 smithi116 ceph-mon[35411]: pgmap v4071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:22 smithi114 ceph-mon[30820]: pgmap v4072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:22 smithi116 ceph-mon[35411]: pgmap v4072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:24 smithi114 ceph-mon[30820]: pgmap v4073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:24 smithi116 ceph-mon[35411]: pgmap v4073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:26 smithi114 ceph-mon[30820]: pgmap v4074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:26 smithi116 ceph-mon[35411]: pgmap v4074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:28 smithi114 ceph-mon[30820]: pgmap v4075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:28 smithi116 ceph-mon[35411]: pgmap v4075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:30 smithi114 ceph-mon[30820]: pgmap v4076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:30 smithi116 ceph-mon[35411]: pgmap v4076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:32.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:32 smithi114 ceph-mon[30820]: pgmap v4077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:32 smithi116 ceph-mon[35411]: pgmap v4077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:34 smithi114 ceph-mon[30820]: pgmap v4078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:34 smithi116 ceph-mon[35411]: pgmap v4078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:36 smithi114 ceph-mon[30820]: pgmap v4079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:36 smithi116 ceph-mon[35411]: pgmap v4079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:38 smithi114 ceph-mon[30820]: pgmap v4080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:38 smithi116 ceph-mon[35411]: pgmap v4080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:40.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:40 smithi114 ceph-mon[30820]: pgmap v4081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:40 smithi116 ceph-mon[35411]: pgmap v4081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:42.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:41 smithi114 ceph-mon[30820]: pgmap v4082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:42.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:41 smithi116 ceph-mon[35411]: pgmap v4082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:44 smithi114 ceph-mon[30820]: pgmap v4083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:44 smithi116 ceph-mon[35411]: pgmap v4083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:46 smithi114 ceph-mon[30820]: pgmap v4084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:46 smithi116 ceph-mon[35411]: pgmap v4084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:48.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:48 smithi114 ceph-mon[30820]: pgmap v4085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:48 smithi116 ceph-mon[35411]: pgmap v4085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:50 smithi114 ceph-mon[30820]: pgmap v4086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:50 smithi116 ceph-mon[35411]: pgmap v4086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:52.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:52 smithi114 ceph-mon[30820]: pgmap v4087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:52 smithi116 ceph-mon[35411]: pgmap v4087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:49:54.367 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:54 smithi116 ceph-mon[35411]: pgmap v4088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:54.367 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:49:54.600 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:54 smithi114 ceph-mon[30820]: pgmap v4088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:49:54.600 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:49:56.339 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-mon[30820]: pgmap v4089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:56.600 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:56 smithi116 ceph-mon[35411]: pgmap v4089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:49:56.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:49:56.500+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:49:56.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:49:56.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:49:56.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:49:56.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:49:56.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:49:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:49:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:49:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:49:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:49:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:49:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:49:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:49:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:49:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:57 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:49:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:57.523 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:57 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:49:57.523 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:49:57.523 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:49:57.524 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:57.525 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:58 smithi114 ceph-mon[30820]: pgmap v4090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:49:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:49:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:58 smithi116 ceph-mon[35411]: pgmap v4090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:49:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:49:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:49:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:49:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:59 smithi114 ceph-mon[30820]: pgmap v4091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:49:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:49:59 smithi114 ceph-mon[30820]: pgmap v4092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:49:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:59 smithi116 ceph-mon[35411]: pgmap v4091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:49:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:49:59 smithi116 ceph-mon[35411]: pgmap v4092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:50:00.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:50:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T08:50:01.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:01 smithi114 ceph-mon[30820]: pgmap v4093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 6 op/s 2024-09-19T08:50:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:01 smithi116 ceph-mon[35411]: pgmap v4093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 6 op/s 2024-09-19T08:50:03.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:03 smithi114 ceph-mon[30820]: pgmap v4094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:50:03.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:03 smithi116 ceph-mon[35411]: pgmap v4094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:50:05.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:05 smithi114 ceph-mon[30820]: pgmap v4095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T08:50:05.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:05 smithi116 ceph-mon[35411]: pgmap v4095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T08:50:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:07 smithi116 ceph-mon[35411]: pgmap v4096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:50:07.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:07 smithi114 ceph-mon[30820]: pgmap v4096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T08:50:09.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:09 smithi114 ceph-mon[30820]: pgmap v4097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:50:09.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:09 smithi116 ceph-mon[35411]: pgmap v4097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:50:11.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:11 smithi114 ceph-mon[30820]: pgmap v4098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:11.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:11 smithi116 ceph-mon[35411]: pgmap v4098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:13.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:13 smithi114 ceph-mon[30820]: pgmap v4099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:13 smithi116 ceph-mon[35411]: pgmap v4099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:15.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:15 smithi114 ceph-mon[30820]: pgmap v4100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:15 smithi116 ceph-mon[35411]: pgmap v4100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:17.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:17 smithi114 ceph-mon[30820]: pgmap v4101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:17.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:17 smithi116 ceph-mon[35411]: pgmap v4101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:19 smithi114 ceph-mon[30820]: pgmap v4102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:19 smithi116 ceph-mon[35411]: pgmap v4102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:21 smithi114 ceph-mon[30820]: pgmap v4103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:21 smithi116 ceph-mon[35411]: pgmap v4103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:23.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:23 smithi114 ceph-mon[30820]: pgmap v4104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:23 smithi116 ceph-mon[35411]: pgmap v4104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:25.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:25 smithi114 ceph-mon[30820]: pgmap v4105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:25 smithi116 ceph-mon[35411]: pgmap v4105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:27.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:27 smithi114 ceph-mon[30820]: pgmap v4106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:27.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:27 smithi116 ceph-mon[35411]: pgmap v4106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:29 smithi114 ceph-mon[30820]: pgmap v4107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:29 smithi116 ceph-mon[35411]: pgmap v4107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:31.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:31 smithi114 ceph-mon[30820]: pgmap v4108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:31 smithi116 ceph-mon[35411]: pgmap v4108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:33 smithi114 ceph-mon[30820]: pgmap v4109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:33.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:33 smithi116 ceph-mon[35411]: pgmap v4109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:35.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:35 smithi114 ceph-mon[30820]: pgmap v4110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:35.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:35 smithi116 ceph-mon[35411]: pgmap v4110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:37.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:37 smithi114 ceph-mon[30820]: pgmap v4111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:37 smithi116 ceph-mon[35411]: pgmap v4111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:39 smithi114 ceph-mon[30820]: pgmap v4112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:39 smithi116 ceph-mon[35411]: pgmap v4112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:41.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:41 smithi114 ceph-mon[30820]: pgmap v4113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:41 smithi116 ceph-mon[35411]: pgmap v4113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:43.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:43 smithi114 ceph-mon[30820]: pgmap v4114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:43 smithi116 ceph-mon[35411]: pgmap v4114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:45.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:45 smithi114 ceph-mon[30820]: pgmap v4115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:45.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:45 smithi116 ceph-mon[35411]: pgmap v4115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:47.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:47 smithi114 ceph-mon[30820]: pgmap v4116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:47 smithi116 ceph-mon[35411]: pgmap v4116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:49.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:49 smithi114 ceph-mon[30820]: pgmap v4117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:49 smithi116 ceph-mon[35411]: pgmap v4117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:51 smithi114 ceph-mon[30820]: pgmap v4118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:51.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:51 smithi116 ceph-mon[35411]: pgmap v4118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:50:53.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:53 smithi114 ceph-mon[30820]: pgmap v4119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:53.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:53 smithi116 ceph-mon[35411]: pgmap v4119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:50:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:55 smithi114 ceph-mon[30820]: pgmap v4120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:55.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:55 smithi116 ceph-mon[35411]: pgmap v4120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:50:57.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:57 smithi114 ceph-mon[30820]: pgmap v4121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:57.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:57 smithi116 ceph-mon[35411]: pgmap v4121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:50:58.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:50:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:50:59.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:50:59 smithi114 ceph-mon[30820]: pgmap v4122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:50:59.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:50:59 smithi116 ceph-mon[35411]: pgmap v4122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:01.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-mon[30820]: pgmap v4123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:51:01.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:51:01.524+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:51:01.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:51:01.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:51:01.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:51:01.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:51:01.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:51:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:51:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:51:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:51:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:51:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:51:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:51:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:01 smithi116 ceph-mon[35411]: pgmap v4123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:51:02.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:02 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:51:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:02 smithi114 ceph-mon[30820]: pgmap v4124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:02 smithi114 ceph-mon[30820]: pgmap v4125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:51:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:51:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:02 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:51:02.677 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:51:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:02 smithi116 ceph-mon[35411]: pgmap v4124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:02 smithi116 ceph-mon[35411]: pgmap v4125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:51:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:51:04.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:04 smithi114 ceph-mon[30820]: pgmap v4126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:51:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:04 smithi116 ceph-mon[35411]: pgmap v4126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:51:06.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:06 smithi114 ceph-mon[30820]: pgmap v4127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:51:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:06 smithi116 ceph-mon[35411]: pgmap v4127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:51:08.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:08 smithi114 ceph-mon[30820]: pgmap v4128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:51:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:08 smithi116 ceph-mon[35411]: pgmap v4128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:51:10.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:10 smithi114 ceph-mon[30820]: pgmap v4129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T08:51:10.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:10 smithi116 ceph-mon[35411]: pgmap v4129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T08:51:12.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:12 smithi114 ceph-mon[30820]: pgmap v4130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:51:12.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:12 smithi116 ceph-mon[35411]: pgmap v4130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:51:14.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:14 smithi114 ceph-mon[30820]: pgmap v4131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:14.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:14 smithi116 ceph-mon[35411]: pgmap v4131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:16.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:16 smithi114 ceph-mon[30820]: pgmap v4132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:16 smithi116 ceph-mon[35411]: pgmap v4132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:18 smithi114 ceph-mon[30820]: pgmap v4133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:18.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:18 smithi116 ceph-mon[35411]: pgmap v4133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:20 smithi114 ceph-mon[30820]: pgmap v4134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:51:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:20 smithi116 ceph-mon[35411]: pgmap v4134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:51:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:22 smithi114 ceph-mon[30820]: pgmap v4135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:51:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:22 smithi116 ceph-mon[35411]: pgmap v4135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:51:24.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:24 smithi114 ceph-mon[30820]: pgmap v4136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:24 smithi116 ceph-mon[35411]: pgmap v4136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:26.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:26 smithi114 ceph-mon[30820]: pgmap v4137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:26 smithi116 ceph-mon[35411]: pgmap v4137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:28 smithi114 ceph-mon[30820]: pgmap v4138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:28 smithi116 ceph-mon[35411]: pgmap v4138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:30.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:30 smithi114 ceph-mon[30820]: pgmap v4139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:51:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:30 smithi116 ceph-mon[35411]: pgmap v4139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:51:32.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:32 smithi114 ceph-mon[30820]: pgmap v4140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:51:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:32 smithi116 ceph-mon[35411]: pgmap v4140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:51:34.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:34 smithi114 ceph-mon[30820]: pgmap v4141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:34.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:34 smithi116 ceph-mon[35411]: pgmap v4141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:36 smithi114 ceph-mon[30820]: pgmap v4142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:36 smithi116 ceph-mon[35411]: pgmap v4142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:38.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:38 smithi114 ceph-mon[30820]: pgmap v4143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:38 smithi116 ceph-mon[35411]: pgmap v4143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:40.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:40 smithi114 ceph-mon[30820]: pgmap v4144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:51:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:40 smithi116 ceph-mon[35411]: pgmap v4144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:51:42.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:42 smithi114 ceph-mon[30820]: pgmap v4145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:51:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:42 smithi116 ceph-mon[35411]: pgmap v4145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:51:44.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:44 smithi114 ceph-mon[30820]: pgmap v4146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:44 smithi116 ceph-mon[35411]: pgmap v4146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:46 smithi114 ceph-mon[30820]: pgmap v4147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:46 smithi116 ceph-mon[35411]: pgmap v4147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:48.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:48 smithi116 ceph-mon[35411]: pgmap v4148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:49.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:48 smithi114 ceph-mon[30820]: pgmap v4148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:50 smithi116 ceph-mon[35411]: pgmap v4149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:51:51.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:50 smithi114 ceph-mon[30820]: pgmap v4149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:51:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:52 smithi116 ceph-mon[35411]: pgmap v4150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:51:53.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:52 smithi114 ceph-mon[30820]: pgmap v4150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:51:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:54 smithi116 ceph-mon[35411]: pgmap v4151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:55.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:54 smithi114 ceph-mon[30820]: pgmap v4151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:56 smithi114 ceph-mon[30820]: pgmap v4152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:57.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:56 smithi116 ceph-mon[35411]: pgmap v4152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:51:58.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:51:58 smithi114 ceph-mon[30820]: pgmap v4153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:51:58.979 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:51:58 smithi116 ceph-mon[35411]: pgmap v4153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:01.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:00 smithi114 ceph-mon[30820]: pgmap v4154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:00 smithi116 ceph-mon[35411]: pgmap v4154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:52:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:52:03.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:02 smithi114 ceph-mon[30820]: pgmap v4155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:03.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:02 smithi116 ceph-mon[35411]: pgmap v4155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:03.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:52:03.284+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:52:03.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:52:03.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:52:03.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:52:03.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:52:03.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:52:03.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:52:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:52:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:52:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:52:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:52:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:52:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:52:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:52:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:52:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:52:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:52:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:52:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:04 smithi114 ceph-mon[30820]: pgmap v4156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:52:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:04 smithi114 ceph-mon[30820]: pgmap v4157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:52:05.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:52:05.177 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:52:05.177 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:52:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:04 smithi116 ceph-mon[35411]: pgmap v4156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:04 smithi116 ceph-mon[35411]: pgmap v4157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:52:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:06 smithi114 ceph-mon[30820]: pgmap v4158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:52:07.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:06 smithi116 ceph-mon[35411]: pgmap v4158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:52:09.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:08 smithi114 ceph-mon[30820]: pgmap v4159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:52:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:08 smithi116 ceph-mon[35411]: pgmap v4159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:52:11.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:10 smithi114 ceph-mon[30820]: pgmap v4160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:52:11.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:10 smithi116 ceph-mon[35411]: pgmap v4160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:52:13.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:12 smithi114 ceph-mon[30820]: pgmap v4161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:52:13.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:12 smithi116 ceph-mon[35411]: pgmap v4161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:52:15.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:14 smithi114 ceph-mon[30820]: pgmap v4162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:52:15.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:14 smithi116 ceph-mon[35411]: pgmap v4162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:52:17.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:16 smithi114 ceph-mon[30820]: pgmap v4163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:17.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:16 smithi116 ceph-mon[35411]: pgmap v4163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:18.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:17 smithi114 ceph-mon[30820]: pgmap v4164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:17 smithi116 ceph-mon[35411]: pgmap v4164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:20 smithi114 ceph-mon[30820]: pgmap v4165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:20 smithi116 ceph-mon[35411]: pgmap v4165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:22.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:22 smithi114 ceph-mon[30820]: pgmap v4166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:22 smithi116 ceph-mon[35411]: pgmap v4166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:24 smithi114 ceph-mon[30820]: pgmap v4167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:24 smithi116 ceph-mon[35411]: pgmap v4167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:26.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:26 smithi114 ceph-mon[30820]: pgmap v4168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:26 smithi116 ceph-mon[35411]: pgmap v4168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:28 smithi114 ceph-mon[30820]: pgmap v4169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:28 smithi116 ceph-mon[35411]: pgmap v4169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:30 smithi114 ceph-mon[30820]: pgmap v4170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:30 smithi116 ceph-mon[35411]: pgmap v4170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:32 smithi114 ceph-mon[30820]: pgmap v4171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:32 smithi116 ceph-mon[35411]: pgmap v4171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:34 smithi114 ceph-mon[30820]: pgmap v4172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:34 smithi116 ceph-mon[35411]: pgmap v4172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:36 smithi114 ceph-mon[30820]: pgmap v4173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:36 smithi116 ceph-mon[35411]: pgmap v4173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:38 smithi114 ceph-mon[30820]: pgmap v4174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:38 smithi116 ceph-mon[35411]: pgmap v4174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:40.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:40 smithi114 ceph-mon[30820]: pgmap v4175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:40 smithi116 ceph-mon[35411]: pgmap v4175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:42 smithi114 ceph-mon[30820]: pgmap v4176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:42 smithi116 ceph-mon[35411]: pgmap v4176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:44 smithi114 ceph-mon[30820]: pgmap v4177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:44 smithi116 ceph-mon[35411]: pgmap v4177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:46 smithi114 ceph-mon[30820]: pgmap v4178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:46 smithi116 ceph-mon[35411]: pgmap v4178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:48.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:48 smithi114 ceph-mon[30820]: pgmap v4179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:48 smithi116 ceph-mon[35411]: pgmap v4179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:50.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:50 smithi114 ceph-mon[30820]: pgmap v4180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:50 smithi116 ceph-mon[35411]: pgmap v4180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:52:52.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:52 smithi114 ceph-mon[30820]: pgmap v4181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:52 smithi116 ceph-mon[35411]: pgmap v4181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:52:54.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:54 smithi116 ceph-mon[35411]: pgmap v4182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:54 smithi114 ceph-mon[30820]: pgmap v4182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:52:56.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:56 smithi116 ceph-mon[35411]: pgmap v4183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:56 smithi114 ceph-mon[30820]: pgmap v4183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:52:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:52:58 smithi116 ceph-mon[35411]: pgmap v4184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:52:58.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:52:58 smithi114 ceph-mon[30820]: pgmap v4184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:53:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:00 smithi114 ceph-mon[30820]: pgmap v4185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:00 smithi116 ceph-mon[35411]: pgmap v4185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:02.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:01 smithi114 ceph-mon[30820]: pgmap v4186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:02.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:01 smithi116 ceph-mon[35411]: pgmap v4186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:04.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:04 smithi114 ceph-mon[30820]: pgmap v4187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:04.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:53:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:04 smithi116 ceph-mon[35411]: pgmap v4187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:04.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:53:06.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-mon[30820]: pgmap v4188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:53:06.580+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:53:06.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:53:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:53:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:53:06.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:06 smithi116 ceph-mon[35411]: pgmap v4188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:53:07.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:53:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:53:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:07 smithi114 ceph-mon[30820]: pgmap v4189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T08:53:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:07 smithi114 ceph-mon[30820]: pgmap v4190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:53:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:53:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:53:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:53:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:07 smithi116 ceph-mon[35411]: pgmap v4189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T08:53:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:07 smithi116 ceph-mon[35411]: pgmap v4190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:53:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:53:09.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:09 smithi114 ceph-mon[30820]: pgmap v4191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:53:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:09 smithi116 ceph-mon[35411]: pgmap v4191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:53:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:11 smithi114 ceph-mon[30820]: pgmap v4192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:53:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:11 smithi116 ceph-mon[35411]: pgmap v4192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:53:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:13 smithi114 ceph-mon[30820]: pgmap v4193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:53:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:13 smithi116 ceph-mon[35411]: pgmap v4193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:53:15.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:15 smithi114 ceph-mon[30820]: pgmap v4194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T08:53:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:15 smithi116 ceph-mon[35411]: pgmap v4194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T08:53:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:17 smithi114 ceph-mon[30820]: pgmap v4195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:53:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:17 smithi116 ceph-mon[35411]: pgmap v4195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:53:19.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:19 smithi114 ceph-mon[30820]: pgmap v4196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:19.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:19 smithi116 ceph-mon[35411]: pgmap v4196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:21 smithi114 ceph-mon[30820]: pgmap v4197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:53:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:21 smithi116 ceph-mon[35411]: pgmap v4197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:53:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:23 smithi116 ceph-mon[35411]: pgmap v4198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:24.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:23 smithi114 ceph-mon[30820]: pgmap v4198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:25 smithi116 ceph-mon[35411]: pgmap v4199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:26.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:25 smithi114 ceph-mon[30820]: pgmap v4199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:27 smithi116 ceph-mon[35411]: pgmap v4200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:28.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:27 smithi114 ceph-mon[30820]: pgmap v4200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:29.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:29 smithi114 ceph-mon[30820]: pgmap v4201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:29 smithi116 ceph-mon[35411]: pgmap v4201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:31 smithi114 ceph-mon[30820]: pgmap v4202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:53:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:31 smithi116 ceph-mon[35411]: pgmap v4202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:53:34.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:33 smithi114 ceph-mon[30820]: pgmap v4203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:33 smithi116 ceph-mon[35411]: pgmap v4203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:36.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:35 smithi114 ceph-mon[30820]: pgmap v4204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:35 smithi116 ceph-mon[35411]: pgmap v4204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:38.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:37 smithi114 ceph-mon[30820]: pgmap v4205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:37 smithi116 ceph-mon[35411]: pgmap v4205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:40.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:39 smithi114 ceph-mon[30820]: pgmap v4206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:39 smithi116 ceph-mon[35411]: pgmap v4206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:41 smithi114 ceph-mon[30820]: pgmap v4207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:53:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:41 smithi116 ceph-mon[35411]: pgmap v4207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:53:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:43 smithi114 ceph-mon[30820]: pgmap v4208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:43 smithi116 ceph-mon[35411]: pgmap v4208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:45 smithi114 ceph-mon[30820]: pgmap v4209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:45 smithi116 ceph-mon[35411]: pgmap v4209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:48.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:47 smithi114 ceph-mon[30820]: pgmap v4210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:47 smithi116 ceph-mon[35411]: pgmap v4210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:49 smithi114 ceph-mon[30820]: pgmap v4211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:50.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:49 smithi116 ceph-mon[35411]: pgmap v4211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:51 smithi114 ceph-mon[30820]: pgmap v4212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:53:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:51 smithi116 ceph-mon[35411]: pgmap v4212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:53:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:53 smithi114 ceph-mon[30820]: pgmap v4213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:53 smithi116 ceph-mon[35411]: pgmap v4213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:53:56.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:55 smithi114 ceph-mon[30820]: pgmap v4214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:55 smithi116 ceph-mon[35411]: pgmap v4214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:53:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:57 smithi114 ceph-mon[30820]: pgmap v4215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:53:58.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:57 smithi116 ceph-mon[35411]: pgmap v4215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:00.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:53:59 smithi114 ceph-mon[30820]: pgmap v4216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:53:59 smithi116 ceph-mon[35411]: pgmap v4216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:01 smithi114 ceph-mon[30820]: pgmap v4217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:01 smithi116 ceph-mon[35411]: pgmap v4217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:03 smithi114 ceph-mon[30820]: pgmap v4218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:03 smithi116 ceph-mon[35411]: pgmap v4218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:05 smithi114 ceph-mon[30820]: pgmap v4219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:54:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:05 smithi116 ceph-mon[35411]: pgmap v4219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:54:07.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:06 smithi114 ceph-mon[30820]: pgmap v4220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:07.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:54:07.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:06 smithi116 ceph-mon[35411]: pgmap v4220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:07.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:54:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:07 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:54:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T08:54:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:07 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T08:54:09.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:08 smithi114 ceph-mon[30820]: pgmap v4221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:08 smithi116 ceph-mon[35411]: pgmap v4221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:54:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:54:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:54:10.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:54:09.941+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:54:10.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:54:10.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:54:10.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:54:10.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:54:10.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:54:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:54:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:54:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:54:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:54:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:54:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:54:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:54:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:54:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:54:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:54:11.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:11 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:54:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:11 smithi114 ceph-mon[30820]: pgmap v4222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:11 smithi114 ceph-mon[30820]: pgmap v4223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:54:11.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:11 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:54:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:11 smithi116 ceph-mon[35411]: pgmap v4222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:54:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:11 smithi116 ceph-mon[35411]: pgmap v4223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:54:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:13 smithi114 ceph-mon[30820]: pgmap v4224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:54:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:13 smithi116 ceph-mon[35411]: pgmap v4224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:54:15.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:15 smithi114 ceph-mon[30820]: pgmap v4225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T08:54:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:15 smithi116 ceph-mon[35411]: pgmap v4225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T08:54:17.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:17 smithi114 ceph-mon[30820]: pgmap v4226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:54:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:17 smithi116 ceph-mon[35411]: pgmap v4226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:54:19.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:19 smithi114 ceph-mon[30820]: pgmap v4227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:54:19.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:19 smithi116 ceph-mon[35411]: pgmap v4227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T08:54:21.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:21 smithi114 ceph-mon[30820]: pgmap v4228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:54:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:21 smithi116 ceph-mon[35411]: pgmap v4228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:54:23.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:23 smithi114 ceph-mon[30820]: pgmap v4229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:23.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:23 smithi116 ceph-mon[35411]: pgmap v4229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:25.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:25 smithi114 ceph-mon[30820]: pgmap v4230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:25 smithi116 ceph-mon[35411]: pgmap v4230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:27.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:27 smithi114 ceph-mon[30820]: pgmap v4231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:27.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:27 smithi116 ceph-mon[35411]: pgmap v4231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:29.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:29 smithi114 ceph-mon[30820]: pgmap v4232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:29.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:29 smithi116 ceph-mon[35411]: pgmap v4232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:31.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:31 smithi114 ceph-mon[30820]: pgmap v4233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:54:31.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:31 smithi116 ceph-mon[35411]: pgmap v4233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:54:33.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:33 smithi114 ceph-mon[30820]: pgmap v4234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:33.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:33 smithi116 ceph-mon[35411]: pgmap v4234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:35.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:35 smithi114 ceph-mon[30820]: pgmap v4235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:35.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:35 smithi116 ceph-mon[35411]: pgmap v4235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:37.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:37 smithi114 ceph-mon[30820]: pgmap v4236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:37.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:37 smithi116 ceph-mon[35411]: pgmap v4236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:39.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:39 smithi114 ceph-mon[30820]: pgmap v4237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:39.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:39 smithi116 ceph-mon[35411]: pgmap v4237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:41.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:41 smithi114 ceph-mon[30820]: pgmap v4238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:54:41.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:41 smithi116 ceph-mon[35411]: pgmap v4238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:54:43.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:43 smithi114 ceph-mon[30820]: pgmap v4239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:43.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:43 smithi116 ceph-mon[35411]: pgmap v4239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:45.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:45 smithi114 ceph-mon[30820]: pgmap v4240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:45.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:45 smithi116 ceph-mon[35411]: pgmap v4240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:47.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:47 smithi114 ceph-mon[30820]: pgmap v4241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:47.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:47 smithi116 ceph-mon[35411]: pgmap v4241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:49.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:49 smithi114 ceph-mon[30820]: pgmap v4242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:49.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:49 smithi116 ceph-mon[35411]: pgmap v4242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:51.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:51 smithi114 ceph-mon[30820]: pgmap v4243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:54:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:51 smithi116 ceph-mon[35411]: pgmap v4243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:54:53.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:53 smithi114 ceph-mon[30820]: pgmap v4244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:53 smithi116 ceph-mon[35411]: pgmap v4244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:54:55.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:55 smithi114 ceph-mon[30820]: pgmap v4245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:55.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:55 smithi116 ceph-mon[35411]: pgmap v4245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:57.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:57 smithi114 ceph-mon[30820]: pgmap v4246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:57 smithi116 ceph-mon[35411]: pgmap v4246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:54:59.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:54:59 smithi114 ceph-mon[30820]: pgmap v4247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:54:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:54:59 smithi116 ceph-mon[35411]: pgmap v4247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:01.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:01 smithi114 ceph-mon[30820]: pgmap v4248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:01 smithi116 ceph-mon[35411]: pgmap v4248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:03.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:03 smithi114 ceph-mon[30820]: pgmap v4249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:03 smithi116 ceph-mon[35411]: pgmap v4249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:05.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:05 smithi114 ceph-mon[30820]: pgmap v4250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:05 smithi116 ceph-mon[35411]: pgmap v4250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:07.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:07 smithi114 ceph-mon[30820]: pgmap v4251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:55:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:07 smithi116 ceph-mon[35411]: pgmap v4251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:55:09.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:09 smithi114 ceph-mon[30820]: pgmap v4252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:09 smithi116 ceph-mon[35411]: pgmap v4252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:10.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:55:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:55:11.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:11 smithi114 ceph-mon[30820]: pgmap v4253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:11.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:11 smithi116 ceph-mon[35411]: pgmap v4253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-mon[30820]: pgmap v4254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:55:13.300+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:55:13.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:55:13.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:55:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:55:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:13 smithi116 ceph-mon[35411]: pgmap v4254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:55:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:55:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:55:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:55:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:55:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:14 smithi114 ceph-mon[30820]: pgmap v4255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:55:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:14 smithi114 ceph-mon[30820]: pgmap v4256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:55:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:55:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:55:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:55:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:55:14.427 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:55:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:14 smithi116 ceph-mon[35411]: pgmap v4255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:14 smithi116 ceph-mon[35411]: pgmap v4256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:55:16.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:16 smithi114 ceph-mon[30820]: pgmap v4257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:55:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:16 smithi116 ceph-mon[35411]: pgmap v4257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:55:18.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:18 smithi114 ceph-mon[30820]: pgmap v4258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:55:18.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:18 smithi116 ceph-mon[35411]: pgmap v4258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:55:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:20 smithi114 ceph-mon[30820]: pgmap v4259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:55:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:20 smithi116 ceph-mon[35411]: pgmap v4259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:55:22.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:22 smithi114 ceph-mon[30820]: pgmap v4260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:55:22.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:22 smithi116 ceph-mon[35411]: pgmap v4260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:55:24.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:24 smithi114 ceph-mon[30820]: pgmap v4261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:55:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:24 smithi116 ceph-mon[35411]: pgmap v4261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:26 smithi114 ceph-mon[30820]: pgmap v4262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:55:26.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:26 smithi116 ceph-mon[35411]: pgmap v4262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:55:28.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:28 smithi114 ceph-mon[30820]: pgmap v4263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:55:28.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:28 smithi116 ceph-mon[35411]: pgmap v4263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:55:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:30 smithi114 ceph-mon[30820]: pgmap v4264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:30 smithi116 ceph-mon[35411]: pgmap v4264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:32 smithi114 ceph-mon[30820]: pgmap v4265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:32 smithi116 ceph-mon[35411]: pgmap v4265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:34.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:34 smithi114 ceph-mon[30820]: pgmap v4266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:34 smithi116 ceph-mon[35411]: pgmap v4266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:36.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:36 smithi114 ceph-mon[30820]: pgmap v4267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:55:36.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:36 smithi116 ceph-mon[35411]: pgmap v4267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:55:38.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:38 smithi114 ceph-mon[30820]: pgmap v4268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:55:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:38 smithi116 ceph-mon[35411]: pgmap v4268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:55:40.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:40 smithi114 ceph-mon[30820]: pgmap v4269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:40 smithi116 ceph-mon[35411]: pgmap v4269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:42.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:42 smithi114 ceph-mon[30820]: pgmap v4270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:42.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:42 smithi116 ceph-mon[35411]: pgmap v4270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:44.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:44 smithi114 ceph-mon[30820]: pgmap v4271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:44 smithi116 ceph-mon[35411]: pgmap v4271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:46.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:46 smithi114 ceph-mon[30820]: pgmap v4272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:55:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:46 smithi116 ceph-mon[35411]: pgmap v4272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:55:48.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:48 smithi116 ceph-mon[35411]: pgmap v4273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:55:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:48 smithi114 ceph-mon[30820]: pgmap v4273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:55:50.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:50 smithi116 ceph-mon[35411]: pgmap v4274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:50.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:50 smithi114 ceph-mon[30820]: pgmap v4274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:55:52.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:52 smithi116 ceph-mon[35411]: pgmap v4275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:52.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:52 smithi114 ceph-mon[30820]: pgmap v4275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:55:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:54 smithi114 ceph-mon[30820]: pgmap v4276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:54 smithi116 ceph-mon[35411]: pgmap v4276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:55:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:56 smithi114 ceph-mon[30820]: pgmap v4277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:55:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:56 smithi116 ceph-mon[35411]: pgmap v4277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:55:58.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:55:58 smithi114 ceph-mon[30820]: pgmap v4278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:55:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:55:58 smithi116 ceph-mon[35411]: pgmap v4278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:56:00.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:00 smithi114 ceph-mon[30820]: pgmap v4279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:00.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:00 smithi116 ceph-mon[35411]: pgmap v4279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:02.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:02 smithi114 ceph-mon[30820]: pgmap v4280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:02.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:02 smithi116 ceph-mon[35411]: pgmap v4280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:04.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:04 smithi114 ceph-mon[30820]: pgmap v4281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:04 smithi116 ceph-mon[35411]: pgmap v4281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:06.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:06 smithi114 ceph-mon[30820]: pgmap v4282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:56:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:06 smithi116 ceph-mon[35411]: pgmap v4282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:56:08.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:08 smithi114 ceph-mon[30820]: pgmap v4283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:56:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:08 smithi116 ceph-mon[35411]: pgmap v4283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:56:10.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:10 smithi114 ceph-mon[30820]: pgmap v4284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:10.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:10 smithi116 ceph-mon[35411]: pgmap v4284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:12.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:12 smithi114 ceph-mon[30820]: pgmap v4285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:12.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:12 smithi116 ceph-mon[35411]: pgmap v4285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:13.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:56:13.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:56:14.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:14 smithi114 ceph-mon[30820]: pgmap v4286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:14.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:14 smithi116 ceph-mon[35411]: pgmap v4286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:16.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-mon[30820]: pgmap v4287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:56:16.661+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:56:16.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:56:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:56:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:56:16.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:16 smithi116 ceph-mon[35411]: pgmap v4287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:56:17.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:56:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:17 smithi114 ceph-mon[30820]: pgmap v4288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T08:56:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:17 smithi114 ceph-mon[30820]: pgmap v4289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:56:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:56:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:56:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:17 smithi116 ceph-mon[35411]: pgmap v4288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T08:56:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:17 smithi116 ceph-mon[35411]: pgmap v4289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:56:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:56:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:19 smithi114 ceph-mon[30820]: pgmap v4290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:56:20.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:19 smithi116 ceph-mon[35411]: pgmap v4290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:56:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:21 smithi114 ceph-mon[30820]: pgmap v4291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:56:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:21 smithi116 ceph-mon[35411]: pgmap v4291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:56:24.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:23 smithi114 ceph-mon[30820]: pgmap v4292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:56:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:23 smithi116 ceph-mon[35411]: pgmap v4292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:56:26.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:25 smithi114 ceph-mon[30820]: pgmap v4293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:56:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:25 smithi116 ceph-mon[35411]: pgmap v4293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:56:28.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:27 smithi114 ceph-mon[30820]: pgmap v4294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:56:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:27 smithi116 ceph-mon[35411]: pgmap v4294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:56:30.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:29 smithi114 ceph-mon[30820]: pgmap v4295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:29 smithi116 ceph-mon[35411]: pgmap v4295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:31 smithi114 ceph-mon[30820]: pgmap v4296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:56:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:31 smithi116 ceph-mon[35411]: pgmap v4296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:56:34.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:33 smithi114 ceph-mon[30820]: pgmap v4297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:33 smithi116 ceph-mon[35411]: pgmap v4297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:36.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:35 smithi114 ceph-mon[30820]: pgmap v4298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:35 smithi116 ceph-mon[35411]: pgmap v4298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:38.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:37 smithi114 ceph-mon[30820]: pgmap v4299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:37 smithi116 ceph-mon[35411]: pgmap v4299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:40.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:39 smithi114 ceph-mon[30820]: pgmap v4300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:39 smithi116 ceph-mon[35411]: pgmap v4300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:41 smithi114 ceph-mon[30820]: pgmap v4301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:56:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:41 smithi116 ceph-mon[35411]: pgmap v4301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:56:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:43 smithi114 ceph-mon[30820]: pgmap v4302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:43 smithi116 ceph-mon[35411]: pgmap v4302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:45 smithi114 ceph-mon[30820]: pgmap v4303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:45 smithi116 ceph-mon[35411]: pgmap v4303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:48.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:47 smithi114 ceph-mon[30820]: pgmap v4304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:47 smithi116 ceph-mon[35411]: pgmap v4304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:49.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:48 smithi114 ceph-mon[30820]: pgmap v4305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:48 smithi116 ceph-mon[35411]: pgmap v4305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:52.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:51 smithi114 ceph-mon[30820]: pgmap v4306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:56:52.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:51 smithi116 ceph-mon[35411]: pgmap v4306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:56:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:53 smithi114 ceph-mon[30820]: pgmap v4307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:53 smithi116 ceph-mon[35411]: pgmap v4307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:56:56.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:55 smithi114 ceph-mon[30820]: pgmap v4308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:55 smithi116 ceph-mon[35411]: pgmap v4308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:56:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:57 smithi114 ceph-mon[30820]: pgmap v4309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:56:58.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:57 smithi116 ceph-mon[35411]: pgmap v4309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:00.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:56:59 smithi114 ceph-mon[30820]: pgmap v4310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:56:59 smithi116 ceph-mon[35411]: pgmap v4310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:01 smithi114 ceph-mon[30820]: pgmap v4311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:57:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:01 smithi116 ceph-mon[35411]: pgmap v4311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:57:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:03 smithi114 ceph-mon[30820]: pgmap v4312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:03 smithi116 ceph-mon[35411]: pgmap v4312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:06.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:05 smithi114 ceph-mon[30820]: pgmap v4313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:57:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:05 smithi116 ceph-mon[35411]: pgmap v4313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:57:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:07 smithi114 ceph-mon[30820]: pgmap v4314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:07 smithi116 ceph-mon[35411]: pgmap v4314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:10.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:09 smithi114 ceph-mon[30820]: pgmap v4315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:10.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:09 smithi116 ceph-mon[35411]: pgmap v4315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:12.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:11 smithi114 ceph-mon[30820]: pgmap v4316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:57:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:11 smithi116 ceph-mon[35411]: pgmap v4316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:57:14.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:13 smithi114 ceph-mon[30820]: pgmap v4317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:13 smithi116 ceph-mon[35411]: pgmap v4317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:15 smithi114 ceph-mon[30820]: pgmap v4318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:57:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:15 smithi116 ceph-mon[35411]: pgmap v4318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:57:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:17 smithi114 ceph-mon[30820]: pgmap v4319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:57:18.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:17 smithi116 ceph-mon[35411]: pgmap v4319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:57:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:19 smithi114 ceph-mon[30820]: pgmap v4320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:19 smithi116 ceph-mon[35411]: pgmap v4320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:20.282 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:57:20.020+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:57:20.282 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:57:20.283 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:57:20.284 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:57:20.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:57:20.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:57:20.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:57:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:57:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:57:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:57:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:21 smithi114 ceph-mon[30820]: pgmap v4321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:57:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:21 smithi114 ceph-mon[30820]: pgmap v4322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:57:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:57:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:57:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:57:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:57:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:21 smithi116 ceph-mon[35411]: pgmap v4321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T08:57:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:21 smithi116 ceph-mon[35411]: pgmap v4322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T08:57:23.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:23 smithi114 ceph-mon[30820]: pgmap v4323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:57:23.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:23 smithi116 ceph-mon[35411]: pgmap v4323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:57:25.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:25 smithi114 ceph-mon[30820]: pgmap v4324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:57:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:25 smithi116 ceph-mon[35411]: pgmap v4324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:57:27.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:27 smithi114 ceph-mon[30820]: pgmap v4325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:57:27.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:27 smithi116 ceph-mon[35411]: pgmap v4325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:57:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:29 smithi114 ceph-mon[30820]: pgmap v4326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:57:29.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:29 smithi116 ceph-mon[35411]: pgmap v4326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:57:31.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:31 smithi114 ceph-mon[30820]: pgmap v4327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:57:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:31 smithi116 ceph-mon[35411]: pgmap v4327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:57:33.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:33 smithi114 ceph-mon[30820]: pgmap v4328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:33.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:33 smithi116 ceph-mon[35411]: pgmap v4328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:35.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:35 smithi114 ceph-mon[30820]: pgmap v4329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:35.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:35 smithi116 ceph-mon[35411]: pgmap v4329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:37.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:37 smithi114 ceph-mon[30820]: pgmap v4330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:57:37.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:37 smithi116 ceph-mon[35411]: pgmap v4330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:57:39.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:39 smithi114 ceph-mon[30820]: pgmap v4331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:57:39.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:39 smithi116 ceph-mon[35411]: pgmap v4331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:57:41.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:41 smithi114 ceph-mon[30820]: pgmap v4332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:57:41.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:41 smithi116 ceph-mon[35411]: pgmap v4332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:57:43.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:43 smithi114 ceph-mon[30820]: pgmap v4333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:43 smithi116 ceph-mon[35411]: pgmap v4333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:45.672 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:45 smithi114 ceph-mon[30820]: pgmap v4334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:57:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:45 smithi116 ceph-mon[35411]: pgmap v4334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:57:47.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:47 smithi114 ceph-mon[30820]: pgmap v4335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2024-09-19T08:57:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:47 smithi116 ceph-mon[35411]: pgmap v4335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2024-09-19T08:57:49.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:49 smithi114 ceph-mon[30820]: pgmap v4336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:57:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:49 smithi116 ceph-mon[35411]: pgmap v4336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:57:51.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:51 smithi114 ceph-mon[30820]: pgmap v4337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:57:51.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:51 smithi116 ceph-mon[35411]: pgmap v4337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T08:57:53.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:53 smithi114 ceph-mon[30820]: pgmap v4338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:53.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:53 smithi116 ceph-mon[35411]: pgmap v4338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:57:55.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:55 smithi114 ceph-mon[30820]: pgmap v4339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:57:55.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:55 smithi116 ceph-mon[35411]: pgmap v4339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T08:57:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:57 smithi114 ceph-mon[30820]: pgmap v4340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:57:57.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:57 smithi116 ceph-mon[35411]: pgmap v4340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:57:59.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:57:59 smithi114 ceph-mon[30820]: pgmap v4341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:57:59.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:57:59 smithi116 ceph-mon[35411]: pgmap v4341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:01.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:01 smithi114 ceph-mon[30820]: pgmap v4342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:01 smithi116 ceph-mon[35411]: pgmap v4342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:03.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:03 smithi114 ceph-mon[30820]: pgmap v4343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:03.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:03 smithi116 ceph-mon[35411]: pgmap v4343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:05.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:05 smithi114 ceph-mon[30820]: pgmap v4344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:05.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:05 smithi116 ceph-mon[35411]: pgmap v4344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:07.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:07 smithi114 ceph-mon[30820]: pgmap v4345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:58:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:07 smithi116 ceph-mon[35411]: pgmap v4345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:58:09.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:09 smithi114 ceph-mon[30820]: pgmap v4346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:09.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:09 smithi116 ceph-mon[35411]: pgmap v4346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:11.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:11 smithi114 ceph-mon[30820]: pgmap v4347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:11.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:11 smithi116 ceph-mon[35411]: pgmap v4347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:13.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:13 smithi114 ceph-mon[30820]: pgmap v4348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:13 smithi116 ceph-mon[35411]: pgmap v4348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:15.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:15 smithi114 ceph-mon[30820]: pgmap v4349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:15.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:15 smithi116 ceph-mon[35411]: pgmap v4349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:17 smithi114 ceph-mon[30820]: pgmap v4350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:58:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:17 smithi116 ceph-mon[35411]: pgmap v4350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:58:19.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:19 smithi114 ceph-mon[30820]: pgmap v4351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:19.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:19 smithi116 ceph-mon[35411]: pgmap v4351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:20.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:58:20.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:58:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:21 smithi114 ceph-mon[30820]: pgmap v4352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:21.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:21 smithi116 ceph-mon[35411]: pgmap v4352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:21.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:23.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-mon[30820]: pgmap v4353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:58:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:58:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:23.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:58:23.316+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:58:23.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:58:23.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:58:23.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:58:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:58:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:58:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:58:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:58:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:58:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:58:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:58:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:58:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:23 smithi116 ceph-mon[35411]: pgmap v4353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:58:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:58:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:58:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:58:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:58:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:24 smithi114 ceph-mon[30820]: pgmap v4354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:58:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:24 smithi114 ceph-mon[30820]: pgmap v4355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:58:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:58:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:58:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:24 smithi116 ceph-mon[35411]: pgmap v4354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:58:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:24 smithi116 ceph-mon[35411]: pgmap v4355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:58:26.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:26 smithi114 ceph-mon[30820]: pgmap v4356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:58:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:26 smithi116 ceph-mon[35411]: pgmap v4356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T08:58:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:28 smithi114 ceph-mon[30820]: pgmap v4357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:58:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:28 smithi116 ceph-mon[35411]: pgmap v4357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:58:30.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:30 smithi114 ceph-mon[30820]: pgmap v4358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:58:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:30 smithi116 ceph-mon[35411]: pgmap v4358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:58:32.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:32 smithi114 ceph-mon[30820]: pgmap v4359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:58:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:32 smithi116 ceph-mon[35411]: pgmap v4359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:58:34.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:34 smithi114 ceph-mon[30820]: pgmap v4360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:58:34.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:34 smithi116 ceph-mon[35411]: pgmap v4360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:58:36.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:36 smithi114 ceph-mon[30820]: pgmap v4361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:58:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:36 smithi116 ceph-mon[35411]: pgmap v4361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:58:38.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:38 smithi114 ceph-mon[30820]: pgmap v4362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:58:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:38 smithi116 ceph-mon[35411]: pgmap v4362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:58:40.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:40 smithi114 ceph-mon[30820]: pgmap v4363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:40 smithi116 ceph-mon[35411]: pgmap v4363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:42.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:42 smithi114 ceph-mon[30820]: pgmap v4364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:42 smithi116 ceph-mon[35411]: pgmap v4364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:44.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:44 smithi114 ceph-mon[30820]: pgmap v4365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:44 smithi116 ceph-mon[35411]: pgmap v4365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:46 smithi114 ceph-mon[30820]: pgmap v4366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:58:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:46 smithi116 ceph-mon[35411]: pgmap v4366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:58:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:48 smithi114 ceph-mon[30820]: pgmap v4367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:58:48.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:48 smithi116 ceph-mon[35411]: pgmap v4367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:58:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:50 smithi114 ceph-mon[30820]: pgmap v4368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:50 smithi116 ceph-mon[35411]: pgmap v4368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:58:52.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:52 smithi114 ceph-mon[30820]: pgmap v4369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:52 smithi116 ceph-mon[35411]: pgmap v4369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:58:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:54 smithi114 ceph-mon[30820]: pgmap v4370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:54 smithi116 ceph-mon[35411]: pgmap v4370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:58:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:56 smithi114 ceph-mon[30820]: pgmap v4371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:58:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:56 smithi116 ceph-mon[35411]: pgmap v4371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:58:58.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:58:58 smithi114 ceph-mon[30820]: pgmap v4372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:58:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:58:58 smithi116 ceph-mon[35411]: pgmap v4372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:59:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:00 smithi114 ceph-mon[30820]: pgmap v4373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:00 smithi116 ceph-mon[35411]: pgmap v4373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:02.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:02 smithi114 ceph-mon[30820]: pgmap v4374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:02.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:02 smithi116 ceph-mon[35411]: pgmap v4374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:04.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:04 smithi114 ceph-mon[30820]: pgmap v4375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:04 smithi116 ceph-mon[35411]: pgmap v4375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:06.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:06 smithi114 ceph-mon[30820]: pgmap v4376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:59:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:06 smithi116 ceph-mon[35411]: pgmap v4376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:59:08.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:08 smithi114 ceph-mon[30820]: pgmap v4377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:59:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:08 smithi116 ceph-mon[35411]: pgmap v4377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:59:10.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:10 smithi114 ceph-mon[30820]: pgmap v4378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:10.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:10 smithi116 ceph-mon[35411]: pgmap v4378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:12.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:12 smithi114 ceph-mon[30820]: pgmap v4379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:12 smithi116 ceph-mon[35411]: pgmap v4379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:14.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:14 smithi114 ceph-mon[30820]: pgmap v4380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:14 smithi116 ceph-mon[35411]: pgmap v4380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:16.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:16 smithi114 ceph-mon[30820]: pgmap v4381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:59:16.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:16 smithi116 ceph-mon[35411]: pgmap v4381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:59:18.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:18 smithi114 ceph-mon[30820]: pgmap v4382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:59:18.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:18 smithi116 ceph-mon[35411]: pgmap v4382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T08:59:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:20 smithi114 ceph-mon[30820]: pgmap v4383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:20 smithi116 ceph-mon[35411]: pgmap v4383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:22.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:22 smithi114 ceph-mon[30820]: pgmap v4384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:22 smithi116 ceph-mon[35411]: pgmap v4384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:23.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:59:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T08:59:24.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:24 smithi114 ceph-mon[30820]: pgmap v4385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:24 smithi116 ceph-mon[35411]: pgmap v4385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:26.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-mon[30820]: pgmap v4386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:59:26.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T08:59:26.676+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:59:26.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T08:59:26.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:59:26.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T08:59:26.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:59:26.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:59:26.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T08:59:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:59:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T08:59:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:59:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:59:26.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 08:59:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:59:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:26 smithi116 ceph-mon[35411]: pgmap v4386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T08:59:27.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:59:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T08:59:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:59:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T08:59:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:59:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:59:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:59:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T08:59:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:59:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:59:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:59:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:59:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:27 smithi114 ceph-mon[30820]: pgmap v4387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T08:59:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:27 smithi114 ceph-mon[30820]: pgmap v4388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:59:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:59:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:59:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T08:59:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T08:59:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:27 smithi116 ceph-mon[35411]: pgmap v4387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T08:59:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:27 smithi116 ceph-mon[35411]: pgmap v4388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:59:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T08:59:30.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:29 smithi114 ceph-mon[30820]: pgmap v4389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:59:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:29 smithi116 ceph-mon[35411]: pgmap v4389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T08:59:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:31 smithi114 ceph-mon[30820]: pgmap v4390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:59:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:31 smithi116 ceph-mon[35411]: pgmap v4390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:59:34.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:33 smithi114 ceph-mon[30820]: pgmap v4391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:59:34.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:33 smithi116 ceph-mon[35411]: pgmap v4391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T08:59:36.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:35 smithi114 ceph-mon[30820]: pgmap v4392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:59:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:35 smithi116 ceph-mon[35411]: pgmap v4392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T08:59:38.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:37 smithi114 ceph-mon[30820]: pgmap v4393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:59:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:37 smithi116 ceph-mon[35411]: pgmap v4393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T08:59:40.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:39 smithi114 ceph-mon[30820]: pgmap v4394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:39 smithi116 ceph-mon[35411]: pgmap v4394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:41 smithi114 ceph-mon[30820]: pgmap v4395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:59:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:41 smithi116 ceph-mon[35411]: pgmap v4395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:59:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:43 smithi114 ceph-mon[30820]: pgmap v4396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:43 smithi116 ceph-mon[35411]: pgmap v4396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:45 smithi114 ceph-mon[30820]: pgmap v4397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:45 smithi116 ceph-mon[35411]: pgmap v4397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:48.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:47 smithi114 ceph-mon[30820]: pgmap v4398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:47 smithi116 ceph-mon[35411]: pgmap v4398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:50.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:49 smithi114 ceph-mon[30820]: pgmap v4399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:50.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:49 smithi116 ceph-mon[35411]: pgmap v4399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:52.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:51 smithi114 ceph-mon[30820]: pgmap v4400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:59:52.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:51 smithi116 ceph-mon[35411]: pgmap v4400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T08:59:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:53 smithi114 ceph-mon[30820]: pgmap v4401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:53 smithi116 ceph-mon[35411]: pgmap v4401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T08:59:56.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:55 smithi114 ceph-mon[30820]: pgmap v4402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:56.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:55 smithi116 ceph-mon[35411]: pgmap v4402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T08:59:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:57 smithi114 ceph-mon[30820]: pgmap v4403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T08:59:58.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:57 smithi116 ceph-mon[35411]: pgmap v4403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:00.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 08:59:59 smithi114 ceph-mon[30820]: pgmap v4404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 08:59:59 smithi116 ceph-mon[35411]: pgmap v4404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:01.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:00:01.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:00:02.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:01 smithi114 ceph-mon[30820]: pgmap v4405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:01 smithi116 ceph-mon[35411]: pgmap v4405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:04.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:03 smithi114 ceph-mon[30820]: pgmap v4406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:03 smithi116 ceph-mon[35411]: pgmap v4406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:05 smithi114 ceph-mon[30820]: pgmap v4407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:00:06.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:05 smithi116 ceph-mon[35411]: pgmap v4407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:00:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:07 smithi114 ceph-mon[30820]: pgmap v4408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:07 smithi116 ceph-mon[35411]: pgmap v4408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:09.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:08 smithi114 ceph-mon[30820]: pgmap v4409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:08 smithi116 ceph-mon[35411]: pgmap v4409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:12.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:11 smithi114 ceph-mon[30820]: pgmap v4410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:12.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:11 smithi116 ceph-mon[35411]: pgmap v4410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:14.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:13 smithi114 ceph-mon[30820]: pgmap v4411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:13 smithi116 ceph-mon[35411]: pgmap v4411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:16.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:15 smithi114 ceph-mon[30820]: pgmap v4412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:00:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:15 smithi116 ceph-mon[35411]: pgmap v4412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:00:18.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:17 smithi114 ceph-mon[30820]: pgmap v4413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:18.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:17 smithi116 ceph-mon[35411]: pgmap v4413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:19 smithi114 ceph-mon[30820]: pgmap v4414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:20.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:19 smithi116 ceph-mon[35411]: pgmap v4414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:22.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:21 smithi114 ceph-mon[30820]: pgmap v4415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:22.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:21 smithi116 ceph-mon[35411]: pgmap v4415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:24.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:23 smithi114 ceph-mon[30820]: pgmap v4416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:23 smithi116 ceph-mon[35411]: pgmap v4416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:26.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:25 smithi114 ceph-mon[30820]: pgmap v4417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:00:26.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:25 smithi116 ceph-mon[35411]: pgmap v4417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:00:28.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:27 smithi114 ceph-mon[30820]: pgmap v4418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:28.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:00:28.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:27 smithi116 ceph-mon[35411]: pgmap v4418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:00:30.037 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:29 smithi114 ceph-mon[30820]: pgmap v4419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:30.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:29 smithi116 ceph-mon[35411]: pgmap v4419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:30.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:00:30.036+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:00:30.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:00:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:00:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:00:31.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:00:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:00:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:00:31.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:00:31.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:00:31.246 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:00:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:00:32.165 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:00:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:31 smithi114 ceph-mon[30820]: pgmap v4420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T09:00:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:31 smithi114 ceph-mon[30820]: pgmap v4421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:00:32.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:00:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:31 smithi116 ceph-mon[35411]: pgmap v4420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T09:00:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:31 smithi116 ceph-mon[35411]: pgmap v4421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:00:34.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:33 smithi114 ceph-mon[30820]: pgmap v4422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:00:34.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:33 smithi116 ceph-mon[35411]: pgmap v4422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:00:36.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:35 smithi114 ceph-mon[30820]: pgmap v4423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:00:36.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:35 smithi116 ceph-mon[35411]: pgmap v4423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:00:38.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:37 smithi114 ceph-mon[30820]: pgmap v4424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:00:38.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:37 smithi116 ceph-mon[35411]: pgmap v4424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:00:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:39 smithi114 ceph-mon[30820]: pgmap v4425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:00:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:39 smithi116 ceph-mon[35411]: pgmap v4425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:00:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:41 smithi114 ceph-mon[30820]: pgmap v4426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:00:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:41 smithi116 ceph-mon[35411]: pgmap v4426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:00:44.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:43 smithi114 ceph-mon[30820]: pgmap v4427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:44.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:43 smithi116 ceph-mon[35411]: pgmap v4427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:45.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:44 smithi114 ceph-mon[30820]: pgmap v4428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:45.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:44 smithi116 ceph-mon[35411]: pgmap v4428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:47 smithi114 ceph-mon[30820]: pgmap v4429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:47.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:47 smithi116 ceph-mon[35411]: pgmap v4429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:49.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:49 smithi114 ceph-mon[30820]: pgmap v4430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:49.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:49 smithi116 ceph-mon[35411]: pgmap v4430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:51 smithi114 ceph-mon[30820]: pgmap v4431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:00:51.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:51 smithi116 ceph-mon[35411]: pgmap v4431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:00:53.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:53 smithi114 ceph-mon[30820]: pgmap v4432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:53.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:53 smithi116 ceph-mon[35411]: pgmap v4432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:00:55.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:55 smithi114 ceph-mon[30820]: pgmap v4433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:55.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:55 smithi116 ceph-mon[35411]: pgmap v4433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:57.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:57 smithi114 ceph-mon[30820]: pgmap v4434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:57.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:57 smithi116 ceph-mon[35411]: pgmap v4434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:00:59.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:00:59 smithi114 ceph-mon[30820]: pgmap v4435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:00:59.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:00:59 smithi116 ceph-mon[35411]: pgmap v4435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:01.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:01 smithi114 ceph-mon[30820]: pgmap v4436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:01.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:01 smithi116 ceph-mon[35411]: pgmap v4436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:03 smithi114 ceph-mon[30820]: pgmap v4437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:03.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:03 smithi116 ceph-mon[35411]: pgmap v4437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:05.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:05 smithi114 ceph-mon[30820]: pgmap v4438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:05 smithi116 ceph-mon[35411]: pgmap v4438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:07.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:07 smithi114 ceph-mon[30820]: pgmap v4439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:01:07.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:07 smithi116 ceph-mon[35411]: pgmap v4439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:01:09.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:09 smithi114 ceph-mon[30820]: pgmap v4440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:09 smithi116 ceph-mon[35411]: pgmap v4440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:11.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:11 smithi114 ceph-mon[30820]: pgmap v4441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:11.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:11 smithi116 ceph-mon[35411]: pgmap v4441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:13.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:13 smithi114 ceph-mon[30820]: pgmap v4442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:13.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:13 smithi116 ceph-mon[35411]: pgmap v4442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:15.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:15 smithi114 ceph-mon[30820]: pgmap v4443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:15 smithi116 ceph-mon[35411]: pgmap v4443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:17.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:17 smithi114 ceph-mon[30820]: pgmap v4444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:01:17.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:17 smithi116 ceph-mon[35411]: pgmap v4444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:01:19.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:19 smithi114 ceph-mon[30820]: pgmap v4445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:19.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:19 smithi116 ceph-mon[35411]: pgmap v4445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:21 smithi116 ceph-mon[35411]: pgmap v4446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:21 smithi114 ceph-mon[30820]: pgmap v4446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:23 smithi116 ceph-mon[35411]: pgmap v4447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:23.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:23 smithi114 ceph-mon[30820]: pgmap v4447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:25 smithi116 ceph-mon[35411]: pgmap v4448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:25.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:25 smithi114 ceph-mon[30820]: pgmap v4448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:27.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:27 smithi114 ceph-mon[30820]: pgmap v4449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:01:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:27 smithi116 ceph-mon[35411]: pgmap v4449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:01:29.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:29 smithi114 ceph-mon[30820]: pgmap v4450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:29.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:29 smithi116 ceph-mon[35411]: pgmap v4450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:30.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:01:30.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:01:31.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:31 smithi114 ceph-mon[30820]: pgmap v4451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:31.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:31 smithi116 ceph-mon[35411]: pgmap v4451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:33.467 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-mon[30820]: pgmap v4452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:33.468 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:01:33.396+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:01:33.468 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:01:33.468 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:01:33.468 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:01:33.468 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:01:33.468 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:01:33.469 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:01:33.470 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:01:33.470 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:01:33.470 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:01:33.470 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:01:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:01:33.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:33 smithi116 ceph-mon[35411]: pgmap v4452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:01:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:01:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:01:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:01:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:34 smithi114 ceph-mon[30820]: pgmap v4453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:01:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:34 smithi114 ceph-mon[30820]: pgmap v4454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:01:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:01:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:01:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:01:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:01:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:34 smithi116 ceph-mon[35411]: pgmap v4453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:01:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:34 smithi116 ceph-mon[35411]: pgmap v4454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:01:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:01:36.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:36 smithi114 ceph-mon[30820]: pgmap v4455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:01:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:36 smithi116 ceph-mon[35411]: pgmap v4455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:01:38.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:38 smithi114 ceph-mon[30820]: pgmap v4456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:01:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:38 smithi116 ceph-mon[35411]: pgmap v4456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:01:40.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:40 smithi114 ceph-mon[30820]: pgmap v4457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:01:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:40 smithi116 ceph-mon[35411]: pgmap v4457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:01:42.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:42 smithi114 ceph-mon[30820]: pgmap v4458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:01:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:42 smithi116 ceph-mon[35411]: pgmap v4458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:01:44.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:44 smithi114 ceph-mon[30820]: pgmap v4459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:01:44.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:44 smithi116 ceph-mon[35411]: pgmap v4459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:01:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:46 smithi114 ceph-mon[30820]: pgmap v4460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:01:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:46 smithi116 ceph-mon[35411]: pgmap v4460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:01:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:48 smithi114 ceph-mon[30820]: pgmap v4461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:01:48.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:48 smithi116 ceph-mon[35411]: pgmap v4461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:01:50.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:50 smithi114 ceph-mon[30820]: pgmap v4462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:50 smithi116 ceph-mon[35411]: pgmap v4462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:01:52.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:52 smithi114 ceph-mon[30820]: pgmap v4463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:52 smithi116 ceph-mon[35411]: pgmap v4463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:01:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:54 smithi114 ceph-mon[30820]: pgmap v4464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:54 smithi116 ceph-mon[35411]: pgmap v4464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:01:56.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:56 smithi114 ceph-mon[30820]: pgmap v4465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:01:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:56 smithi116 ceph-mon[35411]: pgmap v4465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:01:58.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:01:58 smithi114 ceph-mon[30820]: pgmap v4466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:01:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:01:58 smithi116 ceph-mon[35411]: pgmap v4466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:02:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:00 smithi114 ceph-mon[30820]: pgmap v4467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:00.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:00 smithi116 ceph-mon[35411]: pgmap v4467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:02.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:02 smithi114 ceph-mon[30820]: pgmap v4468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:02.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:02 smithi116 ceph-mon[35411]: pgmap v4468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:04 smithi114 ceph-mon[30820]: pgmap v4469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:04 smithi116 ceph-mon[35411]: pgmap v4469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:06 smithi114 ceph-mon[30820]: pgmap v4470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:02:06.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:06 smithi116 ceph-mon[35411]: pgmap v4470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:02:08.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:08 smithi114 ceph-mon[30820]: pgmap v4471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:02:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:08 smithi116 ceph-mon[35411]: pgmap v4471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:02:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:10 smithi114 ceph-mon[30820]: pgmap v4472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:10.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:10 smithi116 ceph-mon[35411]: pgmap v4472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:12 smithi114 ceph-mon[30820]: pgmap v4473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:12.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:12 smithi116 ceph-mon[35411]: pgmap v4473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:14.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:14 smithi114 ceph-mon[30820]: pgmap v4474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:14 smithi116 ceph-mon[35411]: pgmap v4474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:16.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:16 smithi114 ceph-mon[30820]: pgmap v4475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:02:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:16 smithi116 ceph-mon[35411]: pgmap v4475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:02:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:18 smithi114 ceph-mon[30820]: pgmap v4476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:02:18.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:18 smithi116 ceph-mon[35411]: pgmap v4476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:02:20.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:20 smithi114 ceph-mon[30820]: pgmap v4477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:20 smithi116 ceph-mon[35411]: pgmap v4477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:22 smithi114 ceph-mon[30820]: pgmap v4478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:22.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:22 smithi116 ceph-mon[35411]: pgmap v4478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:24.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:24 smithi114 ceph-mon[30820]: pgmap v4479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:24 smithi116 ceph-mon[35411]: pgmap v4479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:26 smithi114 ceph-mon[30820]: pgmap v4480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:02:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:26 smithi116 ceph-mon[35411]: pgmap v4480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:02:28.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:28 smithi114 ceph-mon[30820]: pgmap v4481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:02:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:28 smithi116 ceph-mon[35411]: pgmap v4481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:02:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:30 smithi114 ceph-mon[30820]: pgmap v4482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:30 smithi116 ceph-mon[35411]: pgmap v4482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:32 smithi114 ceph-mon[30820]: pgmap v4483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:32.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:32 smithi116 ceph-mon[35411]: pgmap v4483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:33.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:02:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:02:34.829 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:34 smithi116 ceph-mon[35411]: pgmap v4484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:34.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:34 smithi114 ceph-mon[30820]: pgmap v4484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:36.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-mon[30820]: pgmap v4485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:02:36.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:02:36.757+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:02:36.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:02:36.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:02:36.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:02:36.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:02:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:02:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:02:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:02:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:02:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:02:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:02:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:02:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:02:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:36 smithi116 ceph-mon[35411]: pgmap v4485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:02:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:37 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:02:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:02:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:02:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:02:37.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:02:37.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:02:37.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:02:37.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:02:37.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:02:37.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:02:37.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:02:37.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:02:37.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:02:37.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:02:37.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:02:37.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:02:37.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:02:37.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:37 smithi114 ceph-mon[30820]: pgmap v4486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:02:37.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:37 smithi114 ceph-mon[30820]: pgmap v4487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:02:37.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:02:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:37 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:02:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:02:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:02:37.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:02:37.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:02:37.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:02:37.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:37 smithi116 ceph-mon[35411]: pgmap v4486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:02:37.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:37 smithi116 ceph-mon[35411]: pgmap v4487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:02:37.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:02:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:39 smithi114 ceph-mon[30820]: pgmap v4488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:02:40.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:39 smithi116 ceph-mon[35411]: pgmap v4488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:02:42.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:41 smithi114 ceph-mon[30820]: pgmap v4489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:02:42.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:41 smithi116 ceph-mon[35411]: pgmap v4489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:02:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:43 smithi114 ceph-mon[30820]: pgmap v4490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:02:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:43 smithi116 ceph-mon[35411]: pgmap v4490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:02:46.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:45 smithi114 ceph-mon[30820]: pgmap v4491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:02:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:45 smithi116 ceph-mon[35411]: pgmap v4491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:02:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:47 smithi114 ceph-mon[30820]: pgmap v4492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:02:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:47 smithi116 ceph-mon[35411]: pgmap v4492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:02:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:49 smithi114 ceph-mon[30820]: pgmap v4493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:50.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:49 smithi116 ceph-mon[35411]: pgmap v4493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:52.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:51 smithi114 ceph-mon[30820]: pgmap v4494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:02:52.186 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:51 smithi116 ceph-mon[35411]: pgmap v4494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:02:54.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:53 smithi114 ceph-mon[30820]: pgmap v4495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:53 smithi116 ceph-mon[35411]: pgmap v4495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:02:55.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:54 smithi114 ceph-mon[30820]: pgmap v4496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:55.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:54 smithi116 ceph-mon[35411]: pgmap v4496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:02:58.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:57 smithi114 ceph-mon[30820]: pgmap v4497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:02:58.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:57 smithi116 ceph-mon[35411]: pgmap v4497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:02:59 smithi114 ceph-mon[30820]: pgmap v4498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:00.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:02:59 smithi116 ceph-mon[35411]: pgmap v4498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:01 smithi114 ceph-mon[30820]: pgmap v4499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:02.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:01 smithi116 ceph-mon[35411]: pgmap v4499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:03 smithi114 ceph-mon[30820]: pgmap v4500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:04.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:03 smithi116 ceph-mon[35411]: pgmap v4500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:06.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:05 smithi114 ceph-mon[30820]: pgmap v4501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:03:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:05 smithi116 ceph-mon[35411]: pgmap v4501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:03:08.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:07 smithi114 ceph-mon[30820]: pgmap v4502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:08.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:07 smithi116 ceph-mon[35411]: pgmap v4502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:10.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:09 smithi114 ceph-mon[30820]: pgmap v4503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:09 smithi116 ceph-mon[35411]: pgmap v4503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:10 smithi114 ceph-mon[30820]: pgmap v4504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:10 smithi116 ceph-mon[35411]: pgmap v4504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:14.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:13 smithi114 ceph-mon[30820]: pgmap v4505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:14.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:13 smithi116 ceph-mon[35411]: pgmap v4505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:15 smithi114 ceph-mon[30820]: pgmap v4506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:03:16.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:15 smithi116 ceph-mon[35411]: pgmap v4506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:03:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:17 smithi114 ceph-mon[30820]: pgmap v4507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:18.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:17 smithi116 ceph-mon[35411]: pgmap v4507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:20.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:19 smithi114 ceph-mon[30820]: pgmap v4508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:19 smithi116 ceph-mon[35411]: pgmap v4508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:21 smithi114 ceph-mon[30820]: pgmap v4509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:21 smithi116 ceph-mon[35411]: pgmap v4509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:23 smithi114 ceph-mon[30820]: pgmap v4510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:24.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:23 smithi116 ceph-mon[35411]: pgmap v4510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:24 smithi114 ceph-mon[30820]: pgmap v4511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:03:25.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:24 smithi116 ceph-mon[35411]: pgmap v4511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:03:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:27 smithi114 ceph-mon[30820]: pgmap v4512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:27 smithi116 ceph-mon[35411]: pgmap v4512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:29 smithi114 ceph-mon[30820]: pgmap v4513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:29 smithi116 ceph-mon[35411]: pgmap v4513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:31 smithi114 ceph-mon[30820]: pgmap v4514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:31 smithi116 ceph-mon[35411]: pgmap v4514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:33 smithi114 ceph-mon[30820]: pgmap v4515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:33 smithi116 ceph-mon[35411]: pgmap v4515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:35 smithi114 ceph-mon[30820]: pgmap v4516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:03:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:35 smithi116 ceph-mon[35411]: pgmap v4516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:03:37.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:36 smithi114 ceph-mon[30820]: pgmap v4517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:03:37.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:36 smithi116 ceph-mon[35411]: pgmap v4517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:03:40.118 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:39 smithi114 ceph-mon[30820]: pgmap v4518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:39 smithi116 ceph-mon[35411]: pgmap v4518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:40.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:03:40.116+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:03:40.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:03:40.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:03:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:03:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:03:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:03:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:03:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:03:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:03:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:03:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:03:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:03:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:41 smithi114 ceph-mon[30820]: pgmap v4519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T09:03:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:41 smithi114 ceph-mon[30820]: pgmap v4520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:03:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:41 smithi116 ceph-mon[35411]: pgmap v4519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T09:03:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:41 smithi116 ceph-mon[35411]: pgmap v4520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:03:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:43 smithi114 ceph-mon[30820]: pgmap v4521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:03:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:43 smithi116 ceph-mon[35411]: pgmap v4521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:03:45.165 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:44 smithi114 ceph-mon[30820]: pgmap v4522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:03:45.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:44 smithi116 ceph-mon[35411]: pgmap v4522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:03:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:47 smithi114 ceph-mon[30820]: pgmap v4523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:03:47.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:47 smithi116 ceph-mon[35411]: pgmap v4523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:03:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:49 smithi114 ceph-mon[30820]: pgmap v4524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:03:49.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:49 smithi116 ceph-mon[35411]: pgmap v4524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:03:51.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:51 smithi114 ceph-mon[30820]: pgmap v4525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:03:51.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:51 smithi116 ceph-mon[35411]: pgmap v4525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:03:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:53 smithi114 ceph-mon[30820]: pgmap v4526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:53 smithi116 ceph-mon[35411]: pgmap v4526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:03:55.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:55 smithi114 ceph-mon[30820]: pgmap v4527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:55 smithi116 ceph-mon[35411]: pgmap v4527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:57.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:57 smithi114 ceph-mon[30820]: pgmap v4528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:57 smithi116 ceph-mon[35411]: pgmap v4528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:03:59.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:03:59 smithi114 ceph-mon[30820]: pgmap v4529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:03:59.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:03:59 smithi116 ceph-mon[35411]: pgmap v4529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:01 smithi114 ceph-mon[30820]: pgmap v4530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:04:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:01 smithi116 ceph-mon[35411]: pgmap v4530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:04:03.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:03 smithi114 ceph-mon[30820]: pgmap v4531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:03.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:03 smithi116 ceph-mon[35411]: pgmap v4531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:05.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:05 smithi114 ceph-mon[30820]: pgmap v4532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:05.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:05 smithi116 ceph-mon[35411]: pgmap v4532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:07.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:07 smithi114 ceph-mon[30820]: pgmap v4533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:07.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:07 smithi116 ceph-mon[35411]: pgmap v4533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:09 smithi114 ceph-mon[30820]: pgmap v4534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:09.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:09 smithi116 ceph-mon[35411]: pgmap v4534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:11 smithi114 ceph-mon[30820]: pgmap v4535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:04:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:11 smithi116 ceph-mon[35411]: pgmap v4535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:04:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:13 smithi114 ceph-mon[30820]: pgmap v4536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:13 smithi116 ceph-mon[35411]: pgmap v4536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:15 smithi114 ceph-mon[30820]: pgmap v4537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:15.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:15 smithi116 ceph-mon[35411]: pgmap v4537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:17 smithi114 ceph-mon[30820]: pgmap v4538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:17 smithi116 ceph-mon[35411]: pgmap v4538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:19 smithi114 ceph-mon[30820]: pgmap v4539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:19 smithi116 ceph-mon[35411]: pgmap v4539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:21.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:21 smithi114 ceph-mon[30820]: pgmap v4540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:04:21.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:21 smithi116 ceph-mon[35411]: pgmap v4540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:04:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:23 smithi114 ceph-mon[30820]: pgmap v4541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:23 smithi116 ceph-mon[35411]: pgmap v4541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:25 smithi114 ceph-mon[30820]: pgmap v4542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:25.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:25 smithi116 ceph-mon[35411]: pgmap v4542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:27 smithi114 ceph-mon[30820]: pgmap v4543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:27 smithi116 ceph-mon[35411]: pgmap v4543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:29 smithi114 ceph-mon[30820]: pgmap v4544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:04:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:29 smithi116 ceph-mon[35411]: pgmap v4544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:04:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:31 smithi114 ceph-mon[30820]: pgmap v4545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:04:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:31 smithi116 ceph-mon[35411]: pgmap v4545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:04:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:33 smithi114 ceph-mon[30820]: pgmap v4546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:33 smithi116 ceph-mon[35411]: pgmap v4546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:35 smithi114 ceph-mon[30820]: pgmap v4547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:04:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:35 smithi116 ceph-mon[35411]: pgmap v4547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:04:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:37 smithi114 ceph-mon[30820]: pgmap v4548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:37 smithi116 ceph-mon[35411]: pgmap v4548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:39 smithi114 ceph-mon[30820]: pgmap v4549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:39 smithi116 ceph-mon[35411]: pgmap v4549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:40.665 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:04:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: pgmap v4550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: pgmap v4550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:04:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:04:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:42 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:04:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:42 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:04:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:42 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:04:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:42 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:04:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-mon[30820]: pgmap v4551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:43.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:04:43.477+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:04:43.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:04:43.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:04:43.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:04:43.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:04:43.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:04:43.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:04:43.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:04:43.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:04:43.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:04:43.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:04:43.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:04:43.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:04:43.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:04:43.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:04:43.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:04:43.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:04:43.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:04:43.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:04:43.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:04:43.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:04:43.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:04:43.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:04:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:04:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:43 smithi116 ceph-mon[35411]: pgmap v4551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:44 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:04:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:04:44.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:04:44.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:04:44.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:04:44.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:44 smithi114 ceph-mon[30820]: pgmap v4552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:04:44.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:44 smithi114 ceph-mon[30820]: pgmap v4553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:44.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:44 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:04:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:04:44.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:04:44.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:04:44.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:44 smithi116 ceph-mon[35411]: pgmap v4552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:04:44.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:44 smithi116 ceph-mon[35411]: pgmap v4553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:44.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:04:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:46 smithi114 ceph-mon[30820]: pgmap v4554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:46 smithi116 ceph-mon[35411]: pgmap v4554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:48.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:48 smithi114 ceph-mon[30820]: pgmap v4555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:48 smithi116 ceph-mon[35411]: pgmap v4555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:50 smithi114 ceph-mon[30820]: pgmap v4556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:50 smithi116 ceph-mon[35411]: pgmap v4556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:52 smithi114 ceph-mon[30820]: pgmap v4557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:52 smithi116 ceph-mon[35411]: pgmap v4557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:04:54.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:54 smithi114 ceph-mon[30820]: pgmap v4558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:04:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:54 smithi116 ceph-mon[35411]: pgmap v4558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:04:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:56 smithi114 ceph-mon[30820]: pgmap v4559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:56 smithi116 ceph-mon[35411]: pgmap v4559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:04:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:04:58 smithi114 ceph-mon[30820]: pgmap v4560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:04:58.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:04:58 smithi116 ceph-mon[35411]: pgmap v4560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:00.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:00 smithi114 ceph-mon[30820]: pgmap v4561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:00 smithi116 ceph-mon[35411]: pgmap v4561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:02 smithi114 ceph-mon[30820]: pgmap v4562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:02 smithi116 ceph-mon[35411]: pgmap v4562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:04 smithi114 ceph-mon[30820]: pgmap v4563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:04.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:04 smithi116 ceph-mon[35411]: pgmap v4563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:06 smithi114 ceph-mon[30820]: pgmap v4564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:06 smithi116 ceph-mon[35411]: pgmap v4564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:08 smithi114 ceph-mon[30820]: pgmap v4565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:08.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:08 smithi116 ceph-mon[35411]: pgmap v4565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:10.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:10 smithi114 ceph-mon[30820]: pgmap v4566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:10 smithi116 ceph-mon[35411]: pgmap v4566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:12 smithi114 ceph-mon[30820]: pgmap v4567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:12 smithi116 ceph-mon[35411]: pgmap v4567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:14 smithi114 ceph-mon[30820]: pgmap v4568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:14 smithi116 ceph-mon[35411]: pgmap v4568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:16 smithi114 ceph-mon[30820]: pgmap v4569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:16 smithi116 ceph-mon[35411]: pgmap v4569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:18 smithi114 ceph-mon[30820]: pgmap v4570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:18 smithi116 ceph-mon[35411]: pgmap v4570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:20 smithi114 ceph-mon[30820]: pgmap v4571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:20 smithi116 ceph-mon[35411]: pgmap v4571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:22 smithi114 ceph-mon[30820]: pgmap v4572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:22 smithi116 ceph-mon[35411]: pgmap v4572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:24 smithi114 ceph-mon[30820]: pgmap v4573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:24 smithi116 ceph-mon[35411]: pgmap v4573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:26 smithi114 ceph-mon[30820]: pgmap v4574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:26.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:26 smithi116 ceph-mon[35411]: pgmap v4574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:28 smithi114 ceph-mon[30820]: pgmap v4575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:28 smithi116 ceph-mon[35411]: pgmap v4575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:30 smithi114 ceph-mon[30820]: pgmap v4576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:30 smithi116 ceph-mon[35411]: pgmap v4576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:32 smithi114 ceph-mon[30820]: pgmap v4577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:32 smithi116 ceph-mon[35411]: pgmap v4577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:34 smithi114 ceph-mon[30820]: pgmap v4578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:34 smithi116 ceph-mon[35411]: pgmap v4578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:36 smithi114 ceph-mon[30820]: pgmap v4579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:36 smithi116 ceph-mon[35411]: pgmap v4579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:38 smithi114 ceph-mon[30820]: pgmap v4580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:38 smithi116 ceph-mon[35411]: pgmap v4580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:40.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:40 smithi114 ceph-mon[30820]: pgmap v4581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:40 smithi116 ceph-mon[35411]: pgmap v4581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:05:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:42 smithi114 ceph-mon[30820]: pgmap v4582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:42 smithi116 ceph-mon[35411]: pgmap v4582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:05:43.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:05:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:05:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:44 smithi114 ceph-mon[30820]: pgmap v4583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:44 smithi116 ceph-mon[35411]: pgmap v4583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:46.915 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-mon[30820]: pgmap v4584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:46.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:05:46.836+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:05:46.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:05:46.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:05:46.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:05:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:05:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:05:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:05:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:05:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:05:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:05:46.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:05:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:05:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:46 smithi116 ceph-mon[35411]: pgmap v4584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:05:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:47 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:05:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:05:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:05:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:05:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:05:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:05:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:05:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:05:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:05:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:05:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:05:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:05:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:05:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:47 smithi114 ceph-mon[30820]: pgmap v4585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:05:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:47 smithi114 ceph-mon[30820]: pgmap v4586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:05:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:47 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:05:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:05:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:05:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:05:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:05:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:47 smithi116 ceph-mon[35411]: pgmap v4585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:05:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:47 smithi116 ceph-mon[35411]: pgmap v4586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:05:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:05:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:48 smithi114 ceph-mon[30820]: pgmap v4587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:05:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:48 smithi116 ceph-mon[35411]: pgmap v4587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:05:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:50 smithi116 ceph-mon[35411]: pgmap v4588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:05:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:50 smithi114 ceph-mon[30820]: pgmap v4588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:05:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:52 smithi114 ceph-mon[30820]: pgmap v4589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:05:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:52 smithi116 ceph-mon[35411]: pgmap v4589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:05:55.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:54 smithi114 ceph-mon[30820]: pgmap v4590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:05:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:54 smithi116 ceph-mon[35411]: pgmap v4590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:05:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:56 smithi114 ceph-mon[30820]: pgmap v4591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:05:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:56 smithi116 ceph-mon[35411]: pgmap v4591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:05:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:05:58 smithi114 ceph-mon[30820]: pgmap v4592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:05:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:05:58 smithi116 ceph-mon[35411]: pgmap v4592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:00 smithi114 ceph-mon[30820]: pgmap v4593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:00 smithi116 ceph-mon[35411]: pgmap v4593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:02 smithi114 ceph-mon[30820]: pgmap v4594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:02 smithi116 ceph-mon[35411]: pgmap v4594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:04 smithi114 ceph-mon[30820]: pgmap v4595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:06:05.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:04 smithi116 ceph-mon[35411]: pgmap v4595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:06:07.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:06 smithi114 ceph-mon[30820]: pgmap v4596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:07.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:06 smithi116 ceph-mon[35411]: pgmap v4596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:08 smithi114 ceph-mon[30820]: pgmap v4597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:09.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:08 smithi116 ceph-mon[35411]: pgmap v4597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:10 smithi114 ceph-mon[30820]: pgmap v4598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:10 smithi116 ceph-mon[35411]: pgmap v4598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:12 smithi114 ceph-mon[30820]: pgmap v4599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:12 smithi116 ceph-mon[35411]: pgmap v4599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:14 smithi114 ceph-mon[30820]: pgmap v4600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:06:15.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:14 smithi116 ceph-mon[35411]: pgmap v4600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:06:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:16 smithi114 ceph-mon[30820]: pgmap v4601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:16 smithi116 ceph-mon[35411]: pgmap v4601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:18 smithi114 ceph-mon[30820]: pgmap v4602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:18 smithi116 ceph-mon[35411]: pgmap v4602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:20 smithi114 ceph-mon[30820]: pgmap v4603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:21.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:20 smithi116 ceph-mon[35411]: pgmap v4603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:22 smithi114 ceph-mon[30820]: pgmap v4604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:22 smithi116 ceph-mon[35411]: pgmap v4604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:24 smithi114 ceph-mon[30820]: pgmap v4605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:06:25.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:24 smithi116 ceph-mon[35411]: pgmap v4605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:06:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:26 smithi114 ceph-mon[30820]: pgmap v4606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:26 smithi116 ceph-mon[35411]: pgmap v4606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:28 smithi114 ceph-mon[30820]: pgmap v4607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:28 smithi116 ceph-mon[35411]: pgmap v4607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:30 smithi114 ceph-mon[30820]: pgmap v4608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:31.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:30 smithi116 ceph-mon[35411]: pgmap v4608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:32 smithi114 ceph-mon[30820]: pgmap v4609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:32 smithi116 ceph-mon[35411]: pgmap v4609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:34 smithi114 ceph-mon[30820]: pgmap v4610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:06:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:34 smithi116 ceph-mon[35411]: pgmap v4610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:06:37.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:36 smithi114 ceph-mon[30820]: pgmap v4611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:37.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:36 smithi116 ceph-mon[35411]: pgmap v4611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:38 smithi114 ceph-mon[30820]: pgmap v4612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:06:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:38 smithi116 ceph-mon[35411]: pgmap v4612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:06:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:40 smithi114 ceph-mon[30820]: pgmap v4613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:40 smithi116 ceph-mon[35411]: pgmap v4613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:06:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:42 smithi114 ceph-mon[30820]: pgmap v4614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:06:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:42 smithi116 ceph-mon[35411]: pgmap v4614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:06:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:44 smithi114 ceph-mon[30820]: pgmap v4615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:06:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:44 smithi116 ceph-mon[35411]: pgmap v4615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:06:47.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:46 smithi114 ceph-mon[30820]: pgmap v4616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:46 smithi116 ceph-mon[35411]: pgmap v4616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:06:48.415 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:06:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:06:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:48 smithi114 ceph-mon[30820]: pgmap v4617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:48 smithi116 ceph-mon[35411]: pgmap v4617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:06:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:06:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:06:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:06:50.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:06:50.196+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:06:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:06:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:06:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:06:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:06:50.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:06:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:06:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:06:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:06:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:51 smithi114 ceph-mon[30820]: pgmap v4618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:06:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:51 smithi114 ceph-mon[30820]: pgmap v4619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:06:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:06:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:06:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:06:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:51 smithi116 ceph-mon[35411]: pgmap v4618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:51 smithi116 ceph-mon[35411]: pgmap v4619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:06:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:53 smithi114 ceph-mon[30820]: pgmap v4620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:06:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:53 smithi116 ceph-mon[35411]: pgmap v4620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:06:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:55 smithi114 ceph-mon[30820]: pgmap v4621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:06:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:55 smithi116 ceph-mon[35411]: pgmap v4621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:06:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:57 smithi114 ceph-mon[30820]: pgmap v4622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:06:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:57 smithi116 ceph-mon[35411]: pgmap v4622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:06:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:06:59 smithi114 ceph-mon[30820]: pgmap v4623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:06:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:06:59 smithi116 ceph-mon[35411]: pgmap v4623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:07:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:01 smithi114 ceph-mon[30820]: pgmap v4624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:07:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:01 smithi116 ceph-mon[35411]: pgmap v4624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:07:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:03 smithi114 ceph-mon[30820]: pgmap v4625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:03 smithi116 ceph-mon[35411]: pgmap v4625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:05 smithi114 ceph-mon[30820]: pgmap v4626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:05.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:05 smithi116 ceph-mon[35411]: pgmap v4626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:07 smithi114 ceph-mon[30820]: pgmap v4627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:07 smithi116 ceph-mon[35411]: pgmap v4627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:09 smithi114 ceph-mon[30820]: pgmap v4628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:09 smithi116 ceph-mon[35411]: pgmap v4628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:11 smithi114 ceph-mon[30820]: pgmap v4629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:11 smithi116 ceph-mon[35411]: pgmap v4629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:13 smithi114 ceph-mon[30820]: pgmap v4630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:13.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:13 smithi116 ceph-mon[35411]: pgmap v4630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:15 smithi114 ceph-mon[30820]: pgmap v4631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:15 smithi116 ceph-mon[35411]: pgmap v4631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:17 smithi114 ceph-mon[30820]: pgmap v4632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:17 smithi116 ceph-mon[35411]: pgmap v4632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:19 smithi114 ceph-mon[30820]: pgmap v4633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:19 smithi116 ceph-mon[35411]: pgmap v4633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:21 smithi114 ceph-mon[30820]: pgmap v4634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:21 smithi116 ceph-mon[35411]: pgmap v4634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:23 smithi114 ceph-mon[30820]: pgmap v4635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:23 smithi116 ceph-mon[35411]: pgmap v4635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:25 smithi114 ceph-mon[30820]: pgmap v4636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:25 smithi116 ceph-mon[35411]: pgmap v4636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:27 smithi114 ceph-mon[30820]: pgmap v4637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:27 smithi116 ceph-mon[35411]: pgmap v4637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:29 smithi114 ceph-mon[30820]: pgmap v4638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:29 smithi116 ceph-mon[35411]: pgmap v4638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:31 smithi114 ceph-mon[30820]: pgmap v4639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:31 smithi116 ceph-mon[35411]: pgmap v4639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:33 smithi114 ceph-mon[30820]: pgmap v4640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:33 smithi116 ceph-mon[35411]: pgmap v4640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:35 smithi114 ceph-mon[30820]: pgmap v4641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:35 smithi116 ceph-mon[35411]: pgmap v4641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:37 smithi114 ceph-mon[30820]: pgmap v4642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:37 smithi116 ceph-mon[35411]: pgmap v4642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:39 smithi114 ceph-mon[30820]: pgmap v4643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:39 smithi116 ceph-mon[35411]: pgmap v4643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:41 smithi114 ceph-mon[30820]: pgmap v4644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:41 smithi116 ceph-mon[35411]: pgmap v4644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:43 smithi114 ceph-mon[30820]: pgmap v4645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:43 smithi116 ceph-mon[35411]: pgmap v4645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:45 smithi114 ceph-mon[30820]: pgmap v4646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:45 smithi116 ceph-mon[35411]: pgmap v4646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:07:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:47 smithi114 ceph-mon[30820]: pgmap v4647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:47 smithi116 ceph-mon[35411]: pgmap v4647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:07:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:49 smithi114 ceph-mon[30820]: pgmap v4648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:49 smithi116 ceph-mon[35411]: pgmap v4648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:07:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:07:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:07:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:51 smithi114 ceph-mon[30820]: pgmap v4649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:51.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:51 smithi116 ceph-mon[35411]: pgmap v4649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:07:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-mon[30820]: pgmap v4650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:07:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:07:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:53.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:07:53.556+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:07:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:07:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:07:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:07:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:53 smithi116 ceph-mon[35411]: pgmap v4650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:07:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:07:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:07:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:07:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:07:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:07:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:07:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:07:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:07:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:07:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:07:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:07:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:07:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:07:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:07:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:07:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:07:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:07:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:07:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:07:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:07:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:07:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:07:55.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:07:55.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:07:55.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:07:55.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:07:55.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:54 smithi114 ceph-mon[30820]: pgmap v4651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:07:55.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:54 smithi114 ceph-mon[30820]: pgmap v4652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:07:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:07:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:07:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:07:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:07:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:07:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:07:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:07:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:07:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:07:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:07:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:07:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:54 smithi116 ceph-mon[35411]: pgmap v4651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:07:55.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:54 smithi116 ceph-mon[35411]: pgmap v4652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:07:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:56 smithi114 ceph-mon[30820]: pgmap v4653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:07:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:56 smithi116 ceph-mon[35411]: pgmap v4653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:07:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:07:58 smithi114 ceph-mon[30820]: pgmap v4654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:07:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:07:58 smithi116 ceph-mon[35411]: pgmap v4654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:08:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:00 smithi114 ceph-mon[30820]: pgmap v4655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:08:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:00 smithi116 ceph-mon[35411]: pgmap v4655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:08:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:02 smithi114 ceph-mon[30820]: pgmap v4656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:08:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:02 smithi116 ceph-mon[35411]: pgmap v4656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:08:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:04 smithi114 ceph-mon[30820]: pgmap v4657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:08:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:04 smithi116 ceph-mon[35411]: pgmap v4657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:08:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:06 smithi114 ceph-mon[30820]: pgmap v4658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:06 smithi116 ceph-mon[35411]: pgmap v4658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:08 smithi114 ceph-mon[30820]: pgmap v4659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:08 smithi116 ceph-mon[35411]: pgmap v4659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:10 smithi114 ceph-mon[30820]: pgmap v4660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:10 smithi116 ceph-mon[35411]: pgmap v4660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:12 smithi114 ceph-mon[30820]: pgmap v4661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:12 smithi116 ceph-mon[35411]: pgmap v4661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:14 smithi114 ceph-mon[30820]: pgmap v4662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:14 smithi116 ceph-mon[35411]: pgmap v4662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:16 smithi114 ceph-mon[30820]: pgmap v4663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:16 smithi116 ceph-mon[35411]: pgmap v4663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:18 smithi114 ceph-mon[30820]: pgmap v4664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:18 smithi116 ceph-mon[35411]: pgmap v4664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:20 smithi114 ceph-mon[30820]: pgmap v4665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:20 smithi116 ceph-mon[35411]: pgmap v4665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:22 smithi114 ceph-mon[30820]: pgmap v4666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:22 smithi116 ceph-mon[35411]: pgmap v4666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:24 smithi114 ceph-mon[30820]: pgmap v4667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:24 smithi116 ceph-mon[35411]: pgmap v4667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:26 smithi114 ceph-mon[30820]: pgmap v4668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:26 smithi116 ceph-mon[35411]: pgmap v4668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:28 smithi114 ceph-mon[30820]: pgmap v4669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:28 smithi116 ceph-mon[35411]: pgmap v4669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:30 smithi114 ceph-mon[30820]: pgmap v4670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:30 smithi116 ceph-mon[35411]: pgmap v4670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:32 smithi114 ceph-mon[30820]: pgmap v4671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:32 smithi116 ceph-mon[35411]: pgmap v4671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:34 smithi114 ceph-mon[30820]: pgmap v4672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:34 smithi116 ceph-mon[35411]: pgmap v4672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:36 smithi114 ceph-mon[30820]: pgmap v4673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:36 smithi116 ceph-mon[35411]: pgmap v4673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:38 smithi114 ceph-mon[30820]: pgmap v4674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:38 smithi116 ceph-mon[35411]: pgmap v4674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:40 smithi114 ceph-mon[30820]: pgmap v4675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:40 smithi116 ceph-mon[35411]: pgmap v4675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:42 smithi114 ceph-mon[30820]: pgmap v4676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:42 smithi116 ceph-mon[35411]: pgmap v4676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:44 smithi114 ceph-mon[30820]: pgmap v4677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:44 smithi116 ceph-mon[35411]: pgmap v4677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:46 smithi114 ceph-mon[30820]: pgmap v4678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:46 smithi116 ceph-mon[35411]: pgmap v4678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:48 smithi114 ceph-mon[30820]: pgmap v4679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:48 smithi116 ceph-mon[35411]: pgmap v4679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:50 smithi114 ceph-mon[30820]: pgmap v4680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:50 smithi116 ceph-mon[35411]: pgmap v4680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:08:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:52 smithi114 ceph-mon[30820]: pgmap v4681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:52 smithi116 ceph-mon[35411]: pgmap v4681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:08:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:08:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:08:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:54 smithi114 ceph-mon[30820]: pgmap v4682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:55.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:54 smithi116 ceph-mon[35411]: pgmap v4682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:08:55.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:55.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:55.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:55.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-mon[30820]: pgmap v4683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:08:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:08:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:08:56.852+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:08:57.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:08:57.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:08:57.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:08:57.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:08:57.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:08:57.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:08:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:08:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:56 smithi116 ceph-mon[35411]: pgmap v4683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:08:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:08:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:08:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:08:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:57 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:08:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:08:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:08:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:08:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:08:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:08:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:08:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:08:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:08:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:08:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:57 smithi114 ceph-mon[30820]: pgmap v4684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:08:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:57 smithi114 ceph-mon[30820]: pgmap v4685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:57 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:08:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:08:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:08:58.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:57 smithi116 ceph-mon[35411]: pgmap v4684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:08:58.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:57 smithi116 ceph-mon[35411]: pgmap v4685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:08:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:08:58 smithi116 ceph-mon[35411]: pgmap v4686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:08:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:08:58 smithi114 ceph-mon[30820]: pgmap v4686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:09:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:00 smithi114 ceph-mon[30820]: pgmap v4687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:09:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:00 smithi116 ceph-mon[35411]: pgmap v4687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:09:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:02 smithi114 ceph-mon[30820]: pgmap v4688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:09:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:02 smithi116 ceph-mon[35411]: pgmap v4688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:09:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:04 smithi114 ceph-mon[30820]: pgmap v4689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:09:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:04 smithi116 ceph-mon[35411]: pgmap v4689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:09:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:06 smithi114 ceph-mon[30820]: pgmap v4690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:09:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:06 smithi116 ceph-mon[35411]: pgmap v4690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:09:09.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:08 smithi114 ceph-mon[30820]: pgmap v4691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:09.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:08 smithi116 ceph-mon[35411]: pgmap v4691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:10 smithi114 ceph-mon[30820]: pgmap v4692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:10 smithi116 ceph-mon[35411]: pgmap v4692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:12 smithi114 ceph-mon[30820]: pgmap v4693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:12 smithi116 ceph-mon[35411]: pgmap v4693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:14 smithi114 ceph-mon[30820]: pgmap v4694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:14 smithi116 ceph-mon[35411]: pgmap v4694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:16 smithi114 ceph-mon[30820]: pgmap v4695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:16 smithi116 ceph-mon[35411]: pgmap v4695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:18 smithi114 ceph-mon[30820]: pgmap v4696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:18 smithi116 ceph-mon[35411]: pgmap v4696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:20 smithi114 ceph-mon[30820]: pgmap v4697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:20 smithi116 ceph-mon[35411]: pgmap v4697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:22 smithi114 ceph-mon[30820]: pgmap v4698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:22 smithi116 ceph-mon[35411]: pgmap v4698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:24 smithi114 ceph-mon[30820]: pgmap v4699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:24 smithi116 ceph-mon[35411]: pgmap v4699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:26 smithi114 ceph-mon[30820]: pgmap v4700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:26 smithi116 ceph-mon[35411]: pgmap v4700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:28 smithi114 ceph-mon[30820]: pgmap v4701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:28 smithi116 ceph-mon[35411]: pgmap v4701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:30 smithi114 ceph-mon[30820]: pgmap v4702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:30 smithi116 ceph-mon[35411]: pgmap v4702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:32 smithi114 ceph-mon[30820]: pgmap v4703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:32 smithi116 ceph-mon[35411]: pgmap v4703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:34 smithi114 ceph-mon[30820]: pgmap v4704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:34 smithi116 ceph-mon[35411]: pgmap v4704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:36 smithi114 ceph-mon[30820]: pgmap v4705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:36 smithi116 ceph-mon[35411]: pgmap v4705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:39.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:38 smithi114 ceph-mon[30820]: pgmap v4706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:39.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:38 smithi116 ceph-mon[35411]: pgmap v4706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:40 smithi114 ceph-mon[30820]: pgmap v4707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:40 smithi116 ceph-mon[35411]: pgmap v4707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:42 smithi114 ceph-mon[30820]: pgmap v4708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:42 smithi116 ceph-mon[35411]: pgmap v4708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:44 smithi114 ceph-mon[30820]: pgmap v4709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:44 smithi116 ceph-mon[35411]: pgmap v4709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:46 smithi114 ceph-mon[30820]: pgmap v4710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:46 smithi116 ceph-mon[35411]: pgmap v4710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:48 smithi114 ceph-mon[30820]: pgmap v4711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:48 smithi116 ceph-mon[35411]: pgmap v4711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:50 smithi114 ceph-mon[30820]: pgmap v4712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:50 smithi116 ceph-mon[35411]: pgmap v4712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:09:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:52 smithi114 ceph-mon[30820]: pgmap v4713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:52 smithi116 ceph-mon[35411]: pgmap v4713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:54 smithi114 ceph-mon[30820]: pgmap v4714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:54 smithi116 ceph-mon[35411]: pgmap v4714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:09:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:56 smithi114 ceph-mon[30820]: pgmap v4715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:56 smithi116 ceph-mon[35411]: pgmap v4715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:09:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:09:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:09:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:09:58 smithi114 ceph-mon[30820]: pgmap v4716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:09:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:09:58 smithi116 ceph-mon[35411]: pgmap v4716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:10:00.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:10:00.212+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:10:00.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:10:00.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:10:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:10:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:10:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:10:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:10:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:10:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:10:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:10:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:10:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:10:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:10:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:01 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:10:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:10:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:10:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:01 smithi114 ceph-mon[30820]: pgmap v4717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:01 smithi114 ceph-mon[30820]: pgmap v4718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:10:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:10:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:01 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:10:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:10:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:10:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:10:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:10:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:10:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:01 smithi116 ceph-mon[35411]: pgmap v4717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:01 smithi116 ceph-mon[35411]: pgmap v4718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:10:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:10:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:03 smithi114 ceph-mon[30820]: pgmap v4719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:10:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:03 smithi116 ceph-mon[35411]: pgmap v4719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:10:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:05 smithi114 ceph-mon[30820]: pgmap v4720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:10:05.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:05 smithi116 ceph-mon[35411]: pgmap v4720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:10:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:07 smithi114 ceph-mon[30820]: pgmap v4721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:10:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:07 smithi116 ceph-mon[35411]: pgmap v4721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:10:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:09 smithi114 ceph-mon[30820]: pgmap v4722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:10:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:09 smithi116 ceph-mon[35411]: pgmap v4722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:10:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:11 smithi114 ceph-mon[30820]: pgmap v4723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:10:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:11 smithi116 ceph-mon[35411]: pgmap v4723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:10:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:13 smithi114 ceph-mon[30820]: pgmap v4724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:13 smithi116 ceph-mon[35411]: pgmap v4724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:15 smithi114 ceph-mon[30820]: pgmap v4725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:15 smithi116 ceph-mon[35411]: pgmap v4725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:17 smithi114 ceph-mon[30820]: pgmap v4726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:17 smithi116 ceph-mon[35411]: pgmap v4726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:19 smithi114 ceph-mon[30820]: pgmap v4727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:19 smithi116 ceph-mon[35411]: pgmap v4727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:21 smithi114 ceph-mon[30820]: pgmap v4728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:10:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:21 smithi116 ceph-mon[35411]: pgmap v4728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:10:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:23 smithi114 ceph-mon[30820]: pgmap v4729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:23 smithi116 ceph-mon[35411]: pgmap v4729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:25 smithi114 ceph-mon[30820]: pgmap v4730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:25 smithi116 ceph-mon[35411]: pgmap v4730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:27 smithi114 ceph-mon[30820]: pgmap v4731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:27 smithi116 ceph-mon[35411]: pgmap v4731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:29 smithi114 ceph-mon[30820]: pgmap v4732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:29 smithi116 ceph-mon[35411]: pgmap v4732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:31 smithi114 ceph-mon[30820]: pgmap v4733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:10:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:31 smithi116 ceph-mon[35411]: pgmap v4733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:10:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:33 smithi114 ceph-mon[30820]: pgmap v4734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:33 smithi116 ceph-mon[35411]: pgmap v4734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:35 smithi114 ceph-mon[30820]: pgmap v4735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:35 smithi116 ceph-mon[35411]: pgmap v4735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:37 smithi114 ceph-mon[30820]: pgmap v4736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:37 smithi116 ceph-mon[35411]: pgmap v4736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:39 smithi114 ceph-mon[30820]: pgmap v4737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:39 smithi116 ceph-mon[35411]: pgmap v4737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:41 smithi114 ceph-mon[30820]: pgmap v4738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:10:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:41 smithi116 ceph-mon[35411]: pgmap v4738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:10:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:43 smithi114 ceph-mon[30820]: pgmap v4739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:43 smithi116 ceph-mon[35411]: pgmap v4739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:45 smithi114 ceph-mon[30820]: pgmap v4740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:45 smithi116 ceph-mon[35411]: pgmap v4740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:47 smithi114 ceph-mon[30820]: pgmap v4741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:47 smithi116 ceph-mon[35411]: pgmap v4741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:49 smithi114 ceph-mon[30820]: pgmap v4742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:49 smithi116 ceph-mon[35411]: pgmap v4742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:51 smithi114 ceph-mon[30820]: pgmap v4743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:10:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:51 smithi116 ceph-mon[35411]: pgmap v4743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:10:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:53 smithi114 ceph-mon[30820]: pgmap v4744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:53 smithi116 ceph-mon[35411]: pgmap v4744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:10:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:55 smithi114 ceph-mon[30820]: pgmap v4745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:55 smithi116 ceph-mon[35411]: pgmap v4745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:10:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:57 smithi114 ceph-mon[30820]: pgmap v4746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:57 smithi116 ceph-mon[35411]: pgmap v4746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:10:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:10:59 smithi114 ceph-mon[30820]: pgmap v4747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:10:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:10:59 smithi116 ceph-mon[35411]: pgmap v4747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:11:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:11:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:01 smithi114 ceph-mon[30820]: pgmap v4748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:11:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:01 smithi116 ceph-mon[35411]: pgmap v4748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:11:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-mon[30820]: pgmap v4749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:11:03.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:11:03.572+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:11:03.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:11:03.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:11:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:11:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:03 smithi116 ceph-mon[35411]: pgmap v4749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:11:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:11:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:11:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:11:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:11:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:04 smithi114 ceph-mon[30820]: pgmap v4750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:04 smithi114 ceph-mon[30820]: pgmap v4751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:11:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:11:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:11:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:11:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:11:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:04 smithi116 ceph-mon[35411]: pgmap v4750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:04 smithi116 ceph-mon[35411]: pgmap v4751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:11:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:11:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:06 smithi114 ceph-mon[30820]: pgmap v4752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:11:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:06 smithi116 ceph-mon[35411]: pgmap v4752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:11:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:08 smithi114 ceph-mon[30820]: pgmap v4753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:11:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:08 smithi116 ceph-mon[35411]: pgmap v4753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:11:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:10 smithi114 ceph-mon[30820]: pgmap v4754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:11:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:10 smithi116 ceph-mon[35411]: pgmap v4754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:11:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:12 smithi114 ceph-mon[30820]: pgmap v4755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:11:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:12 smithi116 ceph-mon[35411]: pgmap v4755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:11:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:14 smithi114 ceph-mon[30820]: pgmap v4756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:11:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:14 smithi116 ceph-mon[35411]: pgmap v4756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:11:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:16 smithi114 ceph-mon[30820]: pgmap v4757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:16 smithi116 ceph-mon[35411]: pgmap v4757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:18 smithi114 ceph-mon[30820]: pgmap v4758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:18 smithi116 ceph-mon[35411]: pgmap v4758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:20 smithi114 ceph-mon[30820]: pgmap v4759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:11:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:20 smithi116 ceph-mon[35411]: pgmap v4759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:11:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:22 smithi114 ceph-mon[30820]: pgmap v4760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:11:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:22 smithi116 ceph-mon[35411]: pgmap v4760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:11:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:24 smithi116 ceph-mon[35411]: pgmap v4761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:24 smithi114 ceph-mon[30820]: pgmap v4761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:26 smithi116 ceph-mon[35411]: pgmap v4762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:26 smithi114 ceph-mon[30820]: pgmap v4762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:28 smithi116 ceph-mon[35411]: pgmap v4763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:28 smithi114 ceph-mon[30820]: pgmap v4763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:30 smithi114 ceph-mon[30820]: pgmap v4764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:11:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:30 smithi116 ceph-mon[35411]: pgmap v4764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:11:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:32 smithi114 ceph-mon[30820]: pgmap v4765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:11:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:32 smithi116 ceph-mon[35411]: pgmap v4765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:11:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:34 smithi114 ceph-mon[30820]: pgmap v4766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:11:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:34 smithi116 ceph-mon[35411]: pgmap v4766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:11:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:36 smithi114 ceph-mon[30820]: pgmap v4767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:36 smithi116 ceph-mon[35411]: pgmap v4767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:38 smithi114 ceph-mon[30820]: pgmap v4768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:11:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:38 smithi116 ceph-mon[35411]: pgmap v4768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:11:40.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:40 smithi114 ceph-mon[30820]: pgmap v4769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:11:40.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:40 smithi116 ceph-mon[35411]: pgmap v4769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:11:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:42 smithi114 ceph-mon[30820]: pgmap v4770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:11:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:42 smithi116 ceph-mon[35411]: pgmap v4770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:11:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:44 smithi114 ceph-mon[30820]: pgmap v4771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:44 smithi116 ceph-mon[35411]: pgmap v4771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:46 smithi114 ceph-mon[30820]: pgmap v4772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:46 smithi116 ceph-mon[35411]: pgmap v4772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:48 smithi114 ceph-mon[30820]: pgmap v4773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:48 smithi116 ceph-mon[35411]: pgmap v4773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:50 smithi114 ceph-mon[30820]: pgmap v4774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:11:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:50 smithi116 ceph-mon[35411]: pgmap v4774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:11:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:52 smithi114 ceph-mon[30820]: pgmap v4775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:11:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:52 smithi116 ceph-mon[35411]: pgmap v4775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:11:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:54 smithi114 ceph-mon[30820]: pgmap v4776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:54 smithi116 ceph-mon[35411]: pgmap v4776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:56 smithi114 ceph-mon[30820]: pgmap v4777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:56 smithi116 ceph-mon[35411]: pgmap v4777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:11:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:11:58 smithi114 ceph-mon[30820]: pgmap v4778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:11:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:11:58 smithi116 ceph-mon[35411]: pgmap v4778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:00 smithi114 ceph-mon[30820]: pgmap v4779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:00 smithi116 ceph-mon[35411]: pgmap v4779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:02 smithi114 ceph-mon[30820]: pgmap v4780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:02 smithi116 ceph-mon[35411]: pgmap v4780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:12:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:12:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:04 smithi114 ceph-mon[30820]: pgmap v4781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:04 smithi116 ceph-mon[35411]: pgmap v4781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-mon[30820]: pgmap v4782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:07.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:12:06.932+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:12:07.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:12:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:12:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:12:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:12:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:12:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:12:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:12:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:12:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:12:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:12:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:12:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:06 smithi116 ceph-mon[35411]: pgmap v4782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:12:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:12:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:12:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:07 smithi114 ceph-mon[30820]: pgmap v4783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:07 smithi114 ceph-mon[30820]: pgmap v4784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:12:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:12:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:12:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:12:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:12:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:12:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:12:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:12:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:12:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:12:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:12:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:12:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:12:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:07 smithi116 ceph-mon[35411]: pgmap v4783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:12:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:07 smithi116 ceph-mon[35411]: pgmap v4784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:12:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:12:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:08 smithi114 ceph-mon[30820]: pgmap v4785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:12:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:08 smithi116 ceph-mon[35411]: pgmap v4785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:12:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:10 smithi114 ceph-mon[30820]: pgmap v4786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:12:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:10 smithi116 ceph-mon[35411]: pgmap v4786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:12:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:13 smithi114 ceph-mon[30820]: pgmap v4787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:12:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:13 smithi116 ceph-mon[35411]: pgmap v4787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:12:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:14 smithi114 ceph-mon[30820]: pgmap v4788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:12:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:14 smithi116 ceph-mon[35411]: pgmap v4788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:12:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:16 smithi114 ceph-mon[30820]: pgmap v4789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:12:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:17 smithi116 ceph-mon[35411]: pgmap v4789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:12:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:19 smithi114 ceph-mon[30820]: pgmap v4790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:19 smithi116 ceph-mon[35411]: pgmap v4790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:21 smithi114 ceph-mon[30820]: pgmap v4791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:21 smithi116 ceph-mon[35411]: pgmap v4791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:23 smithi114 ceph-mon[30820]: pgmap v4792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:23 smithi116 ceph-mon[35411]: pgmap v4792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:25 smithi114 ceph-mon[30820]: pgmap v4793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:25 smithi116 ceph-mon[35411]: pgmap v4793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:26 smithi114 ceph-mon[30820]: pgmap v4794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:26 smithi116 ceph-mon[35411]: pgmap v4794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:30 smithi114 ceph-mon[30820]: pgmap v4795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:30 smithi116 ceph-mon[35411]: pgmap v4795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:32 smithi114 ceph-mon[30820]: pgmap v4796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:32 smithi116 ceph-mon[35411]: pgmap v4796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:34 smithi114 ceph-mon[30820]: pgmap v4797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:34 smithi116 ceph-mon[35411]: pgmap v4797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:36 smithi114 ceph-mon[30820]: pgmap v4798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:36 smithi116 ceph-mon[35411]: pgmap v4798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:38 smithi114 ceph-mon[30820]: pgmap v4799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:38 smithi116 ceph-mon[35411]: pgmap v4799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:40 smithi114 ceph-mon[30820]: pgmap v4800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:40 smithi116 ceph-mon[35411]: pgmap v4800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:42 smithi114 ceph-mon[30820]: pgmap v4801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:42 smithi116 ceph-mon[35411]: pgmap v4801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:44 smithi114 ceph-mon[30820]: pgmap v4802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:44 smithi116 ceph-mon[35411]: pgmap v4802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:46 smithi114 ceph-mon[30820]: pgmap v4803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:46 smithi116 ceph-mon[35411]: pgmap v4803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:48 smithi114 ceph-mon[30820]: pgmap v4804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:48 smithi116 ceph-mon[35411]: pgmap v4804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:50 smithi114 ceph-mon[30820]: pgmap v4805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:50 smithi116 ceph-mon[35411]: pgmap v4805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:52 smithi114 ceph-mon[30820]: pgmap v4806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:52 smithi116 ceph-mon[35411]: pgmap v4806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:12:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:54 smithi114 ceph-mon[30820]: pgmap v4807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:54 smithi116 ceph-mon[35411]: pgmap v4807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:12:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:56 smithi114 ceph-mon[30820]: pgmap v4808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:56 smithi116 ceph-mon[35411]: pgmap v4808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:12:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:12:58 smithi114 ceph-mon[30820]: pgmap v4809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:12:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:12:58 smithi116 ceph-mon[35411]: pgmap v4809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:00 smithi114 ceph-mon[30820]: pgmap v4810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:00 smithi116 ceph-mon[35411]: pgmap v4810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:02 smithi114 ceph-mon[30820]: pgmap v4811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:13:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:02 smithi116 ceph-mon[35411]: pgmap v4811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:13:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:04 smithi114 ceph-mon[30820]: pgmap v4812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:04 smithi116 ceph-mon[35411]: pgmap v4812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:06 smithi114 ceph-mon[30820]: pgmap v4813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:06 smithi116 ceph-mon[35411]: pgmap v4813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:13:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:13:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:08 smithi114 ceph-mon[30820]: pgmap v4814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:08 smithi116 ceph-mon[35411]: pgmap v4814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-mon[30820]: pgmap v4815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:13:10.293+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:13:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:13:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:13:10.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:13:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:13:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:10 smithi116 ceph-mon[35411]: pgmap v4815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:11 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:13:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:13:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:13:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:13:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:11 smithi114 ceph-mon[30820]: pgmap v4816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:11 smithi114 ceph-mon[30820]: pgmap v4817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:13:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:13:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:11 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:13:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:13:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:13:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:13:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:13:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:13:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:13:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:13:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:13:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:13:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:13:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:13:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:11 smithi116 ceph-mon[35411]: pgmap v4816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:13:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:11 smithi116 ceph-mon[35411]: pgmap v4817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:13:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:13:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:13 smithi114 ceph-mon[30820]: pgmap v4818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:13:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:13 smithi116 ceph-mon[35411]: pgmap v4818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:13:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:15 smithi114 ceph-mon[30820]: pgmap v4819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:13:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:15 smithi116 ceph-mon[35411]: pgmap v4819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:13:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:17 smithi114 ceph-mon[30820]: pgmap v4820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:13:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:17 smithi116 ceph-mon[35411]: pgmap v4820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:13:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:19 smithi114 ceph-mon[30820]: pgmap v4821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:13:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:19 smithi116 ceph-mon[35411]: pgmap v4821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:13:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:21 smithi114 ceph-mon[30820]: pgmap v4822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:13:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:21 smithi116 ceph-mon[35411]: pgmap v4822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:13:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:23 smithi114 ceph-mon[30820]: pgmap v4823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:13:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:23 smithi116 ceph-mon[35411]: pgmap v4823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:13:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:25 smithi114 ceph-mon[30820]: pgmap v4824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:25 smithi116 ceph-mon[35411]: pgmap v4824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:27 smithi114 ceph-mon[30820]: pgmap v4825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:27 smithi116 ceph-mon[35411]: pgmap v4825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:29 smithi114 ceph-mon[30820]: pgmap v4826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:29 smithi116 ceph-mon[35411]: pgmap v4826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:31 smithi114 ceph-mon[30820]: pgmap v4827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:13:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:31 smithi116 ceph-mon[35411]: pgmap v4827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:13:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:33 smithi114 ceph-mon[30820]: pgmap v4828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:13:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:33 smithi116 ceph-mon[35411]: pgmap v4828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:13:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:35 smithi114 ceph-mon[30820]: pgmap v4829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:35 smithi116 ceph-mon[35411]: pgmap v4829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:37 smithi114 ceph-mon[30820]: pgmap v4830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:37 smithi116 ceph-mon[35411]: pgmap v4830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:39 smithi114 ceph-mon[30820]: pgmap v4831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:39.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:39 smithi116 ceph-mon[35411]: pgmap v4831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:41 smithi114 ceph-mon[30820]: pgmap v4832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:13:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:41 smithi116 ceph-mon[35411]: pgmap v4832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:13:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:43 smithi114 ceph-mon[30820]: pgmap v4833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:13:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:43 smithi116 ceph-mon[35411]: pgmap v4833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:13:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:45 smithi114 ceph-mon[30820]: pgmap v4834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:45 smithi116 ceph-mon[35411]: pgmap v4834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:47 smithi116 ceph-mon[35411]: pgmap v4835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:47 smithi114 ceph-mon[30820]: pgmap v4835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:49 smithi116 ceph-mon[35411]: pgmap v4836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:49 smithi114 ceph-mon[30820]: pgmap v4836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:51 smithi116 ceph-mon[35411]: pgmap v4837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:13:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:51 smithi114 ceph-mon[30820]: pgmap v4837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:13:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:53 smithi116 ceph-mon[35411]: pgmap v4838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:13:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:53 smithi114 ceph-mon[30820]: pgmap v4838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:13:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:55 smithi114 ceph-mon[30820]: pgmap v4839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:55 smithi116 ceph-mon[35411]: pgmap v4839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:13:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:57 smithi114 ceph-mon[30820]: pgmap v4840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:57 smithi116 ceph-mon[35411]: pgmap v4840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:13:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:13:59 smithi114 ceph-mon[30820]: pgmap v4841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:13:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:13:59 smithi116 ceph-mon[35411]: pgmap v4841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:01 smithi114 ceph-mon[30820]: pgmap v4842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:14:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:01 smithi116 ceph-mon[35411]: pgmap v4842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:14:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:03 smithi114 ceph-mon[30820]: pgmap v4843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:14:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:03 smithi116 ceph-mon[35411]: pgmap v4843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:14:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:05 smithi114 ceph-mon[30820]: pgmap v4844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:14:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:05 smithi116 ceph-mon[35411]: pgmap v4844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:14:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:07 smithi114 ceph-mon[30820]: pgmap v4845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:14:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:07 smithi116 ceph-mon[35411]: pgmap v4845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:14:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:09 smithi114 ceph-mon[30820]: pgmap v4846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:09 smithi116 ceph-mon[35411]: pgmap v4846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:14:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:14:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:11 smithi114 ceph-mon[30820]: pgmap v4847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:14:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:11 smithi116 ceph-mon[35411]: pgmap v4847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:14:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:14:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:14:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:14:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:14:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-mon[30820]: pgmap v4848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:14:13.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:14:13.652+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:14:13.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:14:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:14:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:14:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:13 smithi116 ceph-mon[35411]: pgmap v4848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:14:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:14 smithi114 ceph-mon[30820]: pgmap v4849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:14 smithi114 ceph-mon[30820]: pgmap v4850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:14:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:14:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:14:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:14:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:14:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:14:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:14:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:14 smithi116 ceph-mon[35411]: pgmap v4849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:14 smithi116 ceph-mon[35411]: pgmap v4850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:14:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:14:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:16 smithi114 ceph-mon[30820]: pgmap v4851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:14:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:16 smithi116 ceph-mon[35411]: pgmap v4851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:14:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:18 smithi114 ceph-mon[30820]: pgmap v4852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:14:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:18 smithi116 ceph-mon[35411]: pgmap v4852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:14:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:20 smithi114 ceph-mon[30820]: pgmap v4853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:14:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:20 smithi116 ceph-mon[35411]: pgmap v4853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:14:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:22 smithi114 ceph-mon[30820]: pgmap v4854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:14:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:22 smithi116 ceph-mon[35411]: pgmap v4854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:14:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:24 smithi114 ceph-mon[30820]: pgmap v4855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:14:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:24 smithi116 ceph-mon[35411]: pgmap v4855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:14:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:26 smithi114 ceph-mon[30820]: pgmap v4856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:14:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:26 smithi116 ceph-mon[35411]: pgmap v4856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:14:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:28 smithi114 ceph-mon[30820]: pgmap v4857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:28 smithi116 ceph-mon[35411]: pgmap v4857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:30 smithi114 ceph-mon[30820]: pgmap v4858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:14:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:30 smithi116 ceph-mon[35411]: pgmap v4858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:14:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:32 smithi114 ceph-mon[30820]: pgmap v4859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:14:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:32 smithi116 ceph-mon[35411]: pgmap v4859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:14:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:34 smithi114 ceph-mon[30820]: pgmap v4860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:34 smithi116 ceph-mon[35411]: pgmap v4860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:36 smithi114 ceph-mon[30820]: pgmap v4861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:14:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:36 smithi116 ceph-mon[35411]: pgmap v4861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:14:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:38 smithi114 ceph-mon[30820]: pgmap v4862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:38 smithi116 ceph-mon[35411]: pgmap v4862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:40 smithi114 ceph-mon[30820]: pgmap v4863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:14:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:40 smithi116 ceph-mon[35411]: pgmap v4863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:14:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:42 smithi114 ceph-mon[30820]: pgmap v4864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:14:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:42 smithi116 ceph-mon[35411]: pgmap v4864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:14:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:44 smithi114 ceph-mon[30820]: pgmap v4865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:44 smithi116 ceph-mon[35411]: pgmap v4865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:46 smithi114 ceph-mon[30820]: pgmap v4866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:14:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:46 smithi116 ceph-mon[35411]: pgmap v4866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:14:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:48 smithi114 ceph-mon[30820]: pgmap v4867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:48 smithi116 ceph-mon[35411]: pgmap v4867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:50 smithi114 ceph-mon[30820]: pgmap v4868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:14:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:50 smithi116 ceph-mon[35411]: pgmap v4868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:14:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:52 smithi114 ceph-mon[30820]: pgmap v4869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:14:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:52 smithi116 ceph-mon[35411]: pgmap v4869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:14:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:54 smithi114 ceph-mon[30820]: pgmap v4870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:54 smithi116 ceph-mon[35411]: pgmap v4870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:56 smithi114 ceph-mon[30820]: pgmap v4871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:14:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:56 smithi116 ceph-mon[35411]: pgmap v4871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:14:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:14:58 smithi114 ceph-mon[30820]: pgmap v4872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:14:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:14:58 smithi116 ceph-mon[35411]: pgmap v4872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:00 smithi114 ceph-mon[30820]: pgmap v4873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:15:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:00 smithi116 ceph-mon[35411]: pgmap v4873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:15:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:02 smithi114 ceph-mon[30820]: pgmap v4874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:02 smithi116 ceph-mon[35411]: pgmap v4874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:04 smithi114 ceph-mon[30820]: pgmap v4875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:04 smithi116 ceph-mon[35411]: pgmap v4875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:06 smithi114 ceph-mon[30820]: pgmap v4876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:06 smithi116 ceph-mon[35411]: pgmap v4876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:08 smithi114 ceph-mon[30820]: pgmap v4877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:08 smithi116 ceph-mon[35411]: pgmap v4877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:10 smithi114 ceph-mon[30820]: pgmap v4878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:15:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:10 smithi116 ceph-mon[35411]: pgmap v4878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:15:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:12 smithi114 ceph-mon[30820]: pgmap v4879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:12 smithi116 ceph-mon[35411]: pgmap v4879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: pgmap v4880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: pgmap v4880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:15:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:15:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:15:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:15:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:15 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:15:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:15 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:15:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:15 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:15:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:15 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:15:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:15 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:15:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:15 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:15:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:15 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:15:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:15 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:15:17.014 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:16 smithi114 ceph-mon[30820]: pgmap v4881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:16 smithi116 ceph-mon[35411]: pgmap v4881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:15:17.012+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:15:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:15:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:15:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:15:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:15:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:15:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:15:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:15:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:15:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:15:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:15:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:15:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:15:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:15:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:15:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:15:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:15:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:15:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:15:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:18 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:15:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:15:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:15:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:15:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:15:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:15:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:15:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:15:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:15:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:15:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:18 smithi114 ceph-mon[30820]: pgmap v4882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:15:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:18 smithi114 ceph-mon[30820]: pgmap v4883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:15:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:18 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:15:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:15:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:15:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:15:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:15:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:15:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:15:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:15:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:15:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:15:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:18 smithi116 ceph-mon[35411]: pgmap v4882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:15:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:18 smithi116 ceph-mon[35411]: pgmap v4883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:15:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:20 smithi114 ceph-mon[30820]: pgmap v4884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:15:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:20 smithi116 ceph-mon[35411]: pgmap v4884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:15:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:22 smithi114 ceph-mon[30820]: pgmap v4885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:15:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:22 smithi116 ceph-mon[35411]: pgmap v4885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:15:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:24 smithi114 ceph-mon[30820]: pgmap v4886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:15:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:24 smithi116 ceph-mon[35411]: pgmap v4886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:15:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:26 smithi114 ceph-mon[30820]: pgmap v4887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:15:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:26 smithi116 ceph-mon[35411]: pgmap v4887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:15:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:28 smithi114 ceph-mon[30820]: pgmap v4888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:15:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:28 smithi116 ceph-mon[35411]: pgmap v4888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:15:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:30 smithi114 ceph-mon[30820]: pgmap v4889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:30 smithi116 ceph-mon[35411]: pgmap v4889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:32 smithi114 ceph-mon[30820]: pgmap v4890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:32 smithi116 ceph-mon[35411]: pgmap v4890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:34 smithi114 ceph-mon[30820]: pgmap v4891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:34 smithi116 ceph-mon[35411]: pgmap v4891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:36 smithi114 ceph-mon[30820]: pgmap v4892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:15:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:36 smithi116 ceph-mon[35411]: pgmap v4892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:15:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:38 smithi114 ceph-mon[30820]: pgmap v4893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:38 smithi116 ceph-mon[35411]: pgmap v4893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:40 smithi114 ceph-mon[30820]: pgmap v4894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:40 smithi116 ceph-mon[35411]: pgmap v4894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:42 smithi114 ceph-mon[30820]: pgmap v4895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:42 smithi116 ceph-mon[35411]: pgmap v4895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:44 smithi114 ceph-mon[30820]: pgmap v4896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:44 smithi116 ceph-mon[35411]: pgmap v4896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:15:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:46 smithi114 ceph-mon[30820]: pgmap v4897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:15:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:46 smithi116 ceph-mon[35411]: pgmap v4897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:15:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:48 smithi114 ceph-mon[30820]: pgmap v4898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:48 smithi116 ceph-mon[35411]: pgmap v4898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:50 smithi114 ceph-mon[30820]: pgmap v4899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:15:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:50 smithi116 ceph-mon[35411]: pgmap v4899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:15:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:52 smithi114 ceph-mon[30820]: pgmap v4900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:52 smithi116 ceph-mon[35411]: pgmap v4900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:15:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:54 smithi114 ceph-mon[30820]: pgmap v4901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:15:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:54 smithi116 ceph-mon[35411]: pgmap v4901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:15:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:56 smithi114 ceph-mon[30820]: pgmap v4902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:15:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:56 smithi116 ceph-mon[35411]: pgmap v4902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:15:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:15:58 smithi114 ceph-mon[30820]: pgmap v4903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:15:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:15:58 smithi116 ceph-mon[35411]: pgmap v4903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:00 smithi114 ceph-mon[30820]: pgmap v4904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:00 smithi116 ceph-mon[35411]: pgmap v4904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:02 smithi114 ceph-mon[30820]: pgmap v4905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:16:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:02 smithi116 ceph-mon[35411]: pgmap v4905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:16:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:04 smithi114 ceph-mon[30820]: pgmap v4906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:04 smithi116 ceph-mon[35411]: pgmap v4906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:06 smithi114 ceph-mon[30820]: pgmap v4907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:06 smithi116 ceph-mon[35411]: pgmap v4907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:08 smithi114 ceph-mon[30820]: pgmap v4908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:08 smithi116 ceph-mon[35411]: pgmap v4908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:10 smithi114 ceph-mon[30820]: pgmap v4909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:10 smithi116 ceph-mon[35411]: pgmap v4909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:12 smithi114 ceph-mon[30820]: pgmap v4910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:16:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:12 smithi116 ceph-mon[35411]: pgmap v4910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:16:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:14 smithi114 ceph-mon[30820]: pgmap v4911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:14 smithi116 ceph-mon[35411]: pgmap v4911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:16 smithi114 ceph-mon[30820]: pgmap v4912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:16 smithi116 ceph-mon[35411]: pgmap v4912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:18.185 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:16:18.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:16:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:18 smithi114 ceph-mon[30820]: pgmap v4913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:18 smithi116 ceph-mon[35411]: pgmap v4913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:20.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:16:20.372+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:16:20.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:16:20.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:16:20.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:16:20.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:16:20.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:16:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:16:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:16:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:16:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:16:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:16:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:16:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:16:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-mon[30820]: pgmap v4914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:16:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:16:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:16:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:20 smithi116 ceph-mon[35411]: pgmap v4914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:16:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:16:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:16:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:16:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:16:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:16:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:16:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:16:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:16:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:16:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:16:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:16:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:16:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:16:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:16:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:16:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:16:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:16:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:21 smithi114 ceph-mon[30820]: pgmap v4915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:16:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:21 smithi114 ceph-mon[30820]: pgmap v4916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:16:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:16:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:16:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:16:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:16:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:16:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:16:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:16:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:16:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:16:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:16:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:21 smithi116 ceph-mon[35411]: pgmap v4915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:16:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:21 smithi116 ceph-mon[35411]: pgmap v4916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:16:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:23 smithi114 ceph-mon[30820]: pgmap v4917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:16:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:23 smithi116 ceph-mon[35411]: pgmap v4917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:16:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:25 smithi114 ceph-mon[30820]: pgmap v4918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:16:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:25 smithi116 ceph-mon[35411]: pgmap v4918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:16:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:27 smithi114 ceph-mon[30820]: pgmap v4919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:16:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:27 smithi116 ceph-mon[35411]: pgmap v4919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:16:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:29 smithi116 ceph-mon[35411]: pgmap v4920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:16:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:29 smithi114 ceph-mon[30820]: pgmap v4920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:16:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:31 smithi116 ceph-mon[35411]: pgmap v4921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:16:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:31 smithi114 ceph-mon[30820]: pgmap v4921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:16:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:33 smithi116 ceph-mon[35411]: pgmap v4922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:16:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:33 smithi114 ceph-mon[30820]: pgmap v4922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:16:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:35 smithi116 ceph-mon[35411]: pgmap v4923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:35 smithi114 ceph-mon[30820]: pgmap v4923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:37 smithi114 ceph-mon[30820]: pgmap v4924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:37 smithi116 ceph-mon[35411]: pgmap v4924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:39 smithi114 ceph-mon[30820]: pgmap v4925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:39 smithi116 ceph-mon[35411]: pgmap v4925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:41 smithi114 ceph-mon[30820]: pgmap v4926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:16:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:41 smithi116 ceph-mon[35411]: pgmap v4926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:16:44.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:43 smithi114 ceph-mon[30820]: pgmap v4927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:16:44.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:43 smithi116 ceph-mon[35411]: pgmap v4927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:16:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:45 smithi114 ceph-mon[30820]: pgmap v4928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:45 smithi116 ceph-mon[35411]: pgmap v4928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:47 smithi114 ceph-mon[30820]: pgmap v4929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:47 smithi116 ceph-mon[35411]: pgmap v4929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:49 smithi114 ceph-mon[30820]: pgmap v4930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:49 smithi116 ceph-mon[35411]: pgmap v4930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:16:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:51 smithi114 ceph-mon[30820]: pgmap v4931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:16:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:51 smithi116 ceph-mon[35411]: pgmap v4931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:16:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:53 smithi114 ceph-mon[30820]: pgmap v4932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:16:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:53 smithi116 ceph-mon[35411]: pgmap v4932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:16:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:55 smithi114 ceph-mon[30820]: pgmap v4933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:55 smithi116 ceph-mon[35411]: pgmap v4933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:16:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:57 smithi114 ceph-mon[30820]: pgmap v4934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:16:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:57 smithi116 ceph-mon[35411]: pgmap v4934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:16:59 smithi114 ceph-mon[30820]: pgmap v4935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:16:59 smithi116 ceph-mon[35411]: pgmap v4935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:01 smithi114 ceph-mon[30820]: pgmap v4936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:17:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:01 smithi116 ceph-mon[35411]: pgmap v4936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:17:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:03 smithi114 ceph-mon[30820]: pgmap v4937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:17:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:03 smithi116 ceph-mon[35411]: pgmap v4937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:17:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:05 smithi114 ceph-mon[30820]: pgmap v4938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:17:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:05 smithi116 ceph-mon[35411]: pgmap v4938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:17:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:07 smithi114 ceph-mon[30820]: pgmap v4939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:07 smithi116 ceph-mon[35411]: pgmap v4939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:09 smithi114 ceph-mon[30820]: pgmap v4940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:09 smithi116 ceph-mon[35411]: pgmap v4940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:11 smithi114 ceph-mon[30820]: pgmap v4941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:17:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:11 smithi116 ceph-mon[35411]: pgmap v4941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:17:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:13 smithi114 ceph-mon[30820]: pgmap v4942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:17:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:13 smithi116 ceph-mon[35411]: pgmap v4942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:17:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:15 smithi114 ceph-mon[30820]: pgmap v4943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:17:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:15 smithi116 ceph-mon[35411]: pgmap v4943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:17:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:17 smithi114 ceph-mon[30820]: pgmap v4944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:17 smithi116 ceph-mon[35411]: pgmap v4944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:19 smithi114 ceph-mon[30820]: pgmap v4945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:19 smithi116 ceph-mon[35411]: pgmap v4945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:21 smithi114 ceph-mon[30820]: pgmap v4946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:17:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:17:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:21 smithi116 ceph-mon[35411]: pgmap v4946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:17:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:17:23.734 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-mon[30820]: pgmap v4947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:17:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:23 smithi116 ceph-mon[35411]: pgmap v4947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:17:24.175 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:17:23.733+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:17:24.176 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:17:24.177 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:17:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:17:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:24 smithi114 ceph-mon[30820]: pgmap v4948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:24 smithi114 ceph-mon[30820]: pgmap v4949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:17:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:17:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:17:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:17:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:17:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:24 smithi116 ceph-mon[35411]: pgmap v4948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:24 smithi116 ceph-mon[35411]: pgmap v4949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:17:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:17:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:26 smithi114 ceph-mon[30820]: pgmap v4950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:17:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:26 smithi116 ceph-mon[35411]: pgmap v4950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:17:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:28 smithi114 ceph-mon[30820]: pgmap v4951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:17:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:28 smithi116 ceph-mon[35411]: pgmap v4951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:17:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:30 smithi114 ceph-mon[30820]: pgmap v4952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:17:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:30 smithi116 ceph-mon[35411]: pgmap v4952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:17:33.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:32 smithi114 ceph-mon[30820]: pgmap v4953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:17:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:32 smithi116 ceph-mon[35411]: pgmap v4953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:17:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:34 smithi114 ceph-mon[30820]: pgmap v4954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:17:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:34 smithi116 ceph-mon[35411]: pgmap v4954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:17:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:36 smithi114 ceph-mon[30820]: pgmap v4955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:17:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:36 smithi116 ceph-mon[35411]: pgmap v4955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:17:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:38 smithi114 ceph-mon[30820]: pgmap v4956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:38 smithi116 ceph-mon[35411]: pgmap v4956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:40 smithi114 ceph-mon[30820]: pgmap v4957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:17:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:40 smithi116 ceph-mon[35411]: pgmap v4957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:17:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:42 smithi114 ceph-mon[30820]: pgmap v4958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:42 smithi116 ceph-mon[35411]: pgmap v4958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:44 smithi114 ceph-mon[30820]: pgmap v4959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:45.192 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:44 smithi116 ceph-mon[35411]: pgmap v4959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:46 smithi114 ceph-mon[30820]: pgmap v4960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:17:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:46 smithi116 ceph-mon[35411]: pgmap v4960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:17:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:48 smithi114 ceph-mon[30820]: pgmap v4961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:48 smithi116 ceph-mon[35411]: pgmap v4961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:50 smithi114 ceph-mon[30820]: pgmap v4962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:17:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:50 smithi116 ceph-mon[35411]: pgmap v4962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:17:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:52 smithi114 ceph-mon[30820]: pgmap v4963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:52 smithi116 ceph-mon[35411]: pgmap v4963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:17:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:54 smithi114 ceph-mon[30820]: pgmap v4964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:54 smithi116 ceph-mon[35411]: pgmap v4964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:56 smithi114 ceph-mon[30820]: pgmap v4965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:17:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:56 smithi116 ceph-mon[35411]: pgmap v4965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:17:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:17:58 smithi114 ceph-mon[30820]: pgmap v4966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:17:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:17:58 smithi116 ceph-mon[35411]: pgmap v4966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:00 smithi114 ceph-mon[30820]: pgmap v4967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:18:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:00 smithi116 ceph-mon[35411]: pgmap v4967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:18:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:02 smithi114 ceph-mon[30820]: pgmap v4968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:02 smithi116 ceph-mon[35411]: pgmap v4968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:04 smithi114 ceph-mon[30820]: pgmap v4969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:18:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:04 smithi116 ceph-mon[35411]: pgmap v4969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:06 smithi114 ceph-mon[30820]: pgmap v4970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:06 smithi116 ceph-mon[35411]: pgmap v4970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:08 smithi114 ceph-mon[30820]: pgmap v4971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:18:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:08 smithi116 ceph-mon[35411]: pgmap v4971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:18:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:10 smithi114 ceph-mon[30820]: pgmap v4972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:18:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:10 smithi116 ceph-mon[35411]: pgmap v4972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:18:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:12 smithi114 ceph-mon[30820]: pgmap v4973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:12 smithi116 ceph-mon[35411]: pgmap v4973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:14 smithi114 ceph-mon[30820]: pgmap v4974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:14 smithi116 ceph-mon[35411]: pgmap v4974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:16 smithi114 ceph-mon[30820]: pgmap v4975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:16 smithi116 ceph-mon[35411]: pgmap v4975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:18 smithi114 ceph-mon[30820]: pgmap v4976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:18 smithi116 ceph-mon[35411]: pgmap v4976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:20 smithi114 ceph-mon[30820]: pgmap v4977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:18:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:20 smithi116 ceph-mon[35411]: pgmap v4977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:18:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:22 smithi114 ceph-mon[30820]: pgmap v4978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:22 smithi116 ceph-mon[35411]: pgmap v4978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:18:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:18:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:24 smithi114 ceph-mon[30820]: pgmap v4979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:24 smithi116 ceph-mon[35411]: pgmap v4979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:26 smithi114 ceph-mon[30820]: pgmap v4980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:26 smithi116 ceph-mon[35411]: pgmap v4980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:18:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:18:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:18:27.092+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:18:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:18:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:18:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:18:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:18:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:18:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:18:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:18:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:18:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:18:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:18:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:18:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:18:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:18:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:18:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:18:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:18:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:18:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:18:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:18:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:18:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:18:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:18:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:18:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:28 smithi114 ceph-mon[30820]: pgmap v4981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:18:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:28 smithi114 ceph-mon[30820]: pgmap v4982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:18:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:18:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:18:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:18:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:18:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:18:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:18:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:18:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:18:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:18:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:18:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:18:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:28 smithi116 ceph-mon[35411]: pgmap v4981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:18:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:28 smithi116 ceph-mon[35411]: pgmap v4982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:18:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:30 smithi114 ceph-mon[30820]: pgmap v4983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:18:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:30 smithi116 ceph-mon[35411]: pgmap v4983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:18:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:32 smithi114 ceph-mon[30820]: pgmap v4984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:18:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:32 smithi116 ceph-mon[35411]: pgmap v4984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:18:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:34 smithi114 ceph-mon[30820]: pgmap v4985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:18:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:34 smithi116 ceph-mon[35411]: pgmap v4985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:18:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:36 smithi114 ceph-mon[30820]: pgmap v4986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:18:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:36 smithi116 ceph-mon[35411]: pgmap v4986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:18:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:38 smithi114 ceph-mon[30820]: pgmap v4987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:18:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:38 smithi116 ceph-mon[35411]: pgmap v4987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:18:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:40 smithi114 ceph-mon[30820]: pgmap v4988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:40 smithi116 ceph-mon[35411]: pgmap v4988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:42 smithi114 ceph-mon[30820]: pgmap v4989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:42 smithi116 ceph-mon[35411]: pgmap v4989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:44 smithi114 ceph-mon[30820]: pgmap v4990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:44 smithi116 ceph-mon[35411]: pgmap v4990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:46 smithi114 ceph-mon[30820]: pgmap v4991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:18:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:46 smithi116 ceph-mon[35411]: pgmap v4991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:18:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:48 smithi114 ceph-mon[30820]: pgmap v4992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:48 smithi116 ceph-mon[35411]: pgmap v4992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:50 smithi114 ceph-mon[30820]: pgmap v4993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:50 smithi116 ceph-mon[35411]: pgmap v4993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:52 smithi114 ceph-mon[30820]: pgmap v4994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:52 smithi116 ceph-mon[35411]: pgmap v4994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:18:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:54 smithi114 ceph-mon[30820]: pgmap v4995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:54 smithi116 ceph-mon[35411]: pgmap v4995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:18:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:56 smithi114 ceph-mon[30820]: pgmap v4996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:18:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:56 smithi116 ceph-mon[35411]: pgmap v4996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:18:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:18:58 smithi114 ceph-mon[30820]: pgmap v4997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:18:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:18:58 smithi116 ceph-mon[35411]: pgmap v4997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:00 smithi114 ceph-mon[30820]: pgmap v4998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:00 smithi116 ceph-mon[35411]: pgmap v4998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:02 smithi114 ceph-mon[30820]: pgmap v4999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:19:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:02 smithi116 ceph-mon[35411]: pgmap v4999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:19:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:04 smithi114 ceph-mon[30820]: pgmap v5000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:04 smithi116 ceph-mon[35411]: pgmap v5000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:06 smithi114 ceph-mon[30820]: pgmap v5001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:06 smithi116 ceph-mon[35411]: pgmap v5001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:08 smithi116 ceph-mon[35411]: pgmap v5002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:08 smithi114 ceph-mon[30820]: pgmap v5002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:10 smithi116 ceph-mon[35411]: pgmap v5003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:10 smithi114 ceph-mon[30820]: pgmap v5003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:12 smithi116 ceph-mon[35411]: pgmap v5004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:19:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:12 smithi114 ceph-mon[30820]: pgmap v5004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:19:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:14 smithi114 ceph-mon[30820]: pgmap v5005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:14 smithi116 ceph-mon[35411]: pgmap v5005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:16.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:16 smithi114 ceph-mon[30820]: pgmap v5006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:16.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:16 smithi116 ceph-mon[35411]: pgmap v5006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:18 smithi114 ceph-mon[30820]: pgmap v5007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:18 smithi116 ceph-mon[35411]: pgmap v5007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:20 smithi114 ceph-mon[30820]: pgmap v5008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:20 smithi116 ceph-mon[35411]: pgmap v5008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:22 smithi114 ceph-mon[30820]: pgmap v5009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:19:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:22 smithi116 ceph-mon[35411]: pgmap v5009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:19:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:24 smithi114 ceph-mon[30820]: pgmap v5010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:24 smithi116 ceph-mon[35411]: pgmap v5010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:26 smithi114 ceph-mon[30820]: pgmap v5011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:26 smithi116 ceph-mon[35411]: pgmap v5011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:19:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:19:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:28 smithi114 ceph-mon[30820]: pgmap v5012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:28 smithi116 ceph-mon[35411]: pgmap v5012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-mon[30820]: pgmap v5013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:19:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:19:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:30.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:19:30.388+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:19:30.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:19:30.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:19:30.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:19:30.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:19:30.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:19:30.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:19:30.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:19:30.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:19:30.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:19:30.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:19:30.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:19:30.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:19:30.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:19:30.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:19:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:19:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:30 smithi116 ceph-mon[35411]: pgmap v5013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:19:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:19:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:19:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:19:31.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:19:31.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:19:31.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:19:31.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:19:31.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:19:31.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:19:31.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:19:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:19:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:19:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:19:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:19:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:19:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:19:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:19:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:19:31.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:19:31.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:19:31.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:19:31.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:19:31.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:19:31.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:19:31.923 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:19:31.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:31 smithi114 ceph-mon[30820]: pgmap v5014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:19:31.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:31 smithi114 ceph-mon[30820]: pgmap v5015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:19:31.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:19:31.947 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:19:31.947 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:19:31.947 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:19:31.947 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:19:31.948 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:31 smithi116 ceph-mon[35411]: pgmap v5014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:19:31.949 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:31 smithi116 ceph-mon[35411]: pgmap v5015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:19:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:33 smithi114 ceph-mon[30820]: pgmap v5016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:19:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:33 smithi116 ceph-mon[35411]: pgmap v5016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:19:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:35 smithi114 ceph-mon[30820]: pgmap v5017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:19:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:35 smithi116 ceph-mon[35411]: pgmap v5017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:19:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:37 smithi114 ceph-mon[30820]: pgmap v5018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:19:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:37 smithi116 ceph-mon[35411]: pgmap v5018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:19:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:39 smithi114 ceph-mon[30820]: pgmap v5019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:19:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:39 smithi116 ceph-mon[35411]: pgmap v5019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:19:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:41 smithi114 ceph-mon[30820]: pgmap v5020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:19:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:41 smithi116 ceph-mon[35411]: pgmap v5020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:19:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:43 smithi114 ceph-mon[30820]: pgmap v5021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:19:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:43 smithi116 ceph-mon[35411]: pgmap v5021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:19:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:45 smithi114 ceph-mon[30820]: pgmap v5022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:45 smithi116 ceph-mon[35411]: pgmap v5022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:47 smithi114 ceph-mon[30820]: pgmap v5023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:47 smithi116 ceph-mon[35411]: pgmap v5023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:49 smithi114 ceph-mon[30820]: pgmap v5024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:49 smithi116 ceph-mon[35411]: pgmap v5024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:51 smithi114 ceph-mon[30820]: pgmap v5025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:19:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:51 smithi116 ceph-mon[35411]: pgmap v5025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:19:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:53 smithi114 ceph-mon[30820]: pgmap v5026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:19:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:53 smithi116 ceph-mon[35411]: pgmap v5026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:19:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:55 smithi114 ceph-mon[30820]: pgmap v5027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:55 smithi116 ceph-mon[35411]: pgmap v5027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:19:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:57 smithi114 ceph-mon[30820]: pgmap v5028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:57 smithi116 ceph-mon[35411]: pgmap v5028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:19:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:19:59 smithi114 ceph-mon[30820]: pgmap v5029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:19:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:19:59 smithi116 ceph-mon[35411]: pgmap v5029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:20:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:20:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:01 smithi114 ceph-mon[30820]: pgmap v5030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:20:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:01 smithi116 ceph-mon[35411]: pgmap v5030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:20:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:03 smithi114 ceph-mon[30820]: pgmap v5031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:20:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:03 smithi116 ceph-mon[35411]: pgmap v5031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:20:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:05 smithi114 ceph-mon[30820]: pgmap v5032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:05 smithi116 ceph-mon[35411]: pgmap v5032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:07 smithi114 ceph-mon[30820]: pgmap v5033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:20:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:07 smithi116 ceph-mon[35411]: pgmap v5033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:20:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:09 smithi114 ceph-mon[30820]: pgmap v5034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:09 smithi116 ceph-mon[35411]: pgmap v5034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:11 smithi114 ceph-mon[30820]: pgmap v5035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:20:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:11 smithi116 ceph-mon[35411]: pgmap v5035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:20:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:13 smithi114 ceph-mon[30820]: pgmap v5036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:20:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:13 smithi116 ceph-mon[35411]: pgmap v5036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:15 smithi114 ceph-mon[30820]: pgmap v5037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:15 smithi116 ceph-mon[35411]: pgmap v5037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:17 smithi114 ceph-mon[30820]: pgmap v5038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:20:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:17 smithi116 ceph-mon[35411]: pgmap v5038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:20:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:19 smithi114 ceph-mon[30820]: pgmap v5039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:19 smithi116 ceph-mon[35411]: pgmap v5039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:21 smithi114 ceph-mon[30820]: pgmap v5040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:20:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:21 smithi116 ceph-mon[35411]: pgmap v5040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:20:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:23 smithi114 ceph-mon[30820]: pgmap v5041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:20:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:23 smithi116 ceph-mon[35411]: pgmap v5041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:20:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:25 smithi114 ceph-mon[30820]: pgmap v5042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:25 smithi116 ceph-mon[35411]: pgmap v5042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:27 smithi114 ceph-mon[30820]: pgmap v5043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:20:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:27 smithi116 ceph-mon[35411]: pgmap v5043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:20:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:29 smithi114 ceph-mon[30820]: pgmap v5044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:29 smithi116 ceph-mon[35411]: pgmap v5044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:20:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:20:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:31 smithi114 ceph-mon[30820]: pgmap v5045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:20:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:31 smithi116 ceph-mon[35411]: pgmap v5045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:20:33.758 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:33 smithi116 ceph-mon[35411]: pgmap v5046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:20:33.877 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-mon[30820]: pgmap v5046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:20:33.877 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:20:33.748+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:20:33.877 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:20:33.877 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:20:33.877 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:20:33.877 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:20:33.878 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:20:33.879 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:20:33.879 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:20:33.879 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:20:33.879 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:20:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:20:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:20:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:20:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:20:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:20:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:20:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:20:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:20:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:20:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:20:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:20:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:35 smithi114 ceph-mon[30820]: pgmap v5047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:35 smithi116 ceph-mon[35411]: pgmap v5047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:20:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:20:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:36 smithi114 ceph-mon[30820]: pgmap v5048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T09:20:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:36 smithi114 ceph-mon[30820]: pgmap v5049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:20:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:20:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:20:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:36 smithi116 ceph-mon[35411]: pgmap v5048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T09:20:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:36 smithi116 ceph-mon[35411]: pgmap v5049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:20:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:20:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:38 smithi114 ceph-mon[30820]: pgmap v5050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:20:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:38 smithi116 ceph-mon[35411]: pgmap v5050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:20:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:40 smithi114 ceph-mon[30820]: pgmap v5051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:20:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:40 smithi116 ceph-mon[35411]: pgmap v5051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:20:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:42 smithi114 ceph-mon[30820]: pgmap v5052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:20:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:42 smithi116 ceph-mon[35411]: pgmap v5052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:20:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:44 smithi114 ceph-mon[30820]: pgmap v5053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:20:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:44 smithi116 ceph-mon[35411]: pgmap v5053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:20:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:46 smithi114 ceph-mon[30820]: pgmap v5054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:20:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:46 smithi116 ceph-mon[35411]: pgmap v5054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:20:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:48 smithi114 ceph-mon[30820]: pgmap v5055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:49.328 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:48 smithi116 ceph-mon[35411]: pgmap v5055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:50 smithi114 ceph-mon[30820]: pgmap v5056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:50 smithi116 ceph-mon[35411]: pgmap v5056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:20:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:52 smithi114 ceph-mon[30820]: pgmap v5057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:20:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:52 smithi116 ceph-mon[35411]: pgmap v5057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:20:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:54 smithi114 ceph-mon[30820]: pgmap v5058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:54 smithi116 ceph-mon[35411]: pgmap v5058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:56 smithi114 ceph-mon[30820]: pgmap v5059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:20:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:56 smithi116 ceph-mon[35411]: pgmap v5059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:20:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:20:58 smithi116 ceph-mon[35411]: pgmap v5060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:20:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:20:58 smithi114 ceph-mon[30820]: pgmap v5060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:00 smithi116 ceph-mon[35411]: pgmap v5061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:21:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:00 smithi114 ceph-mon[30820]: pgmap v5061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:21:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:02 smithi114 ceph-mon[30820]: pgmap v5062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:21:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:02 smithi116 ceph-mon[35411]: pgmap v5062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:21:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:04 smithi114 ceph-mon[30820]: pgmap v5063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:04 smithi116 ceph-mon[35411]: pgmap v5063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:06 smithi114 ceph-mon[30820]: pgmap v5064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:06 smithi116 ceph-mon[35411]: pgmap v5064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:08 smithi114 ceph-mon[30820]: pgmap v5065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:08 smithi116 ceph-mon[35411]: pgmap v5065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:10 smithi114 ceph-mon[30820]: pgmap v5066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:21:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:10 smithi116 ceph-mon[35411]: pgmap v5066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:21:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:12 smithi114 ceph-mon[30820]: pgmap v5067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:21:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:12 smithi116 ceph-mon[35411]: pgmap v5067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:21:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:14 smithi114 ceph-mon[30820]: pgmap v5068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:14 smithi116 ceph-mon[35411]: pgmap v5068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:16 smithi114 ceph-mon[30820]: pgmap v5069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:16 smithi116 ceph-mon[35411]: pgmap v5069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:18 smithi114 ceph-mon[30820]: pgmap v5070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:18 smithi116 ceph-mon[35411]: pgmap v5070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:20 smithi116 ceph-mon[35411]: pgmap v5071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:21:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:20 smithi114 ceph-mon[30820]: pgmap v5071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:21:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:22 smithi116 ceph-mon[35411]: pgmap v5072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:21:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:22 smithi114 ceph-mon[30820]: pgmap v5072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:21:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:24 smithi116 ceph-mon[35411]: pgmap v5073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:24 smithi114 ceph-mon[30820]: pgmap v5073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:26 smithi114 ceph-mon[30820]: pgmap v5074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:26 smithi116 ceph-mon[35411]: pgmap v5074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:28 smithi114 ceph-mon[30820]: pgmap v5075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:28 smithi116 ceph-mon[35411]: pgmap v5075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:30 smithi114 ceph-mon[30820]: pgmap v5076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:21:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:30 smithi116 ceph-mon[35411]: pgmap v5076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:21:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:32 smithi114 ceph-mon[30820]: pgmap v5077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:21:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:32 smithi116 ceph-mon[35411]: pgmap v5077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:21:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:34 smithi114 ceph-mon[30820]: pgmap v5078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:34 smithi116 ceph-mon[35411]: pgmap v5078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:36 smithi114 ceph-mon[30820]: pgmap v5079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:21:37.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:21:37.172+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:21:37.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:21:37.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:21:37.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:21:37.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:21:37.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:21:37.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:21:37.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:21:37.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:21:37.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:21:37.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:21:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:36 smithi116 ceph-mon[35411]: pgmap v5079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:21:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:21:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:21:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:21:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:21:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:21:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:21:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:38 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:21:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:21:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:21:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:21:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:21:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:38 smithi114 ceph-mon[30820]: pgmap v5080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:21:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:38 smithi114 ceph-mon[30820]: pgmap v5081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:21:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:38 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:21:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:21:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:38 smithi116 ceph-mon[35411]: pgmap v5080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:21:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:38 smithi116 ceph-mon[35411]: pgmap v5081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:21:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:40 smithi114 ceph-mon[30820]: pgmap v5082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:21:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:40 smithi116 ceph-mon[35411]: pgmap v5082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:21:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:42 smithi114 ceph-mon[30820]: pgmap v5083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:21:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:42 smithi116 ceph-mon[35411]: pgmap v5083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:21:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:44 smithi114 ceph-mon[30820]: pgmap v5084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:21:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:44 smithi116 ceph-mon[35411]: pgmap v5084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:21:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:46 smithi114 ceph-mon[30820]: pgmap v5085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:21:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:46 smithi116 ceph-mon[35411]: pgmap v5085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:21:49.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:48 smithi116 ceph-mon[35411]: pgmap v5086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:21:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:48 smithi114 ceph-mon[30820]: pgmap v5086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:21:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:50 smithi114 ceph-mon[30820]: pgmap v5087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:50 smithi116 ceph-mon[35411]: pgmap v5087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:52 smithi114 ceph-mon[30820]: pgmap v5088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:52 smithi116 ceph-mon[35411]: pgmap v5088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:21:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:54 smithi114 ceph-mon[30820]: pgmap v5089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:54 smithi116 ceph-mon[35411]: pgmap v5089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:21:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:56 smithi114 ceph-mon[30820]: pgmap v5090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:21:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:56 smithi116 ceph-mon[35411]: pgmap v5090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:21:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:21:58 smithi114 ceph-mon[30820]: pgmap v5091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:21:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:21:58 smithi116 ceph-mon[35411]: pgmap v5091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:00 smithi114 ceph-mon[30820]: pgmap v5092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:00 smithi116 ceph-mon[35411]: pgmap v5092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:03 smithi114 ceph-mon[30820]: pgmap v5093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:22:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:03 smithi116 ceph-mon[35411]: pgmap v5093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:22:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:05 smithi114 ceph-mon[30820]: pgmap v5094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:05 smithi116 ceph-mon[35411]: pgmap v5094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:07 smithi114 ceph-mon[30820]: pgmap v5095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:22:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:07 smithi116 ceph-mon[35411]: pgmap v5095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:22:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:09 smithi114 ceph-mon[30820]: pgmap v5096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:09 smithi116 ceph-mon[35411]: pgmap v5096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:11 smithi114 ceph-mon[30820]: pgmap v5097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:11 smithi116 ceph-mon[35411]: pgmap v5097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:13 smithi114 ceph-mon[30820]: pgmap v5098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:22:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:13 smithi116 ceph-mon[35411]: pgmap v5098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:22:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:15 smithi114 ceph-mon[30820]: pgmap v5099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:15 smithi116 ceph-mon[35411]: pgmap v5099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:17 smithi114 ceph-mon[30820]: pgmap v5100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:22:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:17 smithi116 ceph-mon[35411]: pgmap v5100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:22:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:19 smithi114 ceph-mon[30820]: pgmap v5101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:19 smithi116 ceph-mon[35411]: pgmap v5101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:21 smithi114 ceph-mon[30820]: pgmap v5102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:21 smithi116 ceph-mon[35411]: pgmap v5102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:23 smithi114 ceph-mon[30820]: pgmap v5103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:22:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:23 smithi116 ceph-mon[35411]: pgmap v5103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:22:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:25 smithi114 ceph-mon[30820]: pgmap v5104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:25 smithi116 ceph-mon[35411]: pgmap v5104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:27 smithi114 ceph-mon[30820]: pgmap v5105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:27 smithi116 ceph-mon[35411]: pgmap v5105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:22:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:29 smithi114 ceph-mon[30820]: pgmap v5106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:29 smithi116 ceph-mon[35411]: pgmap v5106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:30 smithi114 ceph-mon[30820]: pgmap v5107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:30 smithi116 ceph-mon[35411]: pgmap v5107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:32 smithi114 ceph-mon[30820]: pgmap v5108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:22:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:32 smithi116 ceph-mon[35411]: pgmap v5108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:22:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:34 smithi114 ceph-mon[30820]: pgmap v5109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:34 smithi116 ceph-mon[35411]: pgmap v5109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:36 smithi114 ceph-mon[30820]: pgmap v5110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:22:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:36 smithi116 ceph-mon[35411]: pgmap v5110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:22:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:38 smithi114 ceph-mon[30820]: pgmap v5111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:22:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:38 smithi116 ceph-mon[35411]: pgmap v5111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:22:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-mon[30820]: pgmap v5112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:40.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:22:40.468+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:22:40.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:22:40.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:22:40.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:22:40.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:22:40.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:22:40.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:22:40.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:22:40.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:22:40.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:22:40.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:22:40.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:22:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:22:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:40 smithi116 ceph-mon[35411]: pgmap v5112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:22:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:22:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:41 smithi114 ceph-mon[30820]: pgmap v5113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:22:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:41 smithi114 ceph-mon[30820]: pgmap v5114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:22:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:22:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:22:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:41 smithi116 ceph-mon[35411]: pgmap v5113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:22:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:41 smithi116 ceph-mon[35411]: pgmap v5114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:22:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:22:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:43 smithi114 ceph-mon[30820]: pgmap v5115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:22:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:43 smithi116 ceph-mon[35411]: pgmap v5115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:22:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:45 smithi114 ceph-mon[30820]: pgmap v5116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T09:22:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:45 smithi116 ceph-mon[35411]: pgmap v5116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T09:22:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:47 smithi114 ceph-mon[30820]: pgmap v5117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:22:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:47 smithi116 ceph-mon[35411]: pgmap v5117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:22:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:49 smithi114 ceph-mon[30820]: pgmap v5118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:22:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:49 smithi116 ceph-mon[35411]: pgmap v5118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:22:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:51 smithi114 ceph-mon[30820]: pgmap v5119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2024-09-19T09:22:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:51 smithi116 ceph-mon[35411]: pgmap v5119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 6 op/s 2024-09-19T09:22:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:53 smithi114 ceph-mon[30820]: pgmap v5120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:53 smithi116 ceph-mon[35411]: pgmap v5120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:22:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:55 smithi114 ceph-mon[30820]: pgmap v5121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:22:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:55 smithi116 ceph-mon[35411]: pgmap v5121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:22:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:57 smithi114 ceph-mon[30820]: pgmap v5122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:22:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:57 smithi116 ceph-mon[35411]: pgmap v5122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:22:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:22:59 smithi114 ceph-mon[30820]: pgmap v5123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:22:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:22:59 smithi116 ceph-mon[35411]: pgmap v5123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:01 smithi114 ceph-mon[30820]: pgmap v5124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:01 smithi116 ceph-mon[35411]: pgmap v5124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:03 smithi114 ceph-mon[30820]: pgmap v5125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:03 smithi116 ceph-mon[35411]: pgmap v5125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:05 smithi114 ceph-mon[30820]: pgmap v5126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:23:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:05 smithi116 ceph-mon[35411]: pgmap v5126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:23:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:07 smithi114 ceph-mon[30820]: pgmap v5127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:23:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:07 smithi116 ceph-mon[35411]: pgmap v5127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:23:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:09 smithi114 ceph-mon[30820]: pgmap v5128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:09 smithi116 ceph-mon[35411]: pgmap v5128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:11 smithi114 ceph-mon[30820]: pgmap v5129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:11 smithi116 ceph-mon[35411]: pgmap v5129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:13 smithi114 ceph-mon[30820]: pgmap v5130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:13 smithi116 ceph-mon[35411]: pgmap v5130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:15 smithi114 ceph-mon[30820]: pgmap v5131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:23:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:15 smithi116 ceph-mon[35411]: pgmap v5131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:23:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:17 smithi114 ceph-mon[30820]: pgmap v5132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:23:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:17 smithi116 ceph-mon[35411]: pgmap v5132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:23:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:19 smithi114 ceph-mon[30820]: pgmap v5133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:19 smithi116 ceph-mon[35411]: pgmap v5133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:21 smithi114 ceph-mon[30820]: pgmap v5134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:21 smithi116 ceph-mon[35411]: pgmap v5134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:23 smithi114 ceph-mon[30820]: pgmap v5135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:23 smithi116 ceph-mon[35411]: pgmap v5135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:25 smithi114 ceph-mon[30820]: pgmap v5136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:23:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:25 smithi116 ceph-mon[35411]: pgmap v5136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:23:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:27 smithi114 ceph-mon[30820]: pgmap v5137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:23:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:27 smithi116 ceph-mon[35411]: pgmap v5137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:23:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:29 smithi114 ceph-mon[30820]: pgmap v5138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:29 smithi116 ceph-mon[35411]: pgmap v5138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:31 smithi114 ceph-mon[30820]: pgmap v5139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:31 smithi116 ceph-mon[35411]: pgmap v5139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:33 smithi114 ceph-mon[30820]: pgmap v5140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:33 smithi116 ceph-mon[35411]: pgmap v5140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:35 smithi114 ceph-mon[30820]: pgmap v5141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:23:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:35 smithi116 ceph-mon[35411]: pgmap v5141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:23:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:37 smithi114 ceph-mon[30820]: pgmap v5142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:23:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:37 smithi116 ceph-mon[35411]: pgmap v5142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:23:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:39 smithi114 ceph-mon[30820]: pgmap v5143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:39 smithi116 ceph-mon[35411]: pgmap v5143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:23:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:23:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:41 smithi114 ceph-mon[30820]: pgmap v5144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:41 smithi116 ceph-mon[35411]: pgmap v5144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-mon[30820]: pgmap v5145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:23:43.828+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:23:43.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:23:43.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:23:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:23:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:43 smithi116 ceph-mon[35411]: pgmap v5145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:44 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:23:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:23:44.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:44 smithi114 ceph-mon[30820]: pgmap v5146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:23:44.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:44 smithi114 ceph-mon[30820]: pgmap v5147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:23:44.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:44 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:23:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:44 smithi116 ceph-mon[35411]: pgmap v5146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:23:44.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:44 smithi116 ceph-mon[35411]: pgmap v5147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:23:44.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:23:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:46 smithi114 ceph-mon[30820]: pgmap v5148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:23:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:46 smithi116 ceph-mon[35411]: pgmap v5148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:23:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:48 smithi114 ceph-mon[30820]: pgmap v5149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:23:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:48 smithi116 ceph-mon[35411]: pgmap v5149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:23:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:50 smithi114 ceph-mon[30820]: pgmap v5150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:23:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:50 smithi116 ceph-mon[35411]: pgmap v5150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:23:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:52 smithi114 ceph-mon[30820]: pgmap v5151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:23:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:52 smithi116 ceph-mon[35411]: pgmap v5151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:23:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:54 smithi114 ceph-mon[30820]: pgmap v5152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:23:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:54 smithi116 ceph-mon[35411]: pgmap v5152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:23:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:56 smithi114 ceph-mon[30820]: pgmap v5153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:56 smithi116 ceph-mon[35411]: pgmap v5153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:23:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:23:58 smithi114 ceph-mon[30820]: pgmap v5154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:23:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:23:58 smithi116 ceph-mon[35411]: pgmap v5154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:00 smithi116 ceph-mon[35411]: pgmap v5155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:00 smithi114 ceph-mon[30820]: pgmap v5155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:02 smithi116 ceph-mon[35411]: pgmap v5156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:02 smithi114 ceph-mon[30820]: pgmap v5156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:04 smithi116 ceph-mon[35411]: pgmap v5157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:04 smithi114 ceph-mon[30820]: pgmap v5157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:06 smithi114 ceph-mon[30820]: pgmap v5158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:06 smithi116 ceph-mon[35411]: pgmap v5158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:08 smithi114 ceph-mon[30820]: pgmap v5159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:08 smithi116 ceph-mon[35411]: pgmap v5159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:10 smithi114 ceph-mon[30820]: pgmap v5160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:10 smithi116 ceph-mon[35411]: pgmap v5160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:12 smithi114 ceph-mon[30820]: pgmap v5161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:12 smithi116 ceph-mon[35411]: pgmap v5161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:14 smithi114 ceph-mon[30820]: pgmap v5162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:14 smithi116 ceph-mon[35411]: pgmap v5162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:16 smithi114 ceph-mon[30820]: pgmap v5163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:16 smithi116 ceph-mon[35411]: pgmap v5163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:18 smithi114 ceph-mon[30820]: pgmap v5164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:18 smithi116 ceph-mon[35411]: pgmap v5164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:21.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:20 smithi114 ceph-mon[30820]: pgmap v5165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:21.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:20 smithi116 ceph-mon[35411]: pgmap v5165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:22 smithi114 ceph-mon[30820]: pgmap v5166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:22 smithi116 ceph-mon[35411]: pgmap v5166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:24 smithi114 ceph-mon[30820]: pgmap v5167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:24 smithi116 ceph-mon[35411]: pgmap v5167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:26 smithi114 ceph-mon[30820]: pgmap v5168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:26 smithi116 ceph-mon[35411]: pgmap v5168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:28 smithi114 ceph-mon[30820]: pgmap v5169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:28 smithi116 ceph-mon[35411]: pgmap v5169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:30 smithi114 ceph-mon[30820]: pgmap v5170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:30 smithi116 ceph-mon[35411]: pgmap v5170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:32 smithi114 ceph-mon[30820]: pgmap v5171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:32 smithi116 ceph-mon[35411]: pgmap v5171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:34 smithi114 ceph-mon[30820]: pgmap v5172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:34 smithi116 ceph-mon[35411]: pgmap v5172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:36 smithi114 ceph-mon[30820]: pgmap v5173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:36 smithi116 ceph-mon[35411]: pgmap v5173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:38 smithi114 ceph-mon[30820]: pgmap v5174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:38 smithi116 ceph-mon[35411]: pgmap v5174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:40 smithi114 ceph-mon[30820]: pgmap v5175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:40 smithi116 ceph-mon[35411]: pgmap v5175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:24:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:42 smithi114 ceph-mon[30820]: pgmap v5176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:42 smithi116 ceph-mon[35411]: pgmap v5176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:24:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:24:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:24:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:45 smithi114 ceph-mon[30820]: pgmap v5177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:45 smithi116 ceph-mon[35411]: pgmap v5177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:24:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-mon[30820]: pgmap v5178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:24:47.188+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:24:47.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:24:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:24:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:24:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:47 smithi116 ceph-mon[35411]: pgmap v5178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:24:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:24:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:24:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:24:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:24:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:24:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:24:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:24:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:49 smithi114 ceph-mon[30820]: pgmap v5179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:24:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:49 smithi114 ceph-mon[30820]: pgmap v5180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:24:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:24:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:49 smithi116 ceph-mon[35411]: pgmap v5179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:24:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:49 smithi116 ceph-mon[35411]: pgmap v5180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:24:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:51 smithi114 ceph-mon[30820]: pgmap v5181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:24:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:51 smithi116 ceph-mon[35411]: pgmap v5181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:24:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:53 smithi114 ceph-mon[30820]: pgmap v5182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:24:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:53 smithi116 ceph-mon[35411]: pgmap v5182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:24:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:55 smithi114 ceph-mon[30820]: pgmap v5183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:24:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:55 smithi116 ceph-mon[35411]: pgmap v5183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:24:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:57 smithi114 ceph-mon[30820]: pgmap v5184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:24:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:57 smithi116 ceph-mon[35411]: pgmap v5184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:24:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:24:59 smithi114 ceph-mon[30820]: pgmap v5185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:24:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:24:59 smithi116 ceph-mon[35411]: pgmap v5185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:25:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:01 smithi114 ceph-mon[30820]: pgmap v5186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:01 smithi116 ceph-mon[35411]: pgmap v5186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:03 smithi114 ceph-mon[30820]: pgmap v5187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:03 smithi116 ceph-mon[35411]: pgmap v5187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:05 smithi114 ceph-mon[30820]: pgmap v5188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:05 smithi116 ceph-mon[35411]: pgmap v5188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:07 smithi114 ceph-mon[30820]: pgmap v5189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:07 smithi116 ceph-mon[35411]: pgmap v5189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:09 smithi114 ceph-mon[30820]: pgmap v5190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:09 smithi116 ceph-mon[35411]: pgmap v5190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:11 smithi114 ceph-mon[30820]: pgmap v5191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:11 smithi116 ceph-mon[35411]: pgmap v5191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:13 smithi114 ceph-mon[30820]: pgmap v5192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:13 smithi116 ceph-mon[35411]: pgmap v5192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:15 smithi114 ceph-mon[30820]: pgmap v5193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:15 smithi116 ceph-mon[35411]: pgmap v5193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:17 smithi114 ceph-mon[30820]: pgmap v5194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:17 smithi116 ceph-mon[35411]: pgmap v5194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:19 smithi114 ceph-mon[30820]: pgmap v5195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:19 smithi116 ceph-mon[35411]: pgmap v5195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:21 smithi114 ceph-mon[30820]: pgmap v5196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:21 smithi116 ceph-mon[35411]: pgmap v5196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:23 smithi114 ceph-mon[30820]: pgmap v5197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:23 smithi116 ceph-mon[35411]: pgmap v5197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:25 smithi114 ceph-mon[30820]: pgmap v5198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:25 smithi116 ceph-mon[35411]: pgmap v5198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:27 smithi114 ceph-mon[30820]: pgmap v5199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:27 smithi116 ceph-mon[35411]: pgmap v5199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:28 smithi114 ceph-mon[30820]: pgmap v5200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:28 smithi116 ceph-mon[35411]: pgmap v5200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:30 smithi114 ceph-mon[30820]: pgmap v5201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:30 smithi116 ceph-mon[35411]: pgmap v5201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:32 smithi114 ceph-mon[30820]: pgmap v5202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:32 smithi116 ceph-mon[35411]: pgmap v5202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:34 smithi114 ceph-mon[30820]: pgmap v5203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:34 smithi116 ceph-mon[35411]: pgmap v5203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:36 smithi114 ceph-mon[30820]: pgmap v5204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:36 smithi116 ceph-mon[35411]: pgmap v5204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:38 smithi114 ceph-mon[30820]: pgmap v5205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:38 smithi116 ceph-mon[35411]: pgmap v5205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:40 smithi114 ceph-mon[30820]: pgmap v5206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:40 smithi116 ceph-mon[35411]: pgmap v5206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:42 smithi114 ceph-mon[30820]: pgmap v5207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:42 smithi116 ceph-mon[35411]: pgmap v5207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:25:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:44 smithi114 ceph-mon[30820]: pgmap v5208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:44 smithi116 ceph-mon[35411]: pgmap v5208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:46 smithi114 ceph-mon[30820]: pgmap v5209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:46 smithi116 ceph-mon[35411]: pgmap v5209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:25:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: pgmap v5210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:25:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:48 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:25:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: pgmap v5210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:25:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:48 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:25:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-mon[30820]: pgmap v5211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:25:50.485+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:25:50.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:25:50.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:25:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:25:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:50 smithi116 ceph-mon[35411]: pgmap v5211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:25:51.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:25:51.570 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:25:51.570 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:25:51.570 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:25:51.570 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:25:51.570 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:25:51.571 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:25:51.572 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:25:51.572 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:25:51.572 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:25:51.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:25:51.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:25:51.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:51 smithi114 ceph-mon[30820]: pgmap v5212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:25:51.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:51 smithi114 ceph-mon[30820]: pgmap v5213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:25:51.572 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:25:51.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:25:51.580 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:25:51.580 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:25:51.580 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:25:51.580 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:25:51.580 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:25:51.580 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:25:51.580 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:25:51.581 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:25:51.582 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:25:51.582 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:25:51.582 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:25:51.582 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:51 smithi116 ceph-mon[35411]: pgmap v5212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T09:25:51.582 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:51 smithi116 ceph-mon[35411]: pgmap v5213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:25:51.582 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:25:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:53 smithi114 ceph-mon[30820]: pgmap v5214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:25:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:53 smithi116 ceph-mon[35411]: pgmap v5214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:25:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:55 smithi114 ceph-mon[30820]: pgmap v5215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T09:25:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:55 smithi116 ceph-mon[35411]: pgmap v5215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T09:25:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:57 smithi114 ceph-mon[30820]: pgmap v5216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:25:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:57 smithi116 ceph-mon[35411]: pgmap v5216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:25:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:25:59 smithi114 ceph-mon[30820]: pgmap v5217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:25:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:25:59 smithi116 ceph-mon[35411]: pgmap v5217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:26:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:01 smithi114 ceph-mon[30820]: pgmap v5218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:26:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:01 smithi116 ceph-mon[35411]: pgmap v5218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:26:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:03 smithi114 ceph-mon[30820]: pgmap v5219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T09:26:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:03 smithi116 ceph-mon[35411]: pgmap v5219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T09:26:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:05 smithi114 ceph-mon[30820]: pgmap v5220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:26:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:05 smithi116 ceph-mon[35411]: pgmap v5220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:26:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:07 smithi114 ceph-mon[30820]: pgmap v5221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:07 smithi116 ceph-mon[35411]: pgmap v5221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:09 smithi114 ceph-mon[30820]: pgmap v5222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:26:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:09 smithi116 ceph-mon[35411]: pgmap v5222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:26:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:11 smithi114 ceph-mon[30820]: pgmap v5223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:26:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:11 smithi116 ceph-mon[35411]: pgmap v5223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:26:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:13 smithi114 ceph-mon[30820]: pgmap v5224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T09:26:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:13 smithi116 ceph-mon[35411]: pgmap v5224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T09:26:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:15 smithi114 ceph-mon[30820]: pgmap v5225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:26:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:15 smithi116 ceph-mon[35411]: pgmap v5225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:26:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:17 smithi114 ceph-mon[30820]: pgmap v5226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:17 smithi116 ceph-mon[35411]: pgmap v5226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:19 smithi114 ceph-mon[30820]: pgmap v5227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:26:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:19 smithi116 ceph-mon[35411]: pgmap v5227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:26:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:21 smithi114 ceph-mon[30820]: pgmap v5228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:26:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:21 smithi116 ceph-mon[35411]: pgmap v5228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:26:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:23 smithi114 ceph-mon[30820]: pgmap v5229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:26:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:23 smithi116 ceph-mon[35411]: pgmap v5229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:26:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:25 smithi114 ceph-mon[30820]: pgmap v5230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:26:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:25 smithi116 ceph-mon[35411]: pgmap v5230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:26:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:27 smithi114 ceph-mon[30820]: pgmap v5231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:27 smithi116 ceph-mon[35411]: pgmap v5231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:29 smithi114 ceph-mon[30820]: pgmap v5232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:29 smithi116 ceph-mon[35411]: pgmap v5232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:31 smithi114 ceph-mon[30820]: pgmap v5233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:26:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:31 smithi116 ceph-mon[35411]: pgmap v5233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:26:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:33 smithi114 ceph-mon[30820]: pgmap v5234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:26:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:33 smithi116 ceph-mon[35411]: pgmap v5234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:26:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:35 smithi114 ceph-mon[30820]: pgmap v5235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:26:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:35 smithi116 ceph-mon[35411]: pgmap v5235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:26:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:37 smithi114 ceph-mon[30820]: pgmap v5236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:37 smithi116 ceph-mon[35411]: pgmap v5236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:39 smithi114 ceph-mon[30820]: pgmap v5237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:39 smithi116 ceph-mon[35411]: pgmap v5237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:41 smithi114 ceph-mon[30820]: pgmap v5238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:26:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:41 smithi116 ceph-mon[35411]: pgmap v5238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:26:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:43 smithi114 ceph-mon[30820]: pgmap v5239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:43 smithi116 ceph-mon[35411]: pgmap v5239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:45 smithi114 ceph-mon[30820]: pgmap v5240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:26:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:45 smithi116 ceph-mon[35411]: pgmap v5240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:26:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:47 smithi114 ceph-mon[30820]: pgmap v5241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:47 smithi116 ceph-mon[35411]: pgmap v5241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:26:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:49 smithi114 ceph-mon[30820]: pgmap v5242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:49 smithi116 ceph-mon[35411]: pgmap v5242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:26:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:26:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:51 smithi114 ceph-mon[30820]: pgmap v5243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:26:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:51 smithi116 ceph-mon[35411]: pgmap v5243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:26:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-mon[30820]: pgmap v5244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:26:53.845+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:26:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:26:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:26:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:26:53.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:26:53.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:26:53.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:26:53.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:26:53.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:26:53.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:26:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:26:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:53 smithi116 ceph-mon[35411]: pgmap v5244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:26:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:26:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:26:54.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:26:54.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:26:54.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:54 smithi116 ceph-mon[35411]: pgmap v5245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:26:54.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:54 smithi116 ceph-mon[35411]: pgmap v5246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:26:54.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:26:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:26:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:26:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:54 smithi114 ceph-mon[30820]: pgmap v5245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:26:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:54 smithi114 ceph-mon[30820]: pgmap v5246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:26:55.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:26:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:56 smithi114 ceph-mon[30820]: pgmap v5247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:26:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:56 smithi116 ceph-mon[35411]: pgmap v5247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:26:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:26:58 smithi114 ceph-mon[30820]: pgmap v5248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:26:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:26:58 smithi116 ceph-mon[35411]: pgmap v5248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:27:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:00 smithi114 ceph-mon[30820]: pgmap v5249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:27:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:00 smithi116 ceph-mon[35411]: pgmap v5249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:27:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:02 smithi116 ceph-mon[35411]: pgmap v5250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:27:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:02 smithi114 ceph-mon[30820]: pgmap v5250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:27:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:04 smithi116 ceph-mon[35411]: pgmap v5251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:27:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:04 smithi114 ceph-mon[30820]: pgmap v5251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:27:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:06 smithi116 ceph-mon[35411]: pgmap v5252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:06 smithi114 ceph-mon[30820]: pgmap v5252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:08 smithi116 ceph-mon[35411]: pgmap v5253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:08 smithi114 ceph-mon[30820]: pgmap v5253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:10 smithi114 ceph-mon[30820]: pgmap v5254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:10 smithi116 ceph-mon[35411]: pgmap v5254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:12 smithi114 ceph-mon[30820]: pgmap v5255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:12 smithi116 ceph-mon[35411]: pgmap v5255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:14 smithi114 ceph-mon[30820]: pgmap v5256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:14 smithi116 ceph-mon[35411]: pgmap v5256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:16 smithi114 ceph-mon[30820]: pgmap v5257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:16 smithi116 ceph-mon[35411]: pgmap v5257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:18 smithi114 ceph-mon[30820]: pgmap v5258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:18 smithi116 ceph-mon[35411]: pgmap v5258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:21.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:20 smithi114 ceph-mon[30820]: pgmap v5259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:21.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:20 smithi116 ceph-mon[35411]: pgmap v5259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:22 smithi114 ceph-mon[30820]: pgmap v5260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:22 smithi116 ceph-mon[35411]: pgmap v5260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:24 smithi114 ceph-mon[30820]: pgmap v5261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:24 smithi116 ceph-mon[35411]: pgmap v5261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:26 smithi114 ceph-mon[30820]: pgmap v5262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:26 smithi116 ceph-mon[35411]: pgmap v5262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:28 smithi114 ceph-mon[30820]: pgmap v5263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:28 smithi116 ceph-mon[35411]: pgmap v5263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:30 smithi114 ceph-mon[30820]: pgmap v5264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:30 smithi116 ceph-mon[35411]: pgmap v5264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:32 smithi114 ceph-mon[30820]: pgmap v5265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:32 smithi116 ceph-mon[35411]: pgmap v5265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:34 smithi114 ceph-mon[30820]: pgmap v5266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:34 smithi116 ceph-mon[35411]: pgmap v5266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:36 smithi114 ceph-mon[30820]: pgmap v5267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:36 smithi116 ceph-mon[35411]: pgmap v5267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:38 smithi114 ceph-mon[30820]: pgmap v5268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:38 smithi116 ceph-mon[35411]: pgmap v5268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:40 smithi114 ceph-mon[30820]: pgmap v5269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:40 smithi116 ceph-mon[35411]: pgmap v5269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:42 smithi114 ceph-mon[30820]: pgmap v5270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:42 smithi116 ceph-mon[35411]: pgmap v5270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:44 smithi114 ceph-mon[30820]: pgmap v5271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:44 smithi116 ceph-mon[35411]: pgmap v5271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:46 smithi114 ceph-mon[30820]: pgmap v5272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:46 smithi116 ceph-mon[35411]: pgmap v5272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:49 smithi114 ceph-mon[30820]: pgmap v5273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:49 smithi116 ceph-mon[35411]: pgmap v5273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:51 smithi114 ceph-mon[30820]: pgmap v5274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:51 smithi116 ceph-mon[35411]: pgmap v5274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:27:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:53 smithi114 ceph-mon[30820]: pgmap v5275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:53 smithi116 ceph-mon[35411]: pgmap v5275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:27:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:27:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:27:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:55 smithi114 ceph-mon[30820]: pgmap v5276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:55 smithi116 ceph-mon[35411]: pgmap v5276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:27:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-mon[30820]: pgmap v5277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:57.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:27:57.205+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:27:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:27:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:27:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:27:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:27:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:27:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:27:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:27:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:27:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:27:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:27:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:27:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:27:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:27:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:27:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:27:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:27:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:27:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:27:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:27:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:27:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:27:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:27:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:27:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:57 smithi116 ceph-mon[35411]: pgmap v5277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:27:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:27:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:27:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:27:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:27:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:27:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:59 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:27:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:27:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:59 smithi114 ceph-mon[30820]: pgmap v5278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:27:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:27:59 smithi114 ceph-mon[30820]: pgmap v5279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:27:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:59 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:27:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:27:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:59 smithi116 ceph-mon[35411]: pgmap v5278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:27:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:27:59 smithi116 ceph-mon[35411]: pgmap v5279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:28:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:01 smithi114 ceph-mon[30820]: pgmap v5280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:28:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:01 smithi116 ceph-mon[35411]: pgmap v5280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:28:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:03 smithi114 ceph-mon[30820]: pgmap v5281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:28:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:03 smithi116 ceph-mon[35411]: pgmap v5281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:28:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:05 smithi114 ceph-mon[30820]: pgmap v5282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:28:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:05 smithi116 ceph-mon[35411]: pgmap v5282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:28:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:07 smithi114 ceph-mon[30820]: pgmap v5283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:28:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:07 smithi116 ceph-mon[35411]: pgmap v5283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:28:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:09 smithi114 ceph-mon[30820]: pgmap v5284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:28:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:09 smithi116 ceph-mon[35411]: pgmap v5284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:28:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:11 smithi114 ceph-mon[30820]: pgmap v5285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:11 smithi116 ceph-mon[35411]: pgmap v5285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:13 smithi114 ceph-mon[30820]: pgmap v5286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:13 smithi116 ceph-mon[35411]: pgmap v5286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:15 smithi114 ceph-mon[30820]: pgmap v5287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:15 smithi116 ceph-mon[35411]: pgmap v5287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:17 smithi114 ceph-mon[30820]: pgmap v5288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:17 smithi116 ceph-mon[35411]: pgmap v5288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:19 smithi114 ceph-mon[30820]: pgmap v5289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:19 smithi116 ceph-mon[35411]: pgmap v5289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:21 smithi114 ceph-mon[30820]: pgmap v5290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:21 smithi116 ceph-mon[35411]: pgmap v5290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:23 smithi114 ceph-mon[30820]: pgmap v5291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:23 smithi116 ceph-mon[35411]: pgmap v5291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:25 smithi114 ceph-mon[30820]: pgmap v5292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:25 smithi116 ceph-mon[35411]: pgmap v5292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:27 smithi114 ceph-mon[30820]: pgmap v5293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:27 smithi116 ceph-mon[35411]: pgmap v5293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:29 smithi114 ceph-mon[30820]: pgmap v5294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:29 smithi116 ceph-mon[35411]: pgmap v5294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:31 smithi114 ceph-mon[30820]: pgmap v5295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:31 smithi116 ceph-mon[35411]: pgmap v5295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:33 smithi114 ceph-mon[30820]: pgmap v5296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:33 smithi116 ceph-mon[35411]: pgmap v5296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:35 smithi114 ceph-mon[30820]: pgmap v5297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:35 smithi116 ceph-mon[35411]: pgmap v5297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:37 smithi114 ceph-mon[30820]: pgmap v5298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:37 smithi116 ceph-mon[35411]: pgmap v5298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:39 smithi114 ceph-mon[30820]: pgmap v5299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:39 smithi116 ceph-mon[35411]: pgmap v5299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:41 smithi114 ceph-mon[30820]: pgmap v5300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:41 smithi116 ceph-mon[35411]: pgmap v5300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:43 smithi114 ceph-mon[30820]: pgmap v5301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:43 smithi116 ceph-mon[35411]: pgmap v5301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:45 smithi114 ceph-mon[30820]: pgmap v5302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:45 smithi116 ceph-mon[35411]: pgmap v5302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:46 smithi116 ceph-mon[35411]: pgmap v5303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:46 smithi114 ceph-mon[30820]: pgmap v5303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:48 smithi114 ceph-mon[30820]: pgmap v5304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:48 smithi116 ceph-mon[35411]: pgmap v5304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:50 smithi114 ceph-mon[30820]: pgmap v5305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:50 smithi116 ceph-mon[35411]: pgmap v5305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:52 smithi114 ceph-mon[30820]: pgmap v5306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:52 smithi116 ceph-mon[35411]: pgmap v5306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:28:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:54 smithi114 ceph-mon[30820]: pgmap v5307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:54 smithi116 ceph-mon[35411]: pgmap v5307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:28:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:56 smithi114 ceph-mon[30820]: pgmap v5308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:56 smithi116 ceph-mon[35411]: pgmap v5308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:28:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:58 smithi114 ceph-mon[30820]: pgmap v5309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:28:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:58 smithi116 ceph-mon[35411]: pgmap v5309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:28:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:29:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:28:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:28:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:00.892 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:29:00.564+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:29:00.892 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:29:00.893 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:29:00.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:29:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-mon[30820]: pgmap v5310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:29:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:29:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:00 smithi116 ceph-mon[35411]: pgmap v5310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:29:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:29:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:29:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:01 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:29:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:29:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:29:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:29:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:29:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:01 smithi114 ceph-mon[30820]: pgmap v5311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T09:29:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:01 smithi114 ceph-mon[30820]: pgmap v5312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:29:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:01 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:29:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:29:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:29:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:29:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:29:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:29:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:29:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:29:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:29:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:29:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:29:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:29:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:29:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:01 smithi116 ceph-mon[35411]: pgmap v5311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T09:29:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:01 smithi116 ceph-mon[35411]: pgmap v5312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:29:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:03 smithi114 ceph-mon[30820]: pgmap v5313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:29:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:03 smithi116 ceph-mon[35411]: pgmap v5313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:29:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:05 smithi114 ceph-mon[30820]: pgmap v5314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:29:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:05 smithi116 ceph-mon[35411]: pgmap v5314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:29:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:07 smithi114 ceph-mon[30820]: pgmap v5315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:29:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:07 smithi116 ceph-mon[35411]: pgmap v5315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:29:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:09 smithi114 ceph-mon[30820]: pgmap v5316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:29:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:09 smithi116 ceph-mon[35411]: pgmap v5316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:29:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:11 smithi114 ceph-mon[30820]: pgmap v5317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:29:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:11 smithi116 ceph-mon[35411]: pgmap v5317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:29:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:13 smithi114 ceph-mon[30820]: pgmap v5318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:13 smithi116 ceph-mon[35411]: pgmap v5318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:15 smithi116 ceph-mon[35411]: pgmap v5319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:29:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:15 smithi114 ceph-mon[30820]: pgmap v5319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:29:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:17 smithi116 ceph-mon[35411]: pgmap v5320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:17 smithi114 ceph-mon[30820]: pgmap v5320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:19 smithi116 ceph-mon[35411]: pgmap v5321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:19 smithi114 ceph-mon[30820]: pgmap v5321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:22.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:21 smithi114 ceph-mon[30820]: pgmap v5322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:29:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:21 smithi116 ceph-mon[35411]: pgmap v5322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:29:24.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:23 smithi116 ceph-mon[35411]: pgmap v5323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:23 smithi114 ceph-mon[30820]: pgmap v5323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:25 smithi114 ceph-mon[30820]: pgmap v5324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:29:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:25 smithi116 ceph-mon[35411]: pgmap v5324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:29:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:27 smithi114 ceph-mon[30820]: pgmap v5325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:27 smithi116 ceph-mon[35411]: pgmap v5325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:29 smithi114 ceph-mon[30820]: pgmap v5326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:29 smithi116 ceph-mon[35411]: pgmap v5326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:31 smithi114 ceph-mon[30820]: pgmap v5327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:29:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:31 smithi116 ceph-mon[35411]: pgmap v5327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:29:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:33 smithi114 ceph-mon[30820]: pgmap v5328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:29:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:33 smithi116 ceph-mon[35411]: pgmap v5328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:29:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:35 smithi114 ceph-mon[30820]: pgmap v5329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:29:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:35 smithi116 ceph-mon[35411]: pgmap v5329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:29:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:37 smithi114 ceph-mon[30820]: pgmap v5330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:37 smithi116 ceph-mon[35411]: pgmap v5330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:39 smithi114 ceph-mon[30820]: pgmap v5331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:29:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:39 smithi116 ceph-mon[35411]: pgmap v5331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:29:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:41 smithi114 ceph-mon[30820]: pgmap v5332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:29:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:41 smithi116 ceph-mon[35411]: pgmap v5332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:29:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:43 smithi114 ceph-mon[30820]: pgmap v5333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:43 smithi116 ceph-mon[35411]: pgmap v5333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:45 smithi114 ceph-mon[30820]: pgmap v5334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:29:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:45 smithi116 ceph-mon[35411]: pgmap v5334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:29:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:47 smithi116 ceph-mon[35411]: pgmap v5335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:47 smithi114 ceph-mon[30820]: pgmap v5335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:49 smithi116 ceph-mon[35411]: pgmap v5336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:49 smithi114 ceph-mon[30820]: pgmap v5336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:51 smithi116 ceph-mon[35411]: pgmap v5337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:29:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:51 smithi114 ceph-mon[30820]: pgmap v5337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:29:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:53 smithi116 ceph-mon[35411]: pgmap v5338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:53.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:53 smithi114 ceph-mon[30820]: pgmap v5338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:29:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:55 smithi114 ceph-mon[30820]: pgmap v5339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:29:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:55 smithi116 ceph-mon[35411]: pgmap v5339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:29:58.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:57 smithi114 ceph-mon[30820]: pgmap v5340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:29:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:57 smithi116 ceph-mon[35411]: pgmap v5340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:29:59 smithi114 ceph-mon[30820]: pgmap v5341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:29:59 smithi116 ceph-mon[35411]: pgmap v5341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:00.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:30:00.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:30:01.088 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:30:01.088 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:30:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:02 smithi114 ceph-mon[30820]: pgmap v5342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:02 smithi116 ceph-mon[35411]: pgmap v5342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:04 smithi114 ceph-mon[30820]: pgmap v5343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:30:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:30:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:04.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:30:03.861+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:30:04.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:30:04.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:30:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:30:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:30:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:30:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:30:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:30:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:30:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:30:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:30:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:30:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:30:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:04 smithi116 ceph-mon[35411]: pgmap v5343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:30:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:30:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:30:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:05 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:30:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:30:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:30:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:05 smithi114 ceph-mon[30820]: pgmap v5344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:30:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:05 smithi114 ceph-mon[30820]: pgmap v5345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:05 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:30:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:05 smithi116 ceph-mon[35411]: pgmap v5344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:30:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:05 smithi116 ceph-mon[35411]: pgmap v5345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:30:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:07 smithi114 ceph-mon[30820]: pgmap v5346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:30:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:07 smithi116 ceph-mon[35411]: pgmap v5346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:30:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:09 smithi114 ceph-mon[30820]: pgmap v5347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:30:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:09 smithi116 ceph-mon[35411]: pgmap v5347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:30:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:11 smithi114 ceph-mon[30820]: pgmap v5348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:30:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:11 smithi116 ceph-mon[35411]: pgmap v5348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:30:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:13 smithi114 ceph-mon[30820]: pgmap v5349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:30:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:13 smithi116 ceph-mon[35411]: pgmap v5349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:30:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:15 smithi114 ceph-mon[30820]: pgmap v5350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:30:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:15 smithi116 ceph-mon[35411]: pgmap v5350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:30:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:17 smithi114 ceph-mon[30820]: pgmap v5351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:17 smithi116 ceph-mon[35411]: pgmap v5351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:19 smithi114 ceph-mon[30820]: pgmap v5352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:19 smithi116 ceph-mon[35411]: pgmap v5352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:21 smithi114 ceph-mon[30820]: pgmap v5353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:30:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:21 smithi116 ceph-mon[35411]: pgmap v5353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:30:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:23 smithi114 ceph-mon[30820]: pgmap v5354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:23 smithi116 ceph-mon[35411]: pgmap v5354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:25 smithi114 ceph-mon[30820]: pgmap v5355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:25 smithi116 ceph-mon[35411]: pgmap v5355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:27 smithi114 ceph-mon[30820]: pgmap v5356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:27 smithi116 ceph-mon[35411]: pgmap v5356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:29 smithi114 ceph-mon[30820]: pgmap v5357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:29 smithi116 ceph-mon[35411]: pgmap v5357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:31 smithi114 ceph-mon[30820]: pgmap v5358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:30:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:31 smithi116 ceph-mon[35411]: pgmap v5358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:30:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:33 smithi114 ceph-mon[30820]: pgmap v5359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:33 smithi116 ceph-mon[35411]: pgmap v5359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:35 smithi114 ceph-mon[30820]: pgmap v5360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:35 smithi116 ceph-mon[35411]: pgmap v5360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:37 smithi114 ceph-mon[30820]: pgmap v5361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:37 smithi116 ceph-mon[35411]: pgmap v5361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:39 smithi114 ceph-mon[30820]: pgmap v5362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:39 smithi116 ceph-mon[35411]: pgmap v5362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:41 smithi114 ceph-mon[30820]: pgmap v5363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:30:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:41 smithi116 ceph-mon[35411]: pgmap v5363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:30:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:43 smithi114 ceph-mon[30820]: pgmap v5364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:43 smithi116 ceph-mon[35411]: pgmap v5364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:45 smithi114 ceph-mon[30820]: pgmap v5365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:45 smithi116 ceph-mon[35411]: pgmap v5365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:47 smithi114 ceph-mon[30820]: pgmap v5366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:47 smithi116 ceph-mon[35411]: pgmap v5366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:49 smithi114 ceph-mon[30820]: pgmap v5367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:49 smithi116 ceph-mon[35411]: pgmap v5367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:51 smithi114 ceph-mon[30820]: pgmap v5368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:30:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:51 smithi116 ceph-mon[35411]: pgmap v5368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:30:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:53 smithi114 ceph-mon[30820]: pgmap v5369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:53 smithi116 ceph-mon[35411]: pgmap v5369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:30:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:55 smithi114 ceph-mon[30820]: pgmap v5370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:55 smithi116 ceph-mon[35411]: pgmap v5370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:57 smithi114 ceph-mon[30820]: pgmap v5371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:57 smithi116 ceph-mon[35411]: pgmap v5371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:30:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:30:59 smithi114 ceph-mon[30820]: pgmap v5372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:30:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:30:59 smithi116 ceph-mon[35411]: pgmap v5372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:01 smithi114 ceph-mon[30820]: pgmap v5373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:31:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:01 smithi116 ceph-mon[35411]: pgmap v5373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:31:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:03 smithi114 ceph-mon[30820]: pgmap v5374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:31:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:03 smithi116 ceph-mon[35411]: pgmap v5374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:31:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:31:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:31:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:05 smithi114 ceph-mon[30820]: pgmap v5375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:05 smithi116 ceph-mon[35411]: pgmap v5375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-mon[30820]: pgmap v5376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:31:07.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:31:07.220+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:31:07.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:31:07.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:31:07.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:31:07.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:31:07.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:31:07.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:31:07.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:31:07.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:31:07.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:31:07.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:31:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:31:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:07 smithi116 ceph-mon[35411]: pgmap v5376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:31:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:31:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:31:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:31:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:31:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:31:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:31:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:09 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:31:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:31:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:31:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:31:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:31:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:31:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:31:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:31:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:09 smithi114 ceph-mon[30820]: pgmap v5377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:31:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:09 smithi114 ceph-mon[30820]: pgmap v5378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:31:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:09 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:31:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:31:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:09 smithi116 ceph-mon[35411]: pgmap v5377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:31:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:09 smithi116 ceph-mon[35411]: pgmap v5378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:31:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:11 smithi114 ceph-mon[30820]: pgmap v5379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:31:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:11 smithi116 ceph-mon[35411]: pgmap v5379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:31:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:12 smithi114 ceph-mon[30820]: pgmap v5380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:31:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:12 smithi116 ceph-mon[35411]: pgmap v5380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:31:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:14 smithi114 ceph-mon[30820]: pgmap v5381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:31:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:14 smithi116 ceph-mon[35411]: pgmap v5381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:31:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:16 smithi114 ceph-mon[30820]: pgmap v5382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:31:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:16 smithi116 ceph-mon[35411]: pgmap v5382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:31:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:18 smithi114 ceph-mon[30820]: pgmap v5383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:31:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:18 smithi116 ceph-mon[35411]: pgmap v5383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:31:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:20 smithi114 ceph-mon[30820]: pgmap v5384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:20 smithi116 ceph-mon[35411]: pgmap v5384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:22 smithi114 ceph-mon[30820]: pgmap v5385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:31:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:22 smithi116 ceph-mon[35411]: pgmap v5385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:31:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:24 smithi114 ceph-mon[30820]: pgmap v5386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:24 smithi116 ceph-mon[35411]: pgmap v5386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:26 smithi114 ceph-mon[30820]: pgmap v5387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:31:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:26 smithi116 ceph-mon[35411]: pgmap v5387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:31:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:28 smithi114 ceph-mon[30820]: pgmap v5388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:31:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:28 smithi116 ceph-mon[35411]: pgmap v5388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:31:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:30 smithi114 ceph-mon[30820]: pgmap v5389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:31:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:30 smithi116 ceph-mon[35411]: pgmap v5389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:31:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:32 smithi114 ceph-mon[30820]: pgmap v5390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:31:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:32 smithi116 ceph-mon[35411]: pgmap v5390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:31:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:34 smithi114 ceph-mon[30820]: pgmap v5391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:34 smithi116 ceph-mon[35411]: pgmap v5391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:36 smithi114 ceph-mon[30820]: pgmap v5392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:31:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:36 smithi116 ceph-mon[35411]: pgmap v5392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:31:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:38 smithi114 ceph-mon[30820]: pgmap v5393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:31:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:38 smithi116 ceph-mon[35411]: pgmap v5393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:31:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:40 smithi114 ceph-mon[30820]: pgmap v5394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:31:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:40 smithi116 ceph-mon[35411]: pgmap v5394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:31:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:42 smithi114 ceph-mon[30820]: pgmap v5395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:31:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:42 smithi116 ceph-mon[35411]: pgmap v5395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:31:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:44 smithi114 ceph-mon[30820]: pgmap v5396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:44 smithi116 ceph-mon[35411]: pgmap v5396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:46 smithi114 ceph-mon[30820]: pgmap v5397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:31:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:46 smithi116 ceph-mon[35411]: pgmap v5397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:31:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:48 smithi114 ceph-mon[30820]: pgmap v5398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:31:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:48 smithi116 ceph-mon[35411]: pgmap v5398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:31:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:50 smithi114 ceph-mon[30820]: pgmap v5399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:31:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:50 smithi116 ceph-mon[35411]: pgmap v5399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:31:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:52 smithi114 ceph-mon[30820]: pgmap v5400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:31:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:52 smithi116 ceph-mon[35411]: pgmap v5400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:31:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:54 smithi114 ceph-mon[30820]: pgmap v5401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:54 smithi116 ceph-mon[35411]: pgmap v5401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:31:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:56 smithi114 ceph-mon[30820]: pgmap v5402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:31:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:56 smithi116 ceph-mon[35411]: pgmap v5402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:31:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:31:58 smithi114 ceph-mon[30820]: pgmap v5403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:31:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:31:58 smithi116 ceph-mon[35411]: pgmap v5403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:32:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:00 smithi114 ceph-mon[30820]: pgmap v5404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:00 smithi116 ceph-mon[35411]: pgmap v5404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:02 smithi114 ceph-mon[30820]: pgmap v5405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:02 smithi116 ceph-mon[35411]: pgmap v5405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:04 smithi114 ceph-mon[30820]: pgmap v5406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:04 smithi116 ceph-mon[35411]: pgmap v5406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:06 smithi114 ceph-mon[30820]: pgmap v5407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:32:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:06 smithi116 ceph-mon[35411]: pgmap v5407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:32:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:32:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:32:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:08 smithi114 ceph-mon[30820]: pgmap v5408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:32:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:08 smithi116 ceph-mon[35411]: pgmap v5408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:32:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-mon[30820]: pgmap v5409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:32:10.580+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:32:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:32:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:32:10.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:32:10.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:32:10.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:32:10.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:32:10.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:32:10.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:32:10.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:32:10.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:32:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:32:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:10 smithi116 ceph-mon[35411]: pgmap v5409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:11 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:32:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:32:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:11 smithi114 ceph-mon[30820]: pgmap v5410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:32:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:11 smithi114 ceph-mon[30820]: pgmap v5411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:32:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:32:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:11 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:32:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:32:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:11 smithi116 ceph-mon[35411]: pgmap v5410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:32:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:11 smithi116 ceph-mon[35411]: pgmap v5411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:32:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:32:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:13 smithi114 ceph-mon[30820]: pgmap v5412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:32:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:13 smithi116 ceph-mon[35411]: pgmap v5412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:32:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:15 smithi114 ceph-mon[30820]: pgmap v5413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:32:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:15 smithi116 ceph-mon[35411]: pgmap v5413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:32:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:17 smithi114 ceph-mon[30820]: pgmap v5414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:32:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:17 smithi116 ceph-mon[35411]: pgmap v5414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:32:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:19 smithi114 ceph-mon[30820]: pgmap v5415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:32:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:19 smithi116 ceph-mon[35411]: pgmap v5415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:32:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:21 smithi114 ceph-mon[30820]: pgmap v5416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:32:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:21 smithi116 ceph-mon[35411]: pgmap v5416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:32:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:23 smithi114 ceph-mon[30820]: pgmap v5417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:23 smithi116 ceph-mon[35411]: pgmap v5417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:25 smithi114 ceph-mon[30820]: pgmap v5418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:25 smithi116 ceph-mon[35411]: pgmap v5418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:27 smithi116 ceph-mon[35411]: pgmap v5419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:27 smithi114 ceph-mon[30820]: pgmap v5419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:29 smithi116 ceph-mon[35411]: pgmap v5420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:29 smithi114 ceph-mon[30820]: pgmap v5420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:31 smithi116 ceph-mon[35411]: pgmap v5421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:32:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:31 smithi114 ceph-mon[30820]: pgmap v5421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:32:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:33 smithi114 ceph-mon[30820]: pgmap v5422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:33 smithi116 ceph-mon[35411]: pgmap v5422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:35 smithi114 ceph-mon[30820]: pgmap v5423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:35 smithi116 ceph-mon[35411]: pgmap v5423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:37 smithi114 ceph-mon[30820]: pgmap v5424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:37 smithi116 ceph-mon[35411]: pgmap v5424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:39 smithi114 ceph-mon[30820]: pgmap v5425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:39 smithi116 ceph-mon[35411]: pgmap v5425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:41 smithi114 ceph-mon[30820]: pgmap v5426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:32:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:41 smithi116 ceph-mon[35411]: pgmap v5426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:32:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:43 smithi114 ceph-mon[30820]: pgmap v5427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:43 smithi116 ceph-mon[35411]: pgmap v5427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:45 smithi114 ceph-mon[30820]: pgmap v5428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:45 smithi116 ceph-mon[35411]: pgmap v5428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:47 smithi114 ceph-mon[30820]: pgmap v5429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:47 smithi116 ceph-mon[35411]: pgmap v5429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:49 smithi114 ceph-mon[30820]: pgmap v5430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:49 smithi116 ceph-mon[35411]: pgmap v5430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:51 smithi114 ceph-mon[30820]: pgmap v5431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:32:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:51 smithi116 ceph-mon[35411]: pgmap v5431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:32:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:53 smithi114 ceph-mon[30820]: pgmap v5432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:53 smithi116 ceph-mon[35411]: pgmap v5432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:32:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:55 smithi114 ceph-mon[30820]: pgmap v5433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:55 smithi116 ceph-mon[35411]: pgmap v5433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:32:57.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:57 smithi116 ceph-mon[35411]: pgmap v5434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:32:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:57 smithi114 ceph-mon[30820]: pgmap v5434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:32:59 smithi114 ceph-mon[30820]: pgmap v5435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:32:59 smithi116 ceph-mon[35411]: pgmap v5435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:01 smithi114 ceph-mon[30820]: pgmap v5436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:01 smithi116 ceph-mon[35411]: pgmap v5436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:03 smithi114 ceph-mon[30820]: pgmap v5437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:03 smithi116 ceph-mon[35411]: pgmap v5437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:05 smithi114 ceph-mon[30820]: pgmap v5438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:33:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:05 smithi116 ceph-mon[35411]: pgmap v5438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:33:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:07 smithi114 ceph-mon[30820]: pgmap v5439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:07 smithi116 ceph-mon[35411]: pgmap v5439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:09 smithi114 ceph-mon[30820]: pgmap v5440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:09 smithi116 ceph-mon[35411]: pgmap v5440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:33:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:33:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:11 smithi114 ceph-mon[30820]: pgmap v5441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:11 smithi116 ceph-mon[35411]: pgmap v5441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-mon[30820]: pgmap v5442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:33:13.941+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:33:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:33:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:33:14.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:33:14.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:33:14.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:33:14.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:33:14.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:33:14.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:33:14.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:33:14.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:33:14.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:33:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:33:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:13 smithi116 ceph-mon[35411]: pgmap v5442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:33:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:33:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:33:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:33:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:33:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:33:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:14 smithi114 ceph-mon[30820]: pgmap v5443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:33:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:14 smithi114 ceph-mon[30820]: pgmap v5444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:33:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:33:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:33:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:14 smithi116 ceph-mon[35411]: pgmap v5443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:14 smithi116 ceph-mon[35411]: pgmap v5444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:33:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:33:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:17 smithi114 ceph-mon[30820]: pgmap v5445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:33:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:17 smithi116 ceph-mon[35411]: pgmap v5445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:33:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:19 smithi114 ceph-mon[30820]: pgmap v5446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:33:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:19 smithi116 ceph-mon[35411]: pgmap v5446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:33:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:21 smithi114 ceph-mon[30820]: pgmap v5447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:33:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:21 smithi116 ceph-mon[35411]: pgmap v5447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:33:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:23 smithi114 ceph-mon[30820]: pgmap v5448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:33:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:23 smithi116 ceph-mon[35411]: pgmap v5448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:33:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:25 smithi114 ceph-mon[30820]: pgmap v5449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:33:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:25 smithi116 ceph-mon[35411]: pgmap v5449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:33:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:27 smithi114 ceph-mon[30820]: pgmap v5450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:27 smithi116 ceph-mon[35411]: pgmap v5450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:29 smithi114 ceph-mon[30820]: pgmap v5451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:29 smithi116 ceph-mon[35411]: pgmap v5451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:31 smithi114 ceph-mon[30820]: pgmap v5452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:33:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:31 smithi116 ceph-mon[35411]: pgmap v5452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:33:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:33 smithi114 ceph-mon[30820]: pgmap v5453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:33 smithi116 ceph-mon[35411]: pgmap v5453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:35 smithi114 ceph-mon[30820]: pgmap v5454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:35 smithi116 ceph-mon[35411]: pgmap v5454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:37 smithi114 ceph-mon[30820]: pgmap v5455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:37 smithi116 ceph-mon[35411]: pgmap v5455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:39 smithi114 ceph-mon[30820]: pgmap v5456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:39 smithi116 ceph-mon[35411]: pgmap v5456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:41 smithi114 ceph-mon[30820]: pgmap v5457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:33:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:41 smithi116 ceph-mon[35411]: pgmap v5457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:33:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:43 smithi114 ceph-mon[30820]: pgmap v5458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:43 smithi116 ceph-mon[35411]: pgmap v5458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:45 smithi114 ceph-mon[30820]: pgmap v5459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:45 smithi116 ceph-mon[35411]: pgmap v5459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:47 smithi114 ceph-mon[30820]: pgmap v5460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:47 smithi116 ceph-mon[35411]: pgmap v5460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:49 smithi114 ceph-mon[30820]: pgmap v5461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:49 smithi116 ceph-mon[35411]: pgmap v5461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:51 smithi114 ceph-mon[30820]: pgmap v5462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:33:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:51 smithi116 ceph-mon[35411]: pgmap v5462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:33:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:53 smithi114 ceph-mon[30820]: pgmap v5463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:53 smithi116 ceph-mon[35411]: pgmap v5463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:33:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:55 smithi114 ceph-mon[30820]: pgmap v5464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:55 smithi116 ceph-mon[35411]: pgmap v5464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:57 smithi114 ceph-mon[30820]: pgmap v5465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:57 smithi116 ceph-mon[35411]: pgmap v5465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:33:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:33:59 smithi114 ceph-mon[30820]: pgmap v5466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:33:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:33:59 smithi116 ceph-mon[35411]: pgmap v5466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:01 smithi114 ceph-mon[30820]: pgmap v5467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:01 smithi116 ceph-mon[35411]: pgmap v5467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:03 smithi114 ceph-mon[30820]: pgmap v5468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:03 smithi116 ceph-mon[35411]: pgmap v5468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:05 smithi114 ceph-mon[30820]: pgmap v5469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:05 smithi116 ceph-mon[35411]: pgmap v5469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:07 smithi114 ceph-mon[30820]: pgmap v5470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:34:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:07 smithi116 ceph-mon[35411]: pgmap v5470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:34:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:09 smithi114 ceph-mon[30820]: pgmap v5471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:09 smithi116 ceph-mon[35411]: pgmap v5471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:11 smithi114 ceph-mon[30820]: pgmap v5472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:11 smithi116 ceph-mon[35411]: pgmap v5472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:13 smithi114 ceph-mon[30820]: pgmap v5473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:13 smithi116 ceph-mon[35411]: pgmap v5473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:34:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:34:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:15 smithi114 ceph-mon[30820]: pgmap v5474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:15 smithi116 ceph-mon[35411]: pgmap v5474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-mon[30820]: pgmap v5475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:34:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:34:17.300+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:34:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:34:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:34:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:34:17.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:34:17.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:34:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:34:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:34:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:34:17.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:34:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:34:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:17 smithi116 ceph-mon[35411]: pgmap v5475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:34:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:34:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:34:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:34:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:34:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:34:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:34:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:34:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:34:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:34:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:34:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:34:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:34:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:34:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:34:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:34:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:34:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:19 smithi114 ceph-mon[30820]: pgmap v5476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:34:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:19 smithi114 ceph-mon[30820]: pgmap v5477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:34:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:34:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:34:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:34:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:34:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:19 smithi116 ceph-mon[35411]: pgmap v5476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:34:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:19 smithi116 ceph-mon[35411]: pgmap v5477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:34:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:21 smithi114 ceph-mon[30820]: pgmap v5478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:34:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:21 smithi116 ceph-mon[35411]: pgmap v5478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:34:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:23 smithi114 ceph-mon[30820]: pgmap v5479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:34:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:23 smithi116 ceph-mon[35411]: pgmap v5479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:34:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:25 smithi114 ceph-mon[30820]: pgmap v5480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:34:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:25 smithi116 ceph-mon[35411]: pgmap v5480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:34:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:27 smithi114 ceph-mon[30820]: pgmap v5481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:34:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:27 smithi116 ceph-mon[35411]: pgmap v5481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:34:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:29 smithi114 ceph-mon[30820]: pgmap v5482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:34:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:29 smithi116 ceph-mon[35411]: pgmap v5482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:34:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:31 smithi114 ceph-mon[30820]: pgmap v5483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:31 smithi116 ceph-mon[35411]: pgmap v5483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:33 smithi114 ceph-mon[30820]: pgmap v5484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:33 smithi116 ceph-mon[35411]: pgmap v5484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:35 smithi114 ceph-mon[30820]: pgmap v5485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:35 smithi116 ceph-mon[35411]: pgmap v5485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:37 smithi114 ceph-mon[30820]: pgmap v5486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:34:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:37 smithi116 ceph-mon[35411]: pgmap v5486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:34:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:39 smithi114 ceph-mon[30820]: pgmap v5487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:34:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:39 smithi116 ceph-mon[35411]: pgmap v5487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:34:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:41 smithi114 ceph-mon[30820]: pgmap v5488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:41 smithi116 ceph-mon[35411]: pgmap v5488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:43 smithi116 ceph-mon[35411]: pgmap v5489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:43 smithi114 ceph-mon[30820]: pgmap v5489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:45 smithi116 ceph-mon[35411]: pgmap v5490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:45 smithi114 ceph-mon[30820]: pgmap v5490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:47 smithi116 ceph-mon[35411]: pgmap v5491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:34:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:47 smithi114 ceph-mon[30820]: pgmap v5491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:34:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:49 smithi114 ceph-mon[30820]: pgmap v5492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:34:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:49 smithi116 ceph-mon[35411]: pgmap v5492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:34:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:51 smithi114 ceph-mon[30820]: pgmap v5493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:51 smithi116 ceph-mon[35411]: pgmap v5493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:34:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:53 smithi114 ceph-mon[30820]: pgmap v5494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:53 smithi116 ceph-mon[35411]: pgmap v5494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:34:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:54 smithi114 ceph-mon[30820]: pgmap v5495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:54 smithi116 ceph-mon[35411]: pgmap v5495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:34:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:56 smithi114 ceph-mon[30820]: pgmap v5496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:34:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:56 smithi116 ceph-mon[35411]: pgmap v5496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:34:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:34:58 smithi114 ceph-mon[30820]: pgmap v5497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:34:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:34:58 smithi116 ceph-mon[35411]: pgmap v5497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:35:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:00 smithi114 ceph-mon[30820]: pgmap v5498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:00 smithi116 ceph-mon[35411]: pgmap v5498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:02 smithi114 ceph-mon[30820]: pgmap v5499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:02 smithi116 ceph-mon[35411]: pgmap v5499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:04 smithi114 ceph-mon[30820]: pgmap v5500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:04 smithi116 ceph-mon[35411]: pgmap v5500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:06 smithi114 ceph-mon[30820]: pgmap v5501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:35:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:06 smithi116 ceph-mon[35411]: pgmap v5501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:35:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:08 smithi114 ceph-mon[30820]: pgmap v5502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:35:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:08 smithi116 ceph-mon[35411]: pgmap v5502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:35:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:10 smithi114 ceph-mon[30820]: pgmap v5503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:10 smithi116 ceph-mon[35411]: pgmap v5503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:12 smithi114 ceph-mon[30820]: pgmap v5504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:12 smithi116 ceph-mon[35411]: pgmap v5504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:14.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:14 smithi114 ceph-mon[30820]: pgmap v5505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:14 smithi116 ceph-mon[35411]: pgmap v5505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:16 smithi114 ceph-mon[30820]: pgmap v5506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:35:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:16 smithi116 ceph-mon[35411]: pgmap v5506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:35:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:18 smithi114 ceph-mon[30820]: pgmap v5507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:35:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:35:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:18 smithi116 ceph-mon[35411]: pgmap v5507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:35:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:35:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:20 smithi116 ceph-mon[35411]: pgmap v5508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-mon[30820]: pgmap v5508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:20.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:35:20.660+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:35:20.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:35:20.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:35:20.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:35:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:35:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:35:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:35:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:35:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:35:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:35:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:21 smithi116 ceph-mon[35411]: pgmap v5509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:35:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:21 smithi116 ceph-mon[35411]: pgmap v5510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:35:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:35:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:35:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:35:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:35:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:35:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:35:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:35:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:35:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:35:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:35:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:35:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:35:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:35:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:35:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:21 smithi114 ceph-mon[30820]: pgmap v5509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:35:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:21 smithi114 ceph-mon[30820]: pgmap v5510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:35:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:35:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:23 smithi114 ceph-mon[30820]: pgmap v5511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:35:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:23 smithi116 ceph-mon[35411]: pgmap v5511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:35:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:25 smithi114 ceph-mon[30820]: pgmap v5512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:35:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:25 smithi116 ceph-mon[35411]: pgmap v5512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:35:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:27 smithi114 ceph-mon[30820]: pgmap v5513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:35:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:27 smithi116 ceph-mon[35411]: pgmap v5513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:35:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:29 smithi114 ceph-mon[30820]: pgmap v5514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:35:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:29 smithi116 ceph-mon[35411]: pgmap v5514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:35:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:31 smithi114 ceph-mon[30820]: pgmap v5515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:35:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:31 smithi116 ceph-mon[35411]: pgmap v5515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:35:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:33 smithi114 ceph-mon[30820]: pgmap v5516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:33 smithi116 ceph-mon[35411]: pgmap v5516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:35 smithi114 ceph-mon[30820]: pgmap v5517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:35 smithi116 ceph-mon[35411]: pgmap v5517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:37 smithi114 ceph-mon[30820]: pgmap v5518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:37 smithi116 ceph-mon[35411]: pgmap v5518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:39 smithi114 ceph-mon[30820]: pgmap v5519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:39 smithi116 ceph-mon[35411]: pgmap v5519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:41 smithi114 ceph-mon[30820]: pgmap v5520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:35:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:41 smithi116 ceph-mon[35411]: pgmap v5520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:35:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:43 smithi114 ceph-mon[30820]: pgmap v5521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:43 smithi116 ceph-mon[35411]: pgmap v5521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:45 smithi114 ceph-mon[30820]: pgmap v5522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:45 smithi116 ceph-mon[35411]: pgmap v5522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:47 smithi114 ceph-mon[30820]: pgmap v5523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:47 smithi116 ceph-mon[35411]: pgmap v5523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:49 smithi114 ceph-mon[30820]: pgmap v5524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:49 smithi116 ceph-mon[35411]: pgmap v5524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:51 smithi114 ceph-mon[30820]: pgmap v5525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:35:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:51 smithi116 ceph-mon[35411]: pgmap v5525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:35:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:53 smithi114 ceph-mon[30820]: pgmap v5526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:53 smithi116 ceph-mon[35411]: pgmap v5526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:35:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:55 smithi114 ceph-mon[30820]: pgmap v5527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:55 smithi116 ceph-mon[35411]: pgmap v5527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:35:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:57 smithi114 ceph-mon[30820]: pgmap v5528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:35:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:57 smithi116 ceph-mon[35411]: pgmap v5528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:35:59 smithi114 ceph-mon[30820]: pgmap v5529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:35:59 smithi116 ceph-mon[35411]: pgmap v5529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:01 smithi114 ceph-mon[30820]: pgmap v5530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:01 smithi116 ceph-mon[35411]: pgmap v5530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:03 smithi114 ceph-mon[30820]: pgmap v5531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:03 smithi116 ceph-mon[35411]: pgmap v5531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:05 smithi114 ceph-mon[30820]: pgmap v5532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:36:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:05 smithi116 ceph-mon[35411]: pgmap v5532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:36:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:07 smithi114 ceph-mon[30820]: pgmap v5533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:07 smithi116 ceph-mon[35411]: pgmap v5533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:09 smithi114 ceph-mon[30820]: pgmap v5534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:09 smithi116 ceph-mon[35411]: pgmap v5534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:11 smithi114 ceph-mon[30820]: pgmap v5535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:11 smithi116 ceph-mon[35411]: pgmap v5535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:13 smithi114 ceph-mon[30820]: pgmap v5536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:13 smithi116 ceph-mon[35411]: pgmap v5536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:15 smithi114 ceph-mon[30820]: pgmap v5537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:36:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:15 smithi116 ceph-mon[35411]: pgmap v5537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:36:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:17 smithi114 ceph-mon[30820]: pgmap v5538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:17 smithi116 ceph-mon[35411]: pgmap v5538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:19 smithi114 ceph-mon[30820]: pgmap v5539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:19 smithi116 ceph-mon[35411]: pgmap v5539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:36:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: pgmap v5540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: pgmap v5540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:36:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:22 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:36:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:22 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:36:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:22 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:36:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:22 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:36:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:23 smithi114 ceph-mon[30820]: pgmap v5541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:24.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:36:24.021+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:36:24.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:36:24.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:36:24.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:36:24.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:36:24.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:36:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:36:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:36:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:36:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:23 smithi116 ceph-mon[35411]: pgmap v5541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:36:25.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:36:25.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:36:25.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:36:25.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:36:25.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:36:25.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:36:25.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:36:25.173 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:36:25.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:36:25.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:36:25.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-mon[30820]: pgmap v5542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:36:25.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-mon[30820]: pgmap v5543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:36:25.173 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:36:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:36:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:36:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:24 smithi116 ceph-mon[35411]: pgmap v5542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:36:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:24 smithi116 ceph-mon[35411]: pgmap v5543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:36:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:36:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:27 smithi114 ceph-mon[30820]: pgmap v5544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:36:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:27 smithi116 ceph-mon[35411]: pgmap v5544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:36:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:29 smithi114 ceph-mon[30820]: pgmap v5545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:36:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:29 smithi116 ceph-mon[35411]: pgmap v5545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:36:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:31 smithi114 ceph-mon[30820]: pgmap v5546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:36:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:31 smithi116 ceph-mon[35411]: pgmap v5546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:36:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:33 smithi114 ceph-mon[30820]: pgmap v5547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:36:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:33 smithi116 ceph-mon[35411]: pgmap v5547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:36:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:35 smithi114 ceph-mon[30820]: pgmap v5548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:36:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:35 smithi116 ceph-mon[35411]: pgmap v5548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:36:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:37 smithi114 ceph-mon[30820]: pgmap v5549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:37 smithi116 ceph-mon[35411]: pgmap v5549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:39 smithi114 ceph-mon[30820]: pgmap v5550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:39 smithi116 ceph-mon[35411]: pgmap v5550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:41 smithi114 ceph-mon[30820]: pgmap v5551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:36:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:41 smithi116 ceph-mon[35411]: pgmap v5551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:36:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:43 smithi114 ceph-mon[30820]: pgmap v5552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:43 smithi116 ceph-mon[35411]: pgmap v5552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:45 smithi114 ceph-mon[30820]: pgmap v5553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:45 smithi116 ceph-mon[35411]: pgmap v5553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:47 smithi114 ceph-mon[30820]: pgmap v5554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:47 smithi116 ceph-mon[35411]: pgmap v5554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:49 smithi114 ceph-mon[30820]: pgmap v5555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:49 smithi116 ceph-mon[35411]: pgmap v5555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:51 smithi114 ceph-mon[30820]: pgmap v5556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:36:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:51 smithi116 ceph-mon[35411]: pgmap v5556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:36:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:53 smithi114 ceph-mon[30820]: pgmap v5557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:53 smithi116 ceph-mon[35411]: pgmap v5557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:36:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:55 smithi114 ceph-mon[30820]: pgmap v5558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:55 smithi116 ceph-mon[35411]: pgmap v5558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:57 smithi114 ceph-mon[30820]: pgmap v5559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:57 smithi116 ceph-mon[35411]: pgmap v5559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:36:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:36:59 smithi114 ceph-mon[30820]: pgmap v5560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:36:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:36:59 smithi116 ceph-mon[35411]: pgmap v5560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:01 smithi114 ceph-mon[30820]: pgmap v5561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:37:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:01 smithi116 ceph-mon[35411]: pgmap v5561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:37:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:02 smithi114 ceph-mon[30820]: pgmap v5562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:02 smithi116 ceph-mon[35411]: pgmap v5562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:05 smithi114 ceph-mon[30820]: pgmap v5563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:05 smithi116 ceph-mon[35411]: pgmap v5563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:07 smithi114 ceph-mon[30820]: pgmap v5564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:07 smithi116 ceph-mon[35411]: pgmap v5564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:09 smithi114 ceph-mon[30820]: pgmap v5565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:09 smithi116 ceph-mon[35411]: pgmap v5565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:11 smithi114 ceph-mon[30820]: pgmap v5566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:37:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:11 smithi116 ceph-mon[35411]: pgmap v5566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:37:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:13 smithi114 ceph-mon[30820]: pgmap v5567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:13 smithi116 ceph-mon[35411]: pgmap v5567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:15 smithi114 ceph-mon[30820]: pgmap v5568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:15 smithi116 ceph-mon[35411]: pgmap v5568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:17 smithi114 ceph-mon[30820]: pgmap v5569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:17 smithi116 ceph-mon[35411]: pgmap v5569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:19 smithi114 ceph-mon[30820]: pgmap v5570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:19 smithi116 ceph-mon[35411]: pgmap v5570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:21 smithi114 ceph-mon[30820]: pgmap v5571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:37:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:21 smithi116 ceph-mon[35411]: pgmap v5571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:37:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:23 smithi114 ceph-mon[30820]: pgmap v5572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:23 smithi116 ceph-mon[35411]: pgmap v5572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:25 smithi114 ceph-mon[30820]: pgmap v5573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:37:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:25 smithi116 ceph-mon[35411]: pgmap v5573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:37:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-mon[30820]: pgmap v5574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:37:27.380+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:37:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:37:27.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:37:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:37:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:37:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:37:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:37:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:37:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:37:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:37:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:37:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:37:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:37:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:27 smithi116 ceph-mon[35411]: pgmap v5574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:37:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:37:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:37:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:37:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:37:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:29 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:37:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:37:29.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:37:29.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:29 smithi114 ceph-mon[30820]: pgmap v5575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:37:29.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:29 smithi114 ceph-mon[30820]: pgmap v5576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:29 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:29 smithi116 ceph-mon[35411]: pgmap v5575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:37:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:29 smithi116 ceph-mon[35411]: pgmap v5576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:37:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:31 smithi114 ceph-mon[30820]: pgmap v5577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:37:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:31 smithi116 ceph-mon[35411]: pgmap v5577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:37:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:33 smithi114 ceph-mon[30820]: pgmap v5578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:37:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:33 smithi116 ceph-mon[35411]: pgmap v5578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:37:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:35 smithi114 ceph-mon[30820]: pgmap v5579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:37:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:35 smithi116 ceph-mon[35411]: pgmap v5579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:37:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:37 smithi114 ceph-mon[30820]: pgmap v5580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:37:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:37 smithi116 ceph-mon[35411]: pgmap v5580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:37:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:39 smithi114 ceph-mon[30820]: pgmap v5581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:37:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:39 smithi116 ceph-mon[35411]: pgmap v5581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:37:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:41 smithi114 ceph-mon[30820]: pgmap v5582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:37:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:41 smithi116 ceph-mon[35411]: pgmap v5582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:37:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:43 smithi116 ceph-mon[35411]: pgmap v5583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:43 smithi114 ceph-mon[30820]: pgmap v5583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:45 smithi116 ceph-mon[35411]: pgmap v5584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:37:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:45 smithi114 ceph-mon[30820]: pgmap v5584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:37:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:47 smithi116 ceph-mon[35411]: pgmap v5585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:47 smithi114 ceph-mon[30820]: pgmap v5585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:49 smithi114 ceph-mon[30820]: pgmap v5586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:49 smithi116 ceph-mon[35411]: pgmap v5586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:51 smithi114 ceph-mon[30820]: pgmap v5587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:37:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:51 smithi116 ceph-mon[35411]: pgmap v5587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:37:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:53 smithi114 ceph-mon[30820]: pgmap v5588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:53 smithi116 ceph-mon[35411]: pgmap v5588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:37:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:55 smithi114 ceph-mon[30820]: pgmap v5589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:55 smithi116 ceph-mon[35411]: pgmap v5589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:57.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:57 smithi114 ceph-mon[30820]: pgmap v5590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:57 smithi116 ceph-mon[35411]: pgmap v5590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:37:59.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:37:59 smithi116 ceph-mon[35411]: pgmap v5591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:37:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:37:59 smithi114 ceph-mon[30820]: pgmap v5591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:01 smithi114 ceph-mon[30820]: pgmap v5592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:01 smithi116 ceph-mon[35411]: pgmap v5592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:03 smithi114 ceph-mon[30820]: pgmap v5593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:03 smithi116 ceph-mon[35411]: pgmap v5593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:05 smithi114 ceph-mon[30820]: pgmap v5594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:05 smithi116 ceph-mon[35411]: pgmap v5594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:07 smithi114 ceph-mon[30820]: pgmap v5595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:07 smithi116 ceph-mon[35411]: pgmap v5595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:38:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:09 smithi114 ceph-mon[30820]: pgmap v5596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:09 smithi116 ceph-mon[35411]: pgmap v5596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:11 smithi114 ceph-mon[30820]: pgmap v5597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:11 smithi116 ceph-mon[35411]: pgmap v5597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:13 smithi114 ceph-mon[30820]: pgmap v5598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:13 smithi116 ceph-mon[35411]: pgmap v5598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:14 smithi114 ceph-mon[30820]: pgmap v5599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:14 smithi116 ceph-mon[35411]: pgmap v5599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:16 smithi114 ceph-mon[30820]: pgmap v5600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:38:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:16 smithi116 ceph-mon[35411]: pgmap v5600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:38:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:18 smithi114 ceph-mon[30820]: pgmap v5601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:18 smithi116 ceph-mon[35411]: pgmap v5601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:20 smithi114 ceph-mon[30820]: pgmap v5602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:20 smithi116 ceph-mon[35411]: pgmap v5602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:22 smithi114 ceph-mon[30820]: pgmap v5603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:22 smithi116 ceph-mon[35411]: pgmap v5603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:24 smithi114 ceph-mon[30820]: pgmap v5604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:24 smithi116 ceph-mon[35411]: pgmap v5604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:26 smithi114 ceph-mon[30820]: pgmap v5605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:38:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:26 smithi116 ceph-mon[35411]: pgmap v5605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:38:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:28 smithi114 ceph-mon[30820]: pgmap v5606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:38:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:28 smithi116 ceph-mon[35411]: pgmap v5606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:38:30.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-mon[30820]: pgmap v5607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:30 smithi116 ceph-mon[35411]: pgmap v5607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:31.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:38:30.740+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:38:31.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:38:31.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:38:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:38:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:38:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:38:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:38:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:38:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:38:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:38:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:38:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:38:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:38:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:38:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:38:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:38:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:38:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:38:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:38:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:38:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:31 smithi114 ceph-mon[30820]: pgmap v5608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:31 smithi114 ceph-mon[30820]: pgmap v5609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:38:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:38:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:38:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:38:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:38:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:31 smithi116 ceph-mon[35411]: pgmap v5608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:38:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:31 smithi116 ceph-mon[35411]: pgmap v5609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:38:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:38:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:33 smithi114 ceph-mon[30820]: pgmap v5610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:38:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:33 smithi116 ceph-mon[35411]: pgmap v5610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:38:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:35 smithi114 ceph-mon[30820]: pgmap v5611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:38:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:35 smithi116 ceph-mon[35411]: pgmap v5611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:38:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:37 smithi114 ceph-mon[30820]: pgmap v5612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:38:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:37 smithi116 ceph-mon[35411]: pgmap v5612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:38:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:39 smithi114 ceph-mon[30820]: pgmap v5613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:38:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:39 smithi116 ceph-mon[35411]: pgmap v5613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:38:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:41 smithi114 ceph-mon[30820]: pgmap v5614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:38:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:41 smithi116 ceph-mon[35411]: pgmap v5614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:38:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:43 smithi114 ceph-mon[30820]: pgmap v5615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:38:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:43 smithi116 ceph-mon[35411]: pgmap v5615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:38:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:45 smithi114 ceph-mon[30820]: pgmap v5616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:38:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:45 smithi116 ceph-mon[35411]: pgmap v5616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:38:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:47 smithi114 ceph-mon[30820]: pgmap v5617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:47 smithi116 ceph-mon[35411]: pgmap v5617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:49 smithi114 ceph-mon[30820]: pgmap v5618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:38:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:49 smithi116 ceph-mon[35411]: pgmap v5618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:38:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:51 smithi114 ceph-mon[30820]: pgmap v5619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:38:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:51 smithi116 ceph-mon[35411]: pgmap v5619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:38:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:53 smithi114 ceph-mon[30820]: pgmap v5620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:53 smithi116 ceph-mon[35411]: pgmap v5620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:38:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:55 smithi114 ceph-mon[30820]: pgmap v5621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:55 smithi116 ceph-mon[35411]: pgmap v5621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:38:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:57 smithi114 ceph-mon[30820]: pgmap v5622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:38:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:57 smithi116 ceph-mon[35411]: pgmap v5622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:38:59 smithi114 ceph-mon[30820]: pgmap v5623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:38:59 smithi116 ceph-mon[35411]: pgmap v5623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:01 smithi114 ceph-mon[30820]: pgmap v5624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:01 smithi116 ceph-mon[35411]: pgmap v5624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:03 smithi114 ceph-mon[30820]: pgmap v5625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:03 smithi116 ceph-mon[35411]: pgmap v5625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:05 smithi114 ceph-mon[30820]: pgmap v5626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:39:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:05 smithi116 ceph-mon[35411]: pgmap v5626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:39:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:07 smithi114 ceph-mon[30820]: pgmap v5627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:07 smithi116 ceph-mon[35411]: pgmap v5627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:09 smithi114 ceph-mon[30820]: pgmap v5628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:09 smithi116 ceph-mon[35411]: pgmap v5628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:11 smithi114 ceph-mon[30820]: pgmap v5629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:11 smithi116 ceph-mon[35411]: pgmap v5629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:13 smithi114 ceph-mon[30820]: pgmap v5630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:13 smithi116 ceph-mon[35411]: pgmap v5630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:15 smithi114 ceph-mon[30820]: pgmap v5631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:39:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:15 smithi116 ceph-mon[35411]: pgmap v5631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:39:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:17 smithi114 ceph-mon[30820]: pgmap v5632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:17 smithi116 ceph-mon[35411]: pgmap v5632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:19 smithi114 ceph-mon[30820]: pgmap v5633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:19 smithi116 ceph-mon[35411]: pgmap v5633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:21 smithi114 ceph-mon[30820]: pgmap v5634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:21 smithi116 ceph-mon[35411]: pgmap v5634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:23 smithi114 ceph-mon[30820]: pgmap v5635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:23 smithi116 ceph-mon[35411]: pgmap v5635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:25 smithi114 ceph-mon[30820]: pgmap v5636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:39:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:25 smithi116 ceph-mon[35411]: pgmap v5636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:39:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:27 smithi114 ceph-mon[30820]: pgmap v5637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:27 smithi116 ceph-mon[35411]: pgmap v5637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:29 smithi114 ceph-mon[30820]: pgmap v5638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:29 smithi116 ceph-mon[35411]: pgmap v5638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:39:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:39:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:31 smithi114 ceph-mon[30820]: pgmap v5639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:31 smithi116 ceph-mon[35411]: pgmap v5639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:33 smithi114 ceph-mon[30820]: pgmap v5640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:33 smithi116 ceph-mon[35411]: pgmap v5640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:39:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:39:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:39:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:39:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:39:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-mon[30820]: pgmap v5641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-mon[30820]: pgmap v5642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:39:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:34.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:39:34.100+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:39:34.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:39:34.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:39:34.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:39:34.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:39:34.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:39:34.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:39:34.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:39:34.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:39:34.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:39:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:39:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:39:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:39:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:39:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:39:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:39:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:39:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:34 smithi116 ceph-mon[35411]: pgmap v5641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:39:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:34 smithi116 ceph-mon[35411]: pgmap v5642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:39:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:39:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:37 smithi114 ceph-mon[30820]: pgmap v5643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:39:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:37 smithi116 ceph-mon[35411]: pgmap v5643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:39:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:39 smithi114 ceph-mon[30820]: pgmap v5644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:39:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:39 smithi116 ceph-mon[35411]: pgmap v5644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:39:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:41 smithi116 ceph-mon[35411]: pgmap v5645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:39:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:41 smithi114 ceph-mon[30820]: pgmap v5645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:39:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:43 smithi116 ceph-mon[35411]: pgmap v5646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:39:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:43 smithi114 ceph-mon[30820]: pgmap v5646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:39:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:45 smithi116 ceph-mon[35411]: pgmap v5647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:39:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:45 smithi114 ceph-mon[30820]: pgmap v5647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:39:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:47 smithi114 ceph-mon[30820]: pgmap v5648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:47 smithi116 ceph-mon[35411]: pgmap v5648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:49 smithi114 ceph-mon[30820]: pgmap v5649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:49 smithi116 ceph-mon[35411]: pgmap v5649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:51 smithi114 ceph-mon[30820]: pgmap v5650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:39:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:51 smithi116 ceph-mon[35411]: pgmap v5650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:39:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:53 smithi114 ceph-mon[30820]: pgmap v5651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:53 smithi116 ceph-mon[35411]: pgmap v5651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:39:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:55 smithi114 ceph-mon[30820]: pgmap v5652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:55 smithi116 ceph-mon[35411]: pgmap v5652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:57 smithi114 ceph-mon[30820]: pgmap v5653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:57 smithi116 ceph-mon[35411]: pgmap v5653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:39:59.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:39:59 smithi114 ceph-mon[30820]: pgmap v5654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:39:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:39:59 smithi116 ceph-mon[35411]: pgmap v5654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:00.494 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:40:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:40:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:01 smithi114 ceph-mon[30820]: pgmap v5655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:01.580 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:01 smithi116 ceph-mon[35411]: pgmap v5655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:03 smithi114 ceph-mon[30820]: pgmap v5656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:03 smithi116 ceph-mon[35411]: pgmap v5656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:05 smithi114 ceph-mon[30820]: pgmap v5657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:05 smithi116 ceph-mon[35411]: pgmap v5657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:07 smithi114 ceph-mon[30820]: pgmap v5658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:07 smithi116 ceph-mon[35411]: pgmap v5658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:09 smithi114 ceph-mon[30820]: pgmap v5659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:09 smithi116 ceph-mon[35411]: pgmap v5659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:11 smithi114 ceph-mon[30820]: pgmap v5660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:11 smithi116 ceph-mon[35411]: pgmap v5660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:13 smithi114 ceph-mon[30820]: pgmap v5661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:13 smithi116 ceph-mon[35411]: pgmap v5661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:15 smithi114 ceph-mon[30820]: pgmap v5662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:15 smithi116 ceph-mon[35411]: pgmap v5662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:17 smithi114 ceph-mon[30820]: pgmap v5663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:17 smithi116 ceph-mon[35411]: pgmap v5663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:19 smithi114 ceph-mon[30820]: pgmap v5664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:19 smithi116 ceph-mon[35411]: pgmap v5664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:21 smithi114 ceph-mon[30820]: pgmap v5665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:21 smithi116 ceph-mon[35411]: pgmap v5665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:23 smithi114 ceph-mon[30820]: pgmap v5666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:23 smithi116 ceph-mon[35411]: pgmap v5666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:24 smithi114 ceph-mon[30820]: pgmap v5667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:24 smithi116 ceph-mon[35411]: pgmap v5667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:27 smithi114 ceph-mon[30820]: pgmap v5668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:27 smithi116 ceph-mon[35411]: pgmap v5668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:29.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:29 smithi114 ceph-mon[30820]: pgmap v5669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:29.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:29 smithi116 ceph-mon[35411]: pgmap v5669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:31.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:31 smithi116 ceph-mon[35411]: pgmap v5670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:31 smithi114 ceph-mon[30820]: pgmap v5670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:33 smithi114 ceph-mon[30820]: pgmap v5671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:33 smithi116 ceph-mon[35411]: pgmap v5671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:35 smithi114 ceph-mon[30820]: pgmap v5672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:40:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:35 smithi116 ceph-mon[35411]: pgmap v5672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:40:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-mon[30820]: pgmap v5673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:37.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:40:37.396+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:40:37.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:40:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:40:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:40:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:40:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:37 smithi116 ceph-mon[35411]: pgmap v5673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:40:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:40:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:40:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:40:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:40:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:40:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:40:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:40:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:40:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:40:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:40:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:40:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:40:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:40:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:40:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:40:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:40:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:39 smithi114 ceph-mon[30820]: pgmap v5674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:40:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:39 smithi114 ceph-mon[30820]: pgmap v5675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:40:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:40:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:40:39.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:40:39.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:40:39.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:39 smithi116 ceph-mon[35411]: pgmap v5674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:40:39.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:39 smithi116 ceph-mon[35411]: pgmap v5675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:41 smithi114 ceph-mon[30820]: pgmap v5676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:41.720 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:41 smithi116 ceph-mon[35411]: pgmap v5676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:43 smithi114 ceph-mon[30820]: pgmap v5677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:43 smithi116 ceph-mon[35411]: pgmap v5677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:45 smithi114 ceph-mon[30820]: pgmap v5678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:45 smithi116 ceph-mon[35411]: pgmap v5678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:47 smithi114 ceph-mon[30820]: pgmap v5679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:47 smithi116 ceph-mon[35411]: pgmap v5679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:40:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:49 smithi114 ceph-mon[30820]: pgmap v5680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:40:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:49 smithi116 ceph-mon[35411]: pgmap v5680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:40:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:51 smithi114 ceph-mon[30820]: pgmap v5681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:51 smithi116 ceph-mon[35411]: pgmap v5681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:40:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:53 smithi114 ceph-mon[30820]: pgmap v5682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:53 smithi116 ceph-mon[35411]: pgmap v5682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:40:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:55 smithi114 ceph-mon[30820]: pgmap v5683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:55 smithi116 ceph-mon[35411]: pgmap v5683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:57 smithi114 ceph-mon[30820]: pgmap v5684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:57 smithi116 ceph-mon[35411]: pgmap v5684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:40:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:40:59 smithi114 ceph-mon[30820]: pgmap v5685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:40:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:40:59 smithi116 ceph-mon[35411]: pgmap v5685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:01 smithi114 ceph-mon[30820]: pgmap v5686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:01 smithi116 ceph-mon[35411]: pgmap v5686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:03 smithi114 ceph-mon[30820]: pgmap v5687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:03 smithi116 ceph-mon[35411]: pgmap v5687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:05 smithi114 ceph-mon[30820]: pgmap v5688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:05 smithi116 ceph-mon[35411]: pgmap v5688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:07 smithi114 ceph-mon[30820]: pgmap v5689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:41:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:07 smithi116 ceph-mon[35411]: pgmap v5689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:41:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:08 smithi114 ceph-mon[30820]: pgmap v5690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:08 smithi116 ceph-mon[35411]: pgmap v5690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:10 smithi114 ceph-mon[30820]: pgmap v5691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:10 smithi116 ceph-mon[35411]: pgmap v5691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:12 smithi114 ceph-mon[30820]: pgmap v5692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:12 smithi116 ceph-mon[35411]: pgmap v5692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:14 smithi114 ceph-mon[30820]: pgmap v5693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:14 smithi116 ceph-mon[35411]: pgmap v5693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:16 smithi114 ceph-mon[30820]: pgmap v5694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:41:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:16 smithi116 ceph-mon[35411]: pgmap v5694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:41:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:18 smithi114 ceph-mon[30820]: pgmap v5695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:18 smithi116 ceph-mon[35411]: pgmap v5695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:20 smithi114 ceph-mon[30820]: pgmap v5696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:20 smithi116 ceph-mon[35411]: pgmap v5696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:22 smithi114 ceph-mon[30820]: pgmap v5697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:22 smithi116 ceph-mon[35411]: pgmap v5697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:24 smithi114 ceph-mon[30820]: pgmap v5698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:24 smithi116 ceph-mon[35411]: pgmap v5698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:26 smithi114 ceph-mon[30820]: pgmap v5699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:41:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:26 smithi116 ceph-mon[35411]: pgmap v5699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:41:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:28 smithi114 ceph-mon[30820]: pgmap v5700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:41:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:28 smithi116 ceph-mon[35411]: pgmap v5700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:41:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:30 smithi114 ceph-mon[30820]: pgmap v5701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:41:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:30 smithi116 ceph-mon[35411]: pgmap v5701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:41:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:32 smithi114 ceph-mon[30820]: pgmap v5702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:32 smithi116 ceph-mon[35411]: pgmap v5702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:34 smithi114 ceph-mon[30820]: pgmap v5703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:41:34.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:34 smithi116 ceph-mon[35411]: pgmap v5703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:41:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:36 smithi114 ceph-mon[30820]: pgmap v5704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:41:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:36 smithi116 ceph-mon[35411]: pgmap v5704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:41:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:38 smithi114 ceph-mon[30820]: pgmap v5705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:41:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:38 smithi116 ceph-mon[35411]: pgmap v5705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:41:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-mon[30820]: pgmap v5706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:40.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:41:40.756+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:41:40.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:41:40.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:41:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:41:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:40 smithi116 ceph-mon[35411]: pgmap v5706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:41:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:41:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:41:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:41:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:41:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:41 smithi114 ceph-mon[30820]: pgmap v5707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:41 smithi114 ceph-mon[30820]: pgmap v5708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:41:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:41:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:41:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:41:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:41:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:41:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:41:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:41:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:41:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:41:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:41:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:41:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:41:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:41:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:41 smithi116 ceph-mon[35411]: pgmap v5707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:41:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:41 smithi116 ceph-mon[35411]: pgmap v5708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:41:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:41:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:43 smithi114 ceph-mon[30820]: pgmap v5709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:41:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:43 smithi116 ceph-mon[35411]: pgmap v5709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:41:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:45 smithi114 ceph-mon[30820]: pgmap v5710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:41:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:45 smithi116 ceph-mon[35411]: pgmap v5710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:41:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:47 smithi114 ceph-mon[30820]: pgmap v5711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:41:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:47 smithi116 ceph-mon[35411]: pgmap v5711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:41:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:49 smithi114 ceph-mon[30820]: pgmap v5712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:41:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:49 smithi116 ceph-mon[35411]: pgmap v5712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:41:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:51 smithi114 ceph-mon[30820]: pgmap v5713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:41:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:51 smithi116 ceph-mon[35411]: pgmap v5713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:41:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:53 smithi114 ceph-mon[30820]: pgmap v5714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:53 smithi116 ceph-mon[35411]: pgmap v5714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:41:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:55 smithi114 ceph-mon[30820]: pgmap v5715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:55 smithi116 ceph-mon[35411]: pgmap v5715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:41:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:57 smithi114 ceph-mon[30820]: pgmap v5716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:41:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:57 smithi116 ceph-mon[35411]: pgmap v5716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:41:59 smithi114 ceph-mon[30820]: pgmap v5717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:41:59 smithi116 ceph-mon[35411]: pgmap v5717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:01 smithi114 ceph-mon[30820]: pgmap v5718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:01 smithi116 ceph-mon[35411]: pgmap v5718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:03 smithi114 ceph-mon[30820]: pgmap v5719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:03 smithi116 ceph-mon[35411]: pgmap v5719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:05 smithi114 ceph-mon[30820]: pgmap v5720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:42:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:05 smithi116 ceph-mon[35411]: pgmap v5720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:42:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:07 smithi114 ceph-mon[30820]: pgmap v5721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:07 smithi116 ceph-mon[35411]: pgmap v5721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:09 smithi114 ceph-mon[30820]: pgmap v5722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:09 smithi116 ceph-mon[35411]: pgmap v5722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:11 smithi114 ceph-mon[30820]: pgmap v5723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:11 smithi116 ceph-mon[35411]: pgmap v5723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:13 smithi114 ceph-mon[30820]: pgmap v5724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:13 smithi116 ceph-mon[35411]: pgmap v5724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:15 smithi114 ceph-mon[30820]: pgmap v5725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:42:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:15 smithi116 ceph-mon[35411]: pgmap v5725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:42:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:17 smithi114 ceph-mon[30820]: pgmap v5726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:17 smithi116 ceph-mon[35411]: pgmap v5726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:19 smithi114 ceph-mon[30820]: pgmap v5727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:19 smithi116 ceph-mon[35411]: pgmap v5727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:21 smithi114 ceph-mon[30820]: pgmap v5728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:21 smithi116 ceph-mon[35411]: pgmap v5728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:23 smithi114 ceph-mon[30820]: pgmap v5729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:23 smithi116 ceph-mon[35411]: pgmap v5729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:25 smithi114 ceph-mon[30820]: pgmap v5730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:42:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:25 smithi116 ceph-mon[35411]: pgmap v5730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:42:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:27 smithi114 ceph-mon[30820]: pgmap v5731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:27 smithi116 ceph-mon[35411]: pgmap v5731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:29 smithi114 ceph-mon[30820]: pgmap v5732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:29 smithi116 ceph-mon[35411]: pgmap v5732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:31 smithi114 ceph-mon[30820]: pgmap v5733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:31 smithi116 ceph-mon[35411]: pgmap v5733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:33 smithi114 ceph-mon[30820]: pgmap v5734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:33 smithi116 ceph-mon[35411]: pgmap v5734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:35 smithi114 ceph-mon[30820]: pgmap v5735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:42:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:35 smithi116 ceph-mon[35411]: pgmap v5735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:42:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:37 smithi116 ceph-mon[35411]: pgmap v5736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:37 smithi114 ceph-mon[30820]: pgmap v5736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:42:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:39 smithi116 ceph-mon[35411]: pgmap v5737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:39 smithi114 ceph-mon[30820]: pgmap v5737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:42:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:42:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:41 smithi114 ceph-mon[30820]: pgmap v5738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:41 smithi116 ceph-mon[35411]: pgmap v5738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:43 smithi114 ceph-mon[30820]: pgmap v5739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:44.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:42:44.116+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:42:44.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:42:44.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:42:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:42:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:42:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:42:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:42:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:42:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:42:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:42:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:42:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:43 smithi116 ceph-mon[35411]: pgmap v5739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:42:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:42:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-mon[30820]: pgmap v5740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-mon[30820]: pgmap v5741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:42:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:44 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:42:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:42:45.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:42:45.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:44 smithi116 ceph-mon[35411]: pgmap v5740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:42:45.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:44 smithi116 ceph-mon[35411]: pgmap v5741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:42:45.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:42:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:47 smithi116 ceph-mon[35411]: pgmap v5742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:42:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:47 smithi114 ceph-mon[30820]: pgmap v5742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:42:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:49 smithi114 ceph-mon[30820]: pgmap v5743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:42:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:49 smithi116 ceph-mon[35411]: pgmap v5743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:42:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:51 smithi114 ceph-mon[30820]: pgmap v5744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:42:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:51 smithi116 ceph-mon[35411]: pgmap v5744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:42:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:53 smithi114 ceph-mon[30820]: pgmap v5745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:42:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:53 smithi116 ceph-mon[35411]: pgmap v5745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:42:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:55 smithi114 ceph-mon[30820]: pgmap v5746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:42:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:55 smithi116 ceph-mon[35411]: pgmap v5746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:42:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:57 smithi114 ceph-mon[30820]: pgmap v5747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:57 smithi116 ceph-mon[35411]: pgmap v5747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:42:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:42:59 smithi114 ceph-mon[30820]: pgmap v5748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:42:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:42:59 smithi116 ceph-mon[35411]: pgmap v5748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:01.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:01 smithi114 ceph-mon[30820]: pgmap v5749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:01 smithi116 ceph-mon[35411]: pgmap v5749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:03.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:03 smithi116 ceph-mon[35411]: pgmap v5750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:03 smithi114 ceph-mon[30820]: pgmap v5750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:05 smithi114 ceph-mon[30820]: pgmap v5751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:05 smithi116 ceph-mon[35411]: pgmap v5751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:07 smithi114 ceph-mon[30820]: pgmap v5752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:07 smithi116 ceph-mon[35411]: pgmap v5752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:09 smithi114 ceph-mon[30820]: pgmap v5753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:09 smithi116 ceph-mon[35411]: pgmap v5753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:11 smithi114 ceph-mon[30820]: pgmap v5754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:11 smithi116 ceph-mon[35411]: pgmap v5754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:13 smithi114 ceph-mon[30820]: pgmap v5755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:13 smithi116 ceph-mon[35411]: pgmap v5755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:15 smithi114 ceph-mon[30820]: pgmap v5756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:15 smithi116 ceph-mon[35411]: pgmap v5756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:17 smithi114 ceph-mon[30820]: pgmap v5757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:17 smithi116 ceph-mon[35411]: pgmap v5757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:19 smithi114 ceph-mon[30820]: pgmap v5758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:19 smithi116 ceph-mon[35411]: pgmap v5758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:21 smithi114 ceph-mon[30820]: pgmap v5759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:21 smithi116 ceph-mon[35411]: pgmap v5759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:23 smithi114 ceph-mon[30820]: pgmap v5760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:23 smithi116 ceph-mon[35411]: pgmap v5760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:25 smithi114 ceph-mon[30820]: pgmap v5761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:25 smithi116 ceph-mon[35411]: pgmap v5761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:27 smithi114 ceph-mon[30820]: pgmap v5762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:27 smithi116 ceph-mon[35411]: pgmap v5762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:29 smithi114 ceph-mon[30820]: pgmap v5763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:29 smithi116 ceph-mon[35411]: pgmap v5763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:31 smithi114 ceph-mon[30820]: pgmap v5764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:31 smithi116 ceph-mon[35411]: pgmap v5764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:33 smithi114 ceph-mon[30820]: pgmap v5765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:33 smithi116 ceph-mon[35411]: pgmap v5765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:35 smithi114 ceph-mon[30820]: pgmap v5766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:35.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:35 smithi116 ceph-mon[35411]: pgmap v5766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:37 smithi114 ceph-mon[30820]: pgmap v5767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:37 smithi116 ceph-mon[35411]: pgmap v5767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:39 smithi114 ceph-mon[30820]: pgmap v5768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:39 smithi116 ceph-mon[35411]: pgmap v5768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:41 smithi114 ceph-mon[30820]: pgmap v5769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:41 smithi116 ceph-mon[35411]: pgmap v5769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:43:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:43 smithi114 ceph-mon[30820]: pgmap v5770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:43 smithi116 ceph-mon[35411]: pgmap v5770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:43:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:44 smithi114 ceph-mon[30820]: pgmap v5771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:43:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:44 smithi116 ceph-mon[35411]: pgmap v5771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:43:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:43:47.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-mon[30820]: pgmap v5772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:47 smithi116 ceph-mon[35411]: pgmap v5772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:43:47.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:43:47.476+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:43:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:43:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:43:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:43:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:43:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:43:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:43:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:43:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:43:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:43:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:43:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:43:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:49 smithi114 ceph-mon[30820]: pgmap v5773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:43:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:49 smithi114 ceph-mon[30820]: pgmap v5774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:43:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:43:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:43:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:43:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:43:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:49 smithi116 ceph-mon[35411]: pgmap v5773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:43:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:49 smithi116 ceph-mon[35411]: pgmap v5774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:51 smithi114 ceph-mon[30820]: pgmap v5775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:51 smithi116 ceph-mon[35411]: pgmap v5775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:53 smithi114 ceph-mon[30820]: pgmap v5776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:53 smithi116 ceph-mon[35411]: pgmap v5776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:55 smithi114 ceph-mon[30820]: pgmap v5777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:55 smithi116 ceph-mon[35411]: pgmap v5777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:57 smithi114 ceph-mon[30820]: pgmap v5778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:57 smithi116 ceph-mon[35411]: pgmap v5778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:43:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:43:59 smithi114 ceph-mon[30820]: pgmap v5779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:43:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:43:59 smithi116 ceph-mon[35411]: pgmap v5779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:44:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:01 smithi114 ceph-mon[30820]: pgmap v5780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:02.030 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:01 smithi116 ceph-mon[35411]: pgmap v5780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:03 smithi114 ceph-mon[30820]: pgmap v5781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:03 smithi116 ceph-mon[35411]: pgmap v5781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:05 smithi114 ceph-mon[30820]: pgmap v5782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:05 smithi116 ceph-mon[35411]: pgmap v5782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:07.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:07 smithi114 ceph-mon[30820]: pgmap v5783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:07 smithi116 ceph-mon[35411]: pgmap v5783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:09 smithi114 ceph-mon[30820]: pgmap v5784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:09 smithi116 ceph-mon[35411]: pgmap v5784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:11 smithi114 ceph-mon[30820]: pgmap v5785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:11 smithi116 ceph-mon[35411]: pgmap v5785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:13 smithi114 ceph-mon[30820]: pgmap v5786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:13.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:13 smithi116 ceph-mon[35411]: pgmap v5786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:15 smithi114 ceph-mon[30820]: pgmap v5787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:15 smithi116 ceph-mon[35411]: pgmap v5787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:17 smithi114 ceph-mon[30820]: pgmap v5788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:17 smithi116 ceph-mon[35411]: pgmap v5788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:19 smithi114 ceph-mon[30820]: pgmap v5789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:19 smithi116 ceph-mon[35411]: pgmap v5789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:21 smithi114 ceph-mon[30820]: pgmap v5790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:21 smithi116 ceph-mon[35411]: pgmap v5790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:23 smithi114 ceph-mon[30820]: pgmap v5791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:23 smithi116 ceph-mon[35411]: pgmap v5791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:25 smithi114 ceph-mon[30820]: pgmap v5792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:25 smithi116 ceph-mon[35411]: pgmap v5792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:27 smithi114 ceph-mon[30820]: pgmap v5793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:27 smithi116 ceph-mon[35411]: pgmap v5793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:29 smithi114 ceph-mon[30820]: pgmap v5794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:29 smithi116 ceph-mon[35411]: pgmap v5794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:31 smithi114 ceph-mon[30820]: pgmap v5795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:31 smithi116 ceph-mon[35411]: pgmap v5795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:33 smithi114 ceph-mon[30820]: pgmap v5796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:33 smithi116 ceph-mon[35411]: pgmap v5796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:35 smithi114 ceph-mon[30820]: pgmap v5797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:35 smithi116 ceph-mon[35411]: pgmap v5797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:36 smithi114 ceph-mon[30820]: pgmap v5798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:36 smithi116 ceph-mon[35411]: pgmap v5798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:38 smithi114 ceph-mon[30820]: pgmap v5799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:38 smithi116 ceph-mon[35411]: pgmap v5799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:40 smithi114 ceph-mon[30820]: pgmap v5800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:40 smithi116 ceph-mon[35411]: pgmap v5800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:42 smithi114 ceph-mon[30820]: pgmap v5801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:42 smithi116 ceph-mon[35411]: pgmap v5801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:44:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:44 smithi114 ceph-mon[30820]: pgmap v5802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:44 smithi116 ceph-mon[35411]: pgmap v5802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:46 smithi114 ceph-mon[30820]: pgmap v5803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:46 smithi116 ceph-mon[35411]: pgmap v5803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:44:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:48 smithi114 ceph-mon[30820]: pgmap v5804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:44:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:48 smithi116 ceph-mon[35411]: pgmap v5804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:44:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:44:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:44:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:44:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:44:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:44:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-mon[30820]: pgmap v5805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:44:50.837+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:44:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:44:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:44:51.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:44:51.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:44:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:44:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:50 smithi116 ceph-mon[35411]: pgmap v5805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:44:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:44:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:44:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:44:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:44:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:44:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:51 smithi114 ceph-mon[30820]: pgmap v5806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:51 smithi114 ceph-mon[30820]: pgmap v5807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:44:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:44:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:44:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:44:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:44:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:44:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:44:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:51 smithi116 ceph-mon[35411]: pgmap v5806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:44:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:51 smithi116 ceph-mon[35411]: pgmap v5807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:44:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:44:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:53 smithi114 ceph-mon[30820]: pgmap v5808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:44:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:53 smithi116 ceph-mon[35411]: pgmap v5808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:44:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:55 smithi114 ceph-mon[30820]: pgmap v5809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:44:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:55 smithi116 ceph-mon[35411]: pgmap v5809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:44:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:57 smithi114 ceph-mon[30820]: pgmap v5810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:44:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:57 smithi116 ceph-mon[35411]: pgmap v5810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:45:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:44:59 smithi114 ceph-mon[30820]: pgmap v5811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:45:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:44:59 smithi116 ceph-mon[35411]: pgmap v5811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:45:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:01 smithi114 ceph-mon[30820]: pgmap v5812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:45:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:01 smithi116 ceph-mon[35411]: pgmap v5812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:45:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:03 smithi116 ceph-mon[35411]: pgmap v5813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:03 smithi114 ceph-mon[30820]: pgmap v5813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:05 smithi116 ceph-mon[35411]: pgmap v5814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:05 smithi114 ceph-mon[30820]: pgmap v5814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:07 smithi116 ceph-mon[35411]: pgmap v5815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:07 smithi114 ceph-mon[30820]: pgmap v5815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:09 smithi116 ceph-mon[35411]: pgmap v5816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:09 smithi114 ceph-mon[30820]: pgmap v5816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:11 smithi114 ceph-mon[30820]: pgmap v5817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:11 smithi116 ceph-mon[35411]: pgmap v5817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:13 smithi114 ceph-mon[30820]: pgmap v5818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:13 smithi116 ceph-mon[35411]: pgmap v5818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:15 smithi114 ceph-mon[30820]: pgmap v5819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:15 smithi116 ceph-mon[35411]: pgmap v5819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:17 smithi114 ceph-mon[30820]: pgmap v5820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:17 smithi116 ceph-mon[35411]: pgmap v5820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:19 smithi114 ceph-mon[30820]: pgmap v5821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:19 smithi116 ceph-mon[35411]: pgmap v5821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:21 smithi114 ceph-mon[30820]: pgmap v5822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:21 smithi116 ceph-mon[35411]: pgmap v5822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:23 smithi114 ceph-mon[30820]: pgmap v5823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:23 smithi116 ceph-mon[35411]: pgmap v5823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:25 smithi114 ceph-mon[30820]: pgmap v5824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:25 smithi116 ceph-mon[35411]: pgmap v5824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:27 smithi114 ceph-mon[30820]: pgmap v5825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:27 smithi116 ceph-mon[35411]: pgmap v5825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:29 smithi114 ceph-mon[30820]: pgmap v5826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:29 smithi116 ceph-mon[35411]: pgmap v5826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:31 smithi114 ceph-mon[30820]: pgmap v5827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:31 smithi116 ceph-mon[35411]: pgmap v5827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:33 smithi116 ceph-mon[35411]: pgmap v5828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:33 smithi114 ceph-mon[30820]: pgmap v5828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:35 smithi116 ceph-mon[35411]: pgmap v5829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:35 smithi114 ceph-mon[30820]: pgmap v5829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:38.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:37 smithi116 ceph-mon[35411]: pgmap v5830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:37 smithi114 ceph-mon[30820]: pgmap v5830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:39 smithi114 ceph-mon[30820]: pgmap v5831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:39 smithi116 ceph-mon[35411]: pgmap v5831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:41 smithi114 ceph-mon[30820]: pgmap v5832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:41 smithi116 ceph-mon[35411]: pgmap v5832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:43 smithi114 ceph-mon[30820]: pgmap v5833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:43 smithi116 ceph-mon[35411]: pgmap v5833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:45 smithi114 ceph-mon[30820]: pgmap v5834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:45 smithi116 ceph-mon[35411]: pgmap v5834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:45:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:47 smithi114 ceph-mon[30820]: pgmap v5835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:47 smithi116 ceph-mon[35411]: pgmap v5835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:45:50.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:49 smithi114 ceph-mon[30820]: pgmap v5836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:49 smithi116 ceph-mon[35411]: pgmap v5836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:51 smithi114 ceph-mon[30820]: pgmap v5837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:45:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:45:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:45:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:51 smithi116 ceph-mon[35411]: pgmap v5837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:45:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:45:52.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:45:52.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:45:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:53 smithi114 ceph-mon[30820]: pgmap v5838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:45:54.196+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:45:54.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:45:54.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:45:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:53 smithi116 ceph-mon[35411]: pgmap v5838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:45:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:45:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:45:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:45:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:45:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:45:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:45:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:45:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:45:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:45:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:45:55.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:45:55.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:45:55.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:45:55.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-mon[30820]: pgmap v5839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:45:55.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-mon[30820]: pgmap v5840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:45:55.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:45:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:45:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:45:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:45:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:45:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:45:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:45:55.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:54 smithi116 ceph-mon[35411]: pgmap v5839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:54 smithi116 ceph-mon[35411]: pgmap v5840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:45:55.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:45:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:57 smithi114 ceph-mon[30820]: pgmap v5841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:45:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:57 smithi116 ceph-mon[35411]: pgmap v5841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:45:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:45:59 smithi114 ceph-mon[30820]: pgmap v5842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:45:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:45:59 smithi116 ceph-mon[35411]: pgmap v5842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:46:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:01 smithi114 ceph-mon[30820]: pgmap v5843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:46:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:01 smithi116 ceph-mon[35411]: pgmap v5843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:46:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:03 smithi114 ceph-mon[30820]: pgmap v5844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:46:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:03 smithi116 ceph-mon[35411]: pgmap v5844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:46:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:05 smithi114 ceph-mon[30820]: pgmap v5845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:46:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:05 smithi116 ceph-mon[35411]: pgmap v5845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:46:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:07 smithi114 ceph-mon[30820]: pgmap v5846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:07 smithi116 ceph-mon[35411]: pgmap v5846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:09 smithi114 ceph-mon[30820]: pgmap v5847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:09 smithi116 ceph-mon[35411]: pgmap v5847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:11 smithi114 ceph-mon[30820]: pgmap v5848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:11 smithi116 ceph-mon[35411]: pgmap v5848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:13 smithi114 ceph-mon[30820]: pgmap v5849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:13 smithi116 ceph-mon[35411]: pgmap v5849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:15 smithi114 ceph-mon[30820]: pgmap v5850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:15 smithi116 ceph-mon[35411]: pgmap v5850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:17 smithi114 ceph-mon[30820]: pgmap v5851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:17 smithi116 ceph-mon[35411]: pgmap v5851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:19 smithi114 ceph-mon[30820]: pgmap v5852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:19 smithi116 ceph-mon[35411]: pgmap v5852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:21 smithi114 ceph-mon[30820]: pgmap v5853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:21 smithi116 ceph-mon[35411]: pgmap v5853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:23 smithi114 ceph-mon[30820]: pgmap v5854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:23 smithi116 ceph-mon[35411]: pgmap v5854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:25 smithi114 ceph-mon[30820]: pgmap v5855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:25 smithi116 ceph-mon[35411]: pgmap v5855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:27 smithi114 ceph-mon[30820]: pgmap v5856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:27 smithi116 ceph-mon[35411]: pgmap v5856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:28 smithi114 ceph-mon[30820]: pgmap v5857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:28 smithi116 ceph-mon[35411]: pgmap v5857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:31 smithi114 ceph-mon[30820]: pgmap v5858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:31 smithi116 ceph-mon[35411]: pgmap v5858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:33 smithi114 ceph-mon[30820]: pgmap v5859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:33 smithi116 ceph-mon[35411]: pgmap v5859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:35 smithi114 ceph-mon[30820]: pgmap v5860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:35 smithi116 ceph-mon[35411]: pgmap v5860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:37 smithi114 ceph-mon[30820]: pgmap v5861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:37 smithi116 ceph-mon[35411]: pgmap v5861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:39 smithi114 ceph-mon[30820]: pgmap v5862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:39 smithi116 ceph-mon[35411]: pgmap v5862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:41 smithi114 ceph-mon[30820]: pgmap v5863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:41 smithi116 ceph-mon[35411]: pgmap v5863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:43 smithi114 ceph-mon[30820]: pgmap v5864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:43 smithi116 ceph-mon[35411]: pgmap v5864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:45 smithi114 ceph-mon[30820]: pgmap v5865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:45 smithi116 ceph-mon[35411]: pgmap v5865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:47 smithi114 ceph-mon[30820]: pgmap v5866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:47 smithi116 ceph-mon[35411]: pgmap v5866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:49 smithi114 ceph-mon[30820]: pgmap v5867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:49 smithi116 ceph-mon[35411]: pgmap v5867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:51 smithi114 ceph-mon[30820]: pgmap v5868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:51 smithi116 ceph-mon[35411]: pgmap v5868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:46:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:53 smithi114 ceph-mon[30820]: pgmap v5869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:53 smithi116 ceph-mon[35411]: pgmap v5869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:46:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: pgmap v5870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:46:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: pgmap v5870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:46:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:46:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:55.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:46:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:56 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:46:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:56 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:46:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:56 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:46:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:56 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:46:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:56 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:46:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:56 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:46:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:56 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:46:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:56 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:46:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-mon[30820]: pgmap v5871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:57.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:46:57.556+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:46:57.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:46:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:46:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:46:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:46:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:57 smithi116 ceph-mon[35411]: pgmap v5871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:46:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:58 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:46:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:46:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:46:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:46:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:46:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:58 smithi114 ceph-mon[30820]: pgmap v5872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:58 smithi114 ceph-mon[30820]: pgmap v5873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:46:58.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:46:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:46:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:58 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:46:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:46:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:46:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:46:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:46:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:58 smithi116 ceph-mon[35411]: pgmap v5872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:58 smithi116 ceph-mon[35411]: pgmap v5873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:46:58.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:46:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:47:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:00 smithi114 ceph-mon[30820]: pgmap v5874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:47:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:00 smithi116 ceph-mon[35411]: pgmap v5874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:47:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:02 smithi114 ceph-mon[30820]: pgmap v5875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:47:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:02 smithi116 ceph-mon[35411]: pgmap v5875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:47:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:04 smithi114 ceph-mon[30820]: pgmap v5876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:47:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:04 smithi116 ceph-mon[35411]: pgmap v5876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:47:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:06 smithi114 ceph-mon[30820]: pgmap v5877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:47:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:06 smithi116 ceph-mon[35411]: pgmap v5877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:47:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:08 smithi114 ceph-mon[30820]: pgmap v5878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:47:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:08 smithi116 ceph-mon[35411]: pgmap v5878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:47:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:10 smithi114 ceph-mon[30820]: pgmap v5879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:10 smithi116 ceph-mon[35411]: pgmap v5879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:12 smithi114 ceph-mon[30820]: pgmap v5880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:12 smithi116 ceph-mon[35411]: pgmap v5880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:14 smithi114 ceph-mon[30820]: pgmap v5881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:14 smithi116 ceph-mon[35411]: pgmap v5881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:16 smithi114 ceph-mon[30820]: pgmap v5882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:16 smithi116 ceph-mon[35411]: pgmap v5882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:18 smithi114 ceph-mon[30820]: pgmap v5883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:18 smithi116 ceph-mon[35411]: pgmap v5883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:20 smithi114 ceph-mon[30820]: pgmap v5884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:20 smithi116 ceph-mon[35411]: pgmap v5884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:22 smithi114 ceph-mon[30820]: pgmap v5885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:22 smithi116 ceph-mon[35411]: pgmap v5885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:24 smithi116 ceph-mon[35411]: pgmap v5886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:24 smithi114 ceph-mon[30820]: pgmap v5886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:26 smithi116 ceph-mon[35411]: pgmap v5887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:26 smithi114 ceph-mon[30820]: pgmap v5887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:28 smithi116 ceph-mon[35411]: pgmap v5888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:28 smithi114 ceph-mon[30820]: pgmap v5888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:30 smithi116 ceph-mon[35411]: pgmap v5889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:30 smithi114 ceph-mon[30820]: pgmap v5889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:32.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:32 smithi114 ceph-mon[30820]: pgmap v5890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:32 smithi116 ceph-mon[35411]: pgmap v5890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:34.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:34 smithi116 ceph-mon[35411]: pgmap v5891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:34 smithi114 ceph-mon[30820]: pgmap v5891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:36 smithi114 ceph-mon[30820]: pgmap v5892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:36 smithi116 ceph-mon[35411]: pgmap v5892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:38 smithi114 ceph-mon[30820]: pgmap v5893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:38 smithi116 ceph-mon[35411]: pgmap v5893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:40 smithi114 ceph-mon[30820]: pgmap v5894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:40 smithi116 ceph-mon[35411]: pgmap v5894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:42 smithi114 ceph-mon[30820]: pgmap v5895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:42 smithi116 ceph-mon[35411]: pgmap v5895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:44 smithi114 ceph-mon[30820]: pgmap v5896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:44 smithi116 ceph-mon[35411]: pgmap v5896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:46 smithi114 ceph-mon[30820]: pgmap v5897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:46 smithi116 ceph-mon[35411]: pgmap v5897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:48 smithi114 ceph-mon[30820]: pgmap v5898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:48 smithi116 ceph-mon[35411]: pgmap v5898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:47:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:50 smithi114 ceph-mon[30820]: pgmap v5899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:50 smithi116 ceph-mon[35411]: pgmap v5899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:47:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:52 smithi114 ceph-mon[30820]: pgmap v5900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:52 smithi116 ceph-mon[35411]: pgmap v5900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:47:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:54 smithi114 ceph-mon[30820]: pgmap v5901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:47:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:54 smithi116 ceph-mon[35411]: pgmap v5901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:47:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:56 smithi114 ceph-mon[30820]: pgmap v5902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:56 smithi116 ceph-mon[35411]: pgmap v5902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:47:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:47:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:47:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:47:58 smithi114 ceph-mon[30820]: pgmap v5903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:47:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:47:58 smithi116 ceph-mon[35411]: pgmap v5903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:48:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-mon[30820]: pgmap v5904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:48:00.917+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:48:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:48:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:48:01.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:48:01.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:48:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:48:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:00 smithi116 ceph-mon[35411]: pgmap v5904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:01 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:48:02.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:48:02.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:48:02.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:48:02.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:48:02.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:48:02.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:48:02.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:48:02.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:01 smithi114 ceph-mon[30820]: pgmap v5905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T09:48:02.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:01 smithi114 ceph-mon[30820]: pgmap v5906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:48:02.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:48:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:01 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:48:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:48:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:48:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:48:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:01 smithi116 ceph-mon[35411]: pgmap v5905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:01 smithi116 ceph-mon[35411]: pgmap v5906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:48:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:48:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:03 smithi114 ceph-mon[30820]: pgmap v5907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:48:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:03 smithi116 ceph-mon[35411]: pgmap v5907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:48:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:05 smithi114 ceph-mon[30820]: pgmap v5908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:48:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:05 smithi116 ceph-mon[35411]: pgmap v5908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:48:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:07 smithi114 ceph-mon[30820]: pgmap v5909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:48:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:07 smithi116 ceph-mon[35411]: pgmap v5909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:48:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:09 smithi114 ceph-mon[30820]: pgmap v5910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:48:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:09 smithi116 ceph-mon[35411]: pgmap v5910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:48:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:11 smithi114 ceph-mon[30820]: pgmap v5911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:48:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:11 smithi116 ceph-mon[35411]: pgmap v5911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:48:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:13 smithi114 ceph-mon[30820]: pgmap v5912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:13 smithi116 ceph-mon[35411]: pgmap v5912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:16 smithi114 ceph-mon[30820]: pgmap v5913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:48:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:16 smithi116 ceph-mon[35411]: pgmap v5913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:48:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:18 smithi114 ceph-mon[30820]: pgmap v5914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:18 smithi116 ceph-mon[35411]: pgmap v5914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:20 smithi114 ceph-mon[30820]: pgmap v5915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:20 smithi116 ceph-mon[35411]: pgmap v5915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:22 smithi114 ceph-mon[30820]: pgmap v5916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:48:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:22 smithi116 ceph-mon[35411]: pgmap v5916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:48:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:24 smithi114 ceph-mon[30820]: pgmap v5917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:24 smithi116 ceph-mon[35411]: pgmap v5917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:26 smithi114 ceph-mon[30820]: pgmap v5918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:48:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:26 smithi116 ceph-mon[35411]: pgmap v5918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:48:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:28 smithi114 ceph-mon[30820]: pgmap v5919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:28 smithi116 ceph-mon[35411]: pgmap v5919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:30 smithi114 ceph-mon[30820]: pgmap v5920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:30 smithi116 ceph-mon[35411]: pgmap v5920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:32 smithi114 ceph-mon[30820]: pgmap v5921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:48:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:32 smithi116 ceph-mon[35411]: pgmap v5921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:48:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:34 smithi114 ceph-mon[30820]: pgmap v5922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:34 smithi116 ceph-mon[35411]: pgmap v5922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:36 smithi114 ceph-mon[30820]: pgmap v5923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:48:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:36 smithi116 ceph-mon[35411]: pgmap v5923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:48:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:38 smithi114 ceph-mon[30820]: pgmap v5924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:38 smithi116 ceph-mon[35411]: pgmap v5924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:40 smithi114 ceph-mon[30820]: pgmap v5925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:40 smithi116 ceph-mon[35411]: pgmap v5925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:42 smithi114 ceph-mon[30820]: pgmap v5926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:48:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:42 smithi116 ceph-mon[35411]: pgmap v5926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:48:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:44 smithi114 ceph-mon[30820]: pgmap v5927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:44 smithi116 ceph-mon[35411]: pgmap v5927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:46 smithi114 ceph-mon[30820]: pgmap v5928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:48:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:46 smithi116 ceph-mon[35411]: pgmap v5928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:48:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:48 smithi114 ceph-mon[30820]: pgmap v5929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:48 smithi116 ceph-mon[35411]: pgmap v5929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:50 smithi114 ceph-mon[30820]: pgmap v5930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:50 smithi116 ceph-mon[35411]: pgmap v5930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:48:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:52 smithi114 ceph-mon[30820]: pgmap v5931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:48:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:52 smithi116 ceph-mon[35411]: pgmap v5931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:54 smithi114 ceph-mon[30820]: pgmap v5932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:54 smithi116 ceph-mon[35411]: pgmap v5932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:48:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:56 smithi114 ceph-mon[30820]: pgmap v5933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:48:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:56 smithi116 ceph-mon[35411]: pgmap v5933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:48:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:48:58 smithi114 ceph-mon[30820]: pgmap v5934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:48:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:48:58 smithi116 ceph-mon[35411]: pgmap v5934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:49:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:00 smithi114 ceph-mon[30820]: pgmap v5935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:49:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:00 smithi116 ceph-mon[35411]: pgmap v5935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:49:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:49:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:49:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:02 smithi114 ceph-mon[30820]: pgmap v5936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:49:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:02 smithi116 ceph-mon[35411]: pgmap v5936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:49:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-mon[30820]: pgmap v5937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:49:04.277+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:49:04.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:49:04.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:49:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:49:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:04 smithi116 ceph-mon[35411]: pgmap v5937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:05 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:49:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:49:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:49:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:05 smithi114 ceph-mon[30820]: pgmap v5938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:05 smithi114 ceph-mon[30820]: pgmap v5939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:49:05.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:49:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:05 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:49:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:49:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:49:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:49:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:49:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:49:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:05 smithi116 ceph-mon[35411]: pgmap v5938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:05 smithi116 ceph-mon[35411]: pgmap v5939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:49:05.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:49:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:07 smithi114 ceph-mon[30820]: pgmap v5940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:49:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:07 smithi116 ceph-mon[35411]: pgmap v5940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:49:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:09 smithi114 ceph-mon[30820]: pgmap v5941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:49:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:09 smithi116 ceph-mon[35411]: pgmap v5941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:49:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:11 smithi114 ceph-mon[30820]: pgmap v5942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:49:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:11 smithi116 ceph-mon[35411]: pgmap v5942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:49:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:12 smithi114 ceph-mon[30820]: pgmap v5943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:49:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:12 smithi116 ceph-mon[35411]: pgmap v5943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:49:15.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:15 smithi114 ceph-mon[30820]: pgmap v5944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:49:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:15 smithi116 ceph-mon[35411]: pgmap v5944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:49:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:17 smithi114 ceph-mon[30820]: pgmap v5945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:49:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:17 smithi116 ceph-mon[35411]: pgmap v5945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:49:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:19 smithi114 ceph-mon[30820]: pgmap v5946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:19 smithi116 ceph-mon[35411]: pgmap v5946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:20.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:20 smithi114 ceph-mon[30820]: pgmap v5947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:49:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:20 smithi116 ceph-mon[35411]: pgmap v5947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:49:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:23 smithi114 ceph-mon[30820]: pgmap v5948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:49:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:23 smithi116 ceph-mon[35411]: pgmap v5948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:49:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:25 smithi114 ceph-mon[30820]: pgmap v5949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:49:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:25 smithi116 ceph-mon[35411]: pgmap v5949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:49:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:26 smithi114 ceph-mon[30820]: pgmap v5950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:49:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:26 smithi116 ceph-mon[35411]: pgmap v5950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:49:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:29 smithi114 ceph-mon[30820]: pgmap v5951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:29 smithi116 ceph-mon[35411]: pgmap v5951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:30 smithi114 ceph-mon[30820]: pgmap v5952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:49:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:30 smithi116 ceph-mon[35411]: pgmap v5952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:49:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:33 smithi114 ceph-mon[30820]: pgmap v5953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:49:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:33 smithi116 ceph-mon[35411]: pgmap v5953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:49:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:34 smithi114 ceph-mon[30820]: pgmap v5954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:49:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:34 smithi116 ceph-mon[35411]: pgmap v5954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:49:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:37 smithi114 ceph-mon[30820]: pgmap v5955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:49:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:37 smithi116 ceph-mon[35411]: pgmap v5955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:49:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:38 smithi114 ceph-mon[30820]: pgmap v5956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:38 smithi116 ceph-mon[35411]: pgmap v5956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:40 smithi114 ceph-mon[30820]: pgmap v5957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:49:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:40 smithi116 ceph-mon[35411]: pgmap v5957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:49:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:42 smithi114 ceph-mon[30820]: pgmap v5958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:49:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:42 smithi116 ceph-mon[35411]: pgmap v5958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:49:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:44 smithi114 ceph-mon[30820]: pgmap v5959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:49:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:44 smithi116 ceph-mon[35411]: pgmap v5959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:49:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:46 smithi114 ceph-mon[30820]: pgmap v5960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:49:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:46 smithi116 ceph-mon[35411]: pgmap v5960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:49:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:48 smithi114 ceph-mon[30820]: pgmap v5961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:48 smithi116 ceph-mon[35411]: pgmap v5961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:50 smithi114 ceph-mon[30820]: pgmap v5962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:49:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:50 smithi116 ceph-mon[35411]: pgmap v5962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:49:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:52 smithi114 ceph-mon[30820]: pgmap v5963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:49:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:52 smithi116 ceph-mon[35411]: pgmap v5963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:49:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:54 smithi114 ceph-mon[30820]: pgmap v5964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:49:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:54 smithi116 ceph-mon[35411]: pgmap v5964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:49:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:56 smithi114 ceph-mon[30820]: pgmap v5965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:49:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:56 smithi116 ceph-mon[35411]: pgmap v5965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:49:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:49:58 smithi114 ceph-mon[30820]: pgmap v5966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:49:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:49:58 smithi116 ceph-mon[35411]: pgmap v5966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:50:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T09:50:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:01 smithi114 ceph-mon[30820]: pgmap v5967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:50:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:01 smithi116 ceph-mon[35411]: pgmap v5967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:50:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:02 smithi114 ceph-mon[30820]: pgmap v5968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:50:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:02 smithi116 ceph-mon[35411]: pgmap v5968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:50:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:04 smithi114 ceph-mon[30820]: pgmap v5969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:04 smithi116 ceph-mon[35411]: pgmap v5969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:05.662 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:50:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:50:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:06 smithi114 ceph-mon[30820]: pgmap v5970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:06 smithi116 ceph-mon[35411]: pgmap v5970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:50:07.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:50:07.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:07.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:50:07.636+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:50:07.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:50:07.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:50:07.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:50:07.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:50:07.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:50:07.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:50:07.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:50:07.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:50:07.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:50:07.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:50:07.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:50:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:50:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:50:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:50:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:50:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:08 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:50:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:50:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:50:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:08 smithi114 ceph-mon[30820]: pgmap v5971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:50:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:08 smithi114 ceph-mon[30820]: pgmap v5972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:50:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:08 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:50:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:50:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:50:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:50:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:50:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:50:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:50:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:50:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:50:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:50:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:50:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:08 smithi116 ceph-mon[35411]: pgmap v5971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:50:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:08 smithi116 ceph-mon[35411]: pgmap v5972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:50:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:10 smithi114 ceph-mon[30820]: pgmap v5973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:50:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:10 smithi116 ceph-mon[35411]: pgmap v5973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:50:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:12 smithi114 ceph-mon[30820]: pgmap v5974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:50:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:12 smithi116 ceph-mon[35411]: pgmap v5974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:50:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:14 smithi114 ceph-mon[30820]: pgmap v5975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:50:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:14 smithi116 ceph-mon[35411]: pgmap v5975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:50:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:16 smithi114 ceph-mon[30820]: pgmap v5976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:50:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:16 smithi116 ceph-mon[35411]: pgmap v5976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:50:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:18 smithi114 ceph-mon[30820]: pgmap v5977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:50:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:18 smithi116 ceph-mon[35411]: pgmap v5977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:50:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:20 smithi114 ceph-mon[30820]: pgmap v5978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:20 smithi116 ceph-mon[35411]: pgmap v5978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:22 smithi114 ceph-mon[30820]: pgmap v5979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:22 smithi116 ceph-mon[35411]: pgmap v5979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:24 smithi114 ceph-mon[30820]: pgmap v5980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:24 smithi116 ceph-mon[35411]: pgmap v5980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:26 smithi114 ceph-mon[30820]: pgmap v5981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:50:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:26 smithi116 ceph-mon[35411]: pgmap v5981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:50:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:28 smithi114 ceph-mon[30820]: pgmap v5982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:28 smithi116 ceph-mon[35411]: pgmap v5982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:30 smithi114 ceph-mon[30820]: pgmap v5983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:30 smithi116 ceph-mon[35411]: pgmap v5983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:32 smithi114 ceph-mon[30820]: pgmap v5984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:32 smithi116 ceph-mon[35411]: pgmap v5984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:34 smithi114 ceph-mon[30820]: pgmap v5985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:34 smithi116 ceph-mon[35411]: pgmap v5985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:37.130 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:36 smithi114 ceph-mon[30820]: pgmap v5986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:50:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:36 smithi116 ceph-mon[35411]: pgmap v5986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:50:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:38 smithi114 ceph-mon[30820]: pgmap v5987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:38 smithi116 ceph-mon[35411]: pgmap v5987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:40 smithi114 ceph-mon[30820]: pgmap v5988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:40 smithi116 ceph-mon[35411]: pgmap v5988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:42 smithi114 ceph-mon[30820]: pgmap v5989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:42 smithi116 ceph-mon[35411]: pgmap v5989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:44 smithi114 ceph-mon[30820]: pgmap v5990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:44 smithi116 ceph-mon[35411]: pgmap v5990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:46 smithi114 ceph-mon[30820]: pgmap v5991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:50:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:46 smithi116 ceph-mon[35411]: pgmap v5991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:50:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:48 smithi114 ceph-mon[30820]: pgmap v5992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:48 smithi116 ceph-mon[35411]: pgmap v5992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:50 smithi116 ceph-mon[35411]: pgmap v5993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:50 smithi114 ceph-mon[30820]: pgmap v5993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:50:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:52 smithi116 ceph-mon[35411]: pgmap v5994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:52 smithi114 ceph-mon[30820]: pgmap v5994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:50:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:54 smithi116 ceph-mon[35411]: pgmap v5995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:54 smithi114 ceph-mon[30820]: pgmap v5995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:56 smithi114 ceph-mon[30820]: pgmap v5996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:50:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:56 smithi116 ceph-mon[35411]: pgmap v5996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:50:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:50:58 smithi114 ceph-mon[30820]: pgmap v5997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:50:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:50:58 smithi116 ceph-mon[35411]: pgmap v5997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:00 smithi114 ceph-mon[30820]: pgmap v5998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:00 smithi116 ceph-mon[35411]: pgmap v5998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:02 smithi114 ceph-mon[30820]: pgmap v5999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:02 smithi116 ceph-mon[35411]: pgmap v5999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:05.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:04 smithi114 ceph-mon[30820]: pgmap v6000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:04 smithi116 ceph-mon[35411]: pgmap v6000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:06 smithi114 ceph-mon[30820]: pgmap v6001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:51:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:06 smithi116 ceph-mon[35411]: pgmap v6001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:51:08.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:51:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:51:09.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:09 smithi116 ceph-mon[35411]: pgmap v6002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:09.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:09.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:09.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:09.163 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:09 smithi114 ceph-mon[30820]: pgmap v6002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:11 smithi116 ceph-mon[35411]: pgmap v6003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:11.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:11 smithi114 ceph-mon[30820]: pgmap v6003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:11.264 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:51:10.997+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:51:11.265 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:51:11.266 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:51:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:51:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:12 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:51:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:51:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:12 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:51:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:51:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:51:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:51:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:51:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:51:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:51:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:51:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:51:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:13 smithi114 ceph-mon[30820]: pgmap v6004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:13 smithi116 ceph-mon[35411]: pgmap v6004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:51:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:51:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:14 smithi114 ceph-mon[30820]: pgmap v6005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:51:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:14 smithi114 ceph-mon[30820]: pgmap v6006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:51:14.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:51:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:51:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:14 smithi116 ceph-mon[35411]: pgmap v6005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:51:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:14 smithi116 ceph-mon[35411]: pgmap v6006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:51:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:51:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:16 smithi114 ceph-mon[30820]: pgmap v6007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:51:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:16 smithi116 ceph-mon[35411]: pgmap v6007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:51:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:17 smithi116 ceph-mon[35411]: pgmap v6008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:51:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:17 smithi114 ceph-mon[30820]: pgmap v6008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:51:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:20 smithi114 ceph-mon[30820]: pgmap v6009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:51:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:20 smithi116 ceph-mon[35411]: pgmap v6009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:51:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:21 smithi116 ceph-mon[35411]: pgmap v6010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:51:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:21 smithi114 ceph-mon[30820]: pgmap v6010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T09:51:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:24 smithi114 ceph-mon[30820]: pgmap v6011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:51:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:24 smithi116 ceph-mon[35411]: pgmap v6011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:51:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:25 smithi114 ceph-mon[30820]: pgmap v6012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:51:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:25 smithi116 ceph-mon[35411]: pgmap v6012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:51:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:27 smithi116 ceph-mon[35411]: pgmap v6013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:27 smithi114 ceph-mon[30820]: pgmap v6013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:51:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:29 smithi116 ceph-mon[35411]: pgmap v6014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:29 smithi114 ceph-mon[30820]: pgmap v6014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:31 smithi116 ceph-mon[35411]: pgmap v6015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:31 smithi114 ceph-mon[30820]: pgmap v6015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:33 smithi116 ceph-mon[35411]: pgmap v6016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:33 smithi114 ceph-mon[30820]: pgmap v6016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:35 smithi116 ceph-mon[35411]: pgmap v6017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:51:35.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:35 smithi114 ceph-mon[30820]: pgmap v6017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:51:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:37 smithi116 ceph-mon[35411]: pgmap v6018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:51:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:37 smithi114 ceph-mon[30820]: pgmap v6018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:51:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:39 smithi116 ceph-mon[35411]: pgmap v6019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:39 smithi114 ceph-mon[30820]: pgmap v6019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:41 smithi116 ceph-mon[35411]: pgmap v6020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:41 smithi114 ceph-mon[30820]: pgmap v6020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:43 smithi116 ceph-mon[35411]: pgmap v6021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:43 smithi114 ceph-mon[30820]: pgmap v6021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:45 smithi116 ceph-mon[35411]: pgmap v6022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:51:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:45 smithi114 ceph-mon[30820]: pgmap v6022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:51:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:47 smithi116 ceph-mon[35411]: pgmap v6023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:51:47.783 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:47 smithi114 ceph-mon[30820]: pgmap v6023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:51:49.684 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:49 smithi116 ceph-mon[35411]: pgmap v6024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:49.927 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:49 smithi114 ceph-mon[30820]: pgmap v6024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:51 smithi116 ceph-mon[35411]: pgmap v6025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:51 smithi114 ceph-mon[30820]: pgmap v6025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:51:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:53 smithi116 ceph-mon[35411]: pgmap v6026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:53 smithi114 ceph-mon[30820]: pgmap v6026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:51:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:55 smithi116 ceph-mon[35411]: pgmap v6027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:51:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:55 smithi114 ceph-mon[30820]: pgmap v6027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:51:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:57 smithi116 ceph-mon[35411]: pgmap v6028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:51:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:57 smithi114 ceph-mon[30820]: pgmap v6028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:51:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:51:59 smithi116 ceph-mon[35411]: pgmap v6029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:51:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:51:59 smithi114 ceph-mon[30820]: pgmap v6029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:01 smithi116 ceph-mon[35411]: pgmap v6030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:01 smithi114 ceph-mon[30820]: pgmap v6030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:03 smithi116 ceph-mon[35411]: pgmap v6031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:03 smithi114 ceph-mon[30820]: pgmap v6031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:05 smithi116 ceph-mon[35411]: pgmap v6032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:52:05.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:05 smithi114 ceph-mon[30820]: pgmap v6032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:52:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:07 smithi116 ceph-mon[35411]: pgmap v6033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:52:07.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:07 smithi114 ceph-mon[30820]: pgmap v6033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:52:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:09 smithi116 ceph-mon[35411]: pgmap v6034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:09 smithi114 ceph-mon[30820]: pgmap v6034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:11 smithi114 ceph-mon[30820]: pgmap v6035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:11 smithi116 ceph-mon[35411]: pgmap v6035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:13 smithi114 ceph-mon[30820]: pgmap v6036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:13 smithi116 ceph-mon[35411]: pgmap v6036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:52:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:52:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:15 smithi114 ceph-mon[30820]: pgmap v6037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:52:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:15 smithi116 ceph-mon[35411]: pgmap v6037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:52:16.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:52:16.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:52:16.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:52:16.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:52:16.661+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:52:16.923 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:52:16.923 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:52:16.923 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:52:16.923 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:52:16.923 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:52:16.923 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:52:16.923 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:52:16.924 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:52:16.925 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:52:16.925 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:52:16.925 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:52:16.925 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:52:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:52:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:52:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:52:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:52:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:52:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:52:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:52:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:52:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:52:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:52:18.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:52:18.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:52:18.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:52:18.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:52:18.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:52:18.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:52:18.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:52:18.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:52:18.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:52:18.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:52:18.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:52:18.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:52:18.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:52:18.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:52:18.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:52:18.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:52:18.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:52:18.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:17 smithi114 ceph-mon[30820]: pgmap v6038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T09:52:18.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:17 smithi114 ceph-mon[30820]: pgmap v6039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:52:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:52:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:52:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:52:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:52:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:52:18.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:52:18.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:52:18.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:52:18.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:52:18.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:52:18.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:52:18.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:52:18.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:17 smithi116 ceph-mon[35411]: pgmap v6038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T09:52:18.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:17 smithi116 ceph-mon[35411]: pgmap v6039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:52:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:19 smithi114 ceph-mon[30820]: pgmap v6040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:52:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:19 smithi116 ceph-mon[35411]: pgmap v6040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:52:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:21 smithi114 ceph-mon[30820]: pgmap v6041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:52:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:21 smithi116 ceph-mon[35411]: pgmap v6041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:52:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:23 smithi114 ceph-mon[30820]: pgmap v6042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:52:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:23 smithi116 ceph-mon[35411]: pgmap v6042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:52:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:25 smithi114 ceph-mon[30820]: pgmap v6043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:52:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:25 smithi116 ceph-mon[35411]: pgmap v6043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:52:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:27 smithi114 ceph-mon[30820]: pgmap v6044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:52:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:27 smithi116 ceph-mon[35411]: pgmap v6044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:52:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:29 smithi114 ceph-mon[30820]: pgmap v6045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:29 smithi116 ceph-mon[35411]: pgmap v6045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:31 smithi114 ceph-mon[30820]: pgmap v6046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:52:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:31 smithi116 ceph-mon[35411]: pgmap v6046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:52:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:33 smithi114 ceph-mon[30820]: pgmap v6047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:33 smithi116 ceph-mon[35411]: pgmap v6047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:35 smithi114 ceph-mon[30820]: pgmap v6048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:35 smithi116 ceph-mon[35411]: pgmap v6048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:37 smithi114 ceph-mon[30820]: pgmap v6049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:38.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:37 smithi116 ceph-mon[35411]: pgmap v6049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:39 smithi114 ceph-mon[30820]: pgmap v6050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:39 smithi116 ceph-mon[35411]: pgmap v6050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:41 smithi114 ceph-mon[30820]: pgmap v6051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:52:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:41 smithi116 ceph-mon[35411]: pgmap v6051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:52:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:43 smithi114 ceph-mon[30820]: pgmap v6052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:43 smithi116 ceph-mon[35411]: pgmap v6052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:45 smithi114 ceph-mon[30820]: pgmap v6053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:45 smithi116 ceph-mon[35411]: pgmap v6053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:48.045 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:47 smithi114 ceph-mon[30820]: pgmap v6054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:47 smithi116 ceph-mon[35411]: pgmap v6054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:49 smithi114 ceph-mon[30820]: pgmap v6055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:49 smithi116 ceph-mon[35411]: pgmap v6055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:51 smithi114 ceph-mon[30820]: pgmap v6056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:52:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:51 smithi116 ceph-mon[35411]: pgmap v6056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:52:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:53 smithi114 ceph-mon[30820]: pgmap v6057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:53 smithi116 ceph-mon[35411]: pgmap v6057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:52:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:55 smithi114 ceph-mon[30820]: pgmap v6058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:55 smithi116 ceph-mon[35411]: pgmap v6058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:52:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:57 smithi114 ceph-mon[30820]: pgmap v6059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:52:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:57 smithi116 ceph-mon[35411]: pgmap v6059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:52:59 smithi114 ceph-mon[30820]: pgmap v6060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:52:59 smithi116 ceph-mon[35411]: pgmap v6060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:01 smithi114 ceph-mon[30820]: pgmap v6061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:01 smithi116 ceph-mon[35411]: pgmap v6061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:03 smithi114 ceph-mon[30820]: pgmap v6062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:03 smithi116 ceph-mon[35411]: pgmap v6062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:05 smithi114 ceph-mon[30820]: pgmap v6063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:53:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:05 smithi116 ceph-mon[35411]: pgmap v6063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:53:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:07 smithi114 ceph-mon[30820]: pgmap v6064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:07 smithi116 ceph-mon[35411]: pgmap v6064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:09 smithi114 ceph-mon[30820]: pgmap v6065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:09 smithi116 ceph-mon[35411]: pgmap v6065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:11 smithi114 ceph-mon[30820]: pgmap v6066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:11 smithi116 ceph-mon[35411]: pgmap v6066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:13 smithi114 ceph-mon[30820]: pgmap v6067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:13 smithi116 ceph-mon[35411]: pgmap v6067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:15 smithi114 ceph-mon[30820]: pgmap v6068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:53:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:15 smithi116 ceph-mon[35411]: pgmap v6068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:53:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:53:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:53:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-mon[30820]: pgmap v6069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:53:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:53:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:53:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:53:17.780+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:53:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:53:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:53:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:53:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:53:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:53:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:53:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:53:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:53:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:53:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:53:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:53:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:17 smithi116 ceph-mon[35411]: pgmap v6069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:53:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:53:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:53:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:18 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:18 smithi114 ceph-mon[30820]: pgmap v6070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:53:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:18 smithi114 ceph-mon[30820]: pgmap v6071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T09:53:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:18 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:53:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:53:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:18 smithi116 ceph-mon[35411]: pgmap v6070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:53:19.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:18 smithi116 ceph-mon[35411]: pgmap v6071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T09:53:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:20 smithi114 ceph-mon[30820]: pgmap v6072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:53:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:20 smithi116 ceph-mon[35411]: pgmap v6072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:53:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:22 smithi114 ceph-mon[30820]: pgmap v6073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T09:53:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:22 smithi116 ceph-mon[35411]: pgmap v6073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T09:53:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:24 smithi114 ceph-mon[30820]: pgmap v6074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T09:53:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:24 smithi116 ceph-mon[35411]: pgmap v6074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T09:53:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:26 smithi114 ceph-mon[30820]: pgmap v6075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T09:53:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:26 smithi116 ceph-mon[35411]: pgmap v6075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T09:53:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:28 smithi114 ceph-mon[30820]: pgmap v6076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:53:29.184 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:28 smithi116 ceph-mon[35411]: pgmap v6076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:53:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:30 smithi114 ceph-mon[30820]: pgmap v6077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:53:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:30 smithi116 ceph-mon[35411]: pgmap v6077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:53:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:32 smithi114 ceph-mon[30820]: pgmap v6078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:32 smithi116 ceph-mon[35411]: pgmap v6078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:34 smithi114 ceph-mon[30820]: pgmap v6079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:34 smithi116 ceph-mon[35411]: pgmap v6079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:36 smithi114 ceph-mon[30820]: pgmap v6080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:36 smithi116 ceph-mon[35411]: pgmap v6080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:38 smithi114 ceph-mon[30820]: pgmap v6081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:53:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:38 smithi116 ceph-mon[35411]: pgmap v6081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:53:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:40 smithi114 ceph-mon[30820]: pgmap v6082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:53:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:40 smithi116 ceph-mon[35411]: pgmap v6082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T09:53:43.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:42 smithi114 ceph-mon[30820]: pgmap v6083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:42 smithi116 ceph-mon[35411]: pgmap v6083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:44 smithi114 ceph-mon[30820]: pgmap v6084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:53:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:44 smithi116 ceph-mon[35411]: pgmap v6084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:53:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:46 smithi114 ceph-mon[30820]: pgmap v6085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:46 smithi116 ceph-mon[35411]: pgmap v6085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:48 smithi114 ceph-mon[30820]: pgmap v6086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:48 smithi116 ceph-mon[35411]: pgmap v6086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:50 smithi114 ceph-mon[30820]: pgmap v6087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:53:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:50 smithi116 ceph-mon[35411]: pgmap v6087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:53:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:52 smithi114 ceph-mon[30820]: pgmap v6088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:52 smithi116 ceph-mon[35411]: pgmap v6088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:53:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:54 smithi114 ceph-mon[30820]: pgmap v6089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:54 smithi116 ceph-mon[35411]: pgmap v6089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:56 smithi114 ceph-mon[30820]: pgmap v6090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:56 smithi116 ceph-mon[35411]: pgmap v6090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:53:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:53:58 smithi114 ceph-mon[30820]: pgmap v6091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:53:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:53:58 smithi116 ceph-mon[35411]: pgmap v6091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:00 smithi114 ceph-mon[30820]: pgmap v6092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:00 smithi116 ceph-mon[35411]: pgmap v6092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:02 smithi114 ceph-mon[30820]: pgmap v6093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:02 smithi116 ceph-mon[35411]: pgmap v6093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:04 smithi114 ceph-mon[30820]: pgmap v6094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:04 smithi116 ceph-mon[35411]: pgmap v6094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:06 smithi114 ceph-mon[30820]: pgmap v6095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:54:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:06 smithi116 ceph-mon[35411]: pgmap v6095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:54:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:08 smithi116 ceph-mon[35411]: pgmap v6096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:08 smithi114 ceph-mon[30820]: pgmap v6096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:10 smithi116 ceph-mon[35411]: pgmap v6097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:10 smithi114 ceph-mon[30820]: pgmap v6097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:12 smithi116 ceph-mon[35411]: pgmap v6098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:12 smithi114 ceph-mon[30820]: pgmap v6098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:14 smithi114 ceph-mon[30820]: pgmap v6099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:14 smithi116 ceph-mon[35411]: pgmap v6099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:16 smithi114 ceph-mon[30820]: pgmap v6100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:54:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:16 smithi116 ceph-mon[35411]: pgmap v6100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:54:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:54:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:54:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:18 smithi114 ceph-mon[30820]: pgmap v6101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:18 smithi116 ceph-mon[35411]: pgmap v6101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:20 smithi114 ceph-mon[30820]: pgmap v6102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:54:21.140+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:54:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:54:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:54:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:54:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:54:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:54:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:54:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:54:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:54:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:54:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:54:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:20 smithi116 ceph-mon[35411]: pgmap v6102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:54:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:54:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:54:22.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:54:22.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:54:22.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:54:22.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-mon[30820]: pgmap v6103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:54:22.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-mon[30820]: pgmap v6104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:54:22.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:54:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:54:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:54:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:54:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:21 smithi116 ceph-mon[35411]: pgmap v6103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:21 smithi116 ceph-mon[35411]: pgmap v6104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:54:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:54:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:24 smithi114 ceph-mon[30820]: pgmap v6105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:54:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:24 smithi116 ceph-mon[35411]: pgmap v6105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:54:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:26 smithi114 ceph-mon[30820]: pgmap v6106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:54:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:26 smithi116 ceph-mon[35411]: pgmap v6106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:54:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:28 smithi114 ceph-mon[30820]: pgmap v6107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:54:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:28 smithi116 ceph-mon[35411]: pgmap v6107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:54:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:30 smithi114 ceph-mon[30820]: pgmap v6108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:54:30.717 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:30 smithi116 ceph-mon[35411]: pgmap v6108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:54:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:32 smithi114 ceph-mon[30820]: pgmap v6109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:54:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:32 smithi116 ceph-mon[35411]: pgmap v6109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:54:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:34 smithi114 ceph-mon[30820]: pgmap v6110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:34 smithi116 ceph-mon[35411]: pgmap v6110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:36 smithi114 ceph-mon[30820]: pgmap v6111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:36 smithi116 ceph-mon[35411]: pgmap v6111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:38 smithi114 ceph-mon[30820]: pgmap v6112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:38 smithi116 ceph-mon[35411]: pgmap v6112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:40.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:40 smithi114 ceph-mon[30820]: pgmap v6113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:40 smithi116 ceph-mon[35411]: pgmap v6113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:42 smithi114 ceph-mon[30820]: pgmap v6114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:54:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:42 smithi116 ceph-mon[35411]: pgmap v6114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:54:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:44 smithi114 ceph-mon[30820]: pgmap v6115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:44 smithi116 ceph-mon[35411]: pgmap v6115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:46 smithi114 ceph-mon[30820]: pgmap v6116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:46 smithi116 ceph-mon[35411]: pgmap v6116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:48 smithi114 ceph-mon[30820]: pgmap v6117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:48 smithi116 ceph-mon[35411]: pgmap v6117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:50 smithi114 ceph-mon[30820]: pgmap v6118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:50 smithi116 ceph-mon[35411]: pgmap v6118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:52 smithi114 ceph-mon[30820]: pgmap v6119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:54:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:52 smithi116 ceph-mon[35411]: pgmap v6119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:54:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:54 smithi114 ceph-mon[30820]: pgmap v6120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:54 smithi116 ceph-mon[35411]: pgmap v6120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:54:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:56 smithi114 ceph-mon[30820]: pgmap v6121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:56 smithi116 ceph-mon[35411]: pgmap v6121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:54:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:54:58 smithi114 ceph-mon[30820]: pgmap v6122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:54:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:54:58 smithi116 ceph-mon[35411]: pgmap v6122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:00 smithi114 ceph-mon[30820]: pgmap v6123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:00 smithi116 ceph-mon[35411]: pgmap v6123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:02 smithi114 ceph-mon[30820]: pgmap v6124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:02 smithi116 ceph-mon[35411]: pgmap v6124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:04 smithi114 ceph-mon[30820]: pgmap v6125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:04 smithi116 ceph-mon[35411]: pgmap v6125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:06 smithi114 ceph-mon[30820]: pgmap v6126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:55:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:06 smithi116 ceph-mon[35411]: pgmap v6126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:55:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:08 smithi114 ceph-mon[30820]: pgmap v6127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:08 smithi116 ceph-mon[35411]: pgmap v6127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:10 smithi114 ceph-mon[30820]: pgmap v6128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:10 smithi116 ceph-mon[35411]: pgmap v6128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:12 smithi114 ceph-mon[30820]: pgmap v6129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:12 smithi116 ceph-mon[35411]: pgmap v6129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:14 smithi114 ceph-mon[30820]: pgmap v6130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:14 smithi116 ceph-mon[35411]: pgmap v6130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:16 smithi114 ceph-mon[30820]: pgmap v6131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:55:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:16 smithi116 ceph-mon[35411]: pgmap v6131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:55:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:18 smithi114 ceph-mon[30820]: pgmap v6132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:18 smithi116 ceph-mon[35411]: pgmap v6132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:20.312 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:20 smithi116 ceph-mon[35411]: pgmap v6133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:20 smithi114 ceph-mon[30820]: pgmap v6133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:55:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:55:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:22 smithi114 ceph-mon[30820]: pgmap v6134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:22 smithi116 ceph-mon[35411]: pgmap v6134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-mon[30820]: pgmap v6135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:55:24.501+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:55:24.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:55:24.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:55:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:55:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:24 smithi116 ceph-mon[35411]: pgmap v6135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:55:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:55:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:55:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:55:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:55:25.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:55:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:55:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:26 smithi114 ceph-mon[30820]: pgmap v6136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T09:55:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:26 smithi114 ceph-mon[30820]: pgmap v6137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:55:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:26 smithi116 ceph-mon[35411]: pgmap v6136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T09:55:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:26 smithi116 ceph-mon[35411]: pgmap v6137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:28 smithi114 ceph-mon[30820]: pgmap v6138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:28 smithi116 ceph-mon[35411]: pgmap v6138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:30.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:30 smithi114 ceph-mon[30820]: pgmap v6139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:30 smithi116 ceph-mon[35411]: pgmap v6139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:32 smithi114 ceph-mon[30820]: pgmap v6140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:32 smithi116 ceph-mon[35411]: pgmap v6140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:34 smithi114 ceph-mon[30820]: pgmap v6141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:34 smithi116 ceph-mon[35411]: pgmap v6141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:55:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:36 smithi114 ceph-mon[30820]: pgmap v6142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:55:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:36 smithi116 ceph-mon[35411]: pgmap v6142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:55:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:38 smithi114 ceph-mon[30820]: pgmap v6143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:38 smithi116 ceph-mon[35411]: pgmap v6143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:40 smithi114 ceph-mon[30820]: pgmap v6144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:40 smithi116 ceph-mon[35411]: pgmap v6144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:42 smithi114 ceph-mon[30820]: pgmap v6145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:42 smithi116 ceph-mon[35411]: pgmap v6145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:44 smithi114 ceph-mon[30820]: pgmap v6146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:44 smithi116 ceph-mon[35411]: pgmap v6146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:46 smithi114 ceph-mon[30820]: pgmap v6147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:55:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:46 smithi116 ceph-mon[35411]: pgmap v6147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:55:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:48 smithi114 ceph-mon[30820]: pgmap v6148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:48 smithi116 ceph-mon[35411]: pgmap v6148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:50 smithi114 ceph-mon[30820]: pgmap v6149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:50 smithi116 ceph-mon[35411]: pgmap v6149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:52 smithi114 ceph-mon[30820]: pgmap v6150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:52 smithi116 ceph-mon[35411]: pgmap v6150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:55:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:54 smithi114 ceph-mon[30820]: pgmap v6151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:54 smithi116 ceph-mon[35411]: pgmap v6151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:55:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:56 smithi114 ceph-mon[30820]: pgmap v6152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:55:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:56 smithi116 ceph-mon[35411]: pgmap v6152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:55:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:55:58 smithi114 ceph-mon[30820]: pgmap v6153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:55:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:55:58 smithi116 ceph-mon[35411]: pgmap v6153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:00 smithi114 ceph-mon[30820]: pgmap v6154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:00 smithi116 ceph-mon[35411]: pgmap v6154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:02 smithi114 ceph-mon[30820]: pgmap v6155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:02 smithi116 ceph-mon[35411]: pgmap v6155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:04 smithi114 ceph-mon[30820]: pgmap v6156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:04 smithi116 ceph-mon[35411]: pgmap v6156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:06 smithi114 ceph-mon[30820]: pgmap v6157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:06 smithi116 ceph-mon[35411]: pgmap v6157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:08 smithi114 ceph-mon[30820]: pgmap v6158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:08 smithi116 ceph-mon[35411]: pgmap v6158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:10 smithi114 ceph-mon[30820]: pgmap v6159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:10 smithi116 ceph-mon[35411]: pgmap v6159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:12 smithi116 ceph-mon[35411]: pgmap v6160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:12 smithi114 ceph-mon[30820]: pgmap v6160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:14 smithi116 ceph-mon[35411]: pgmap v6161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:14 smithi114 ceph-mon[30820]: pgmap v6161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:15 smithi116 ceph-mon[35411]: pgmap v6162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:15 smithi114 ceph-mon[30820]: pgmap v6162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:17 smithi114 ceph-mon[30820]: pgmap v6163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:17 smithi116 ceph-mon[35411]: pgmap v6163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:19 smithi114 ceph-mon[30820]: pgmap v6164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:19 smithi116 ceph-mon[35411]: pgmap v6164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:21 smithi114 ceph-mon[30820]: pgmap v6165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:21 smithi116 ceph-mon[35411]: pgmap v6165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:23 smithi114 ceph-mon[30820]: pgmap v6166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:23 smithi116 ceph-mon[35411]: pgmap v6166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:25 smithi114 ceph-mon[30820]: pgmap v6167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:25.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:56:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:25 smithi116 ceph-mon[35411]: pgmap v6167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:56:27.863 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-mon[30820]: pgmap v6168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:27 smithi116 ceph-mon[35411]: pgmap v6168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:56:27.860+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:56:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:56:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:56:28.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:56:28.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:56:28.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:56:28.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:56:28.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:56:28.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:56:28.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:56:28.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:56:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:56:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:56:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:56:28.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:56:28.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:56:28.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:28 smithi114 ceph-mon[30820]: pgmap v6169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:56:28.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:28 smithi114 ceph-mon[30820]: pgmap v6170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:56:28.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:56:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:56:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:56:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:56:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:28 smithi116 ceph-mon[35411]: pgmap v6169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:28 smithi116 ceph-mon[35411]: pgmap v6170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:56:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:56:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:30 smithi116 ceph-mon[35411]: pgmap v6171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:56:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:30 smithi114 ceph-mon[30820]: pgmap v6171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:56:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:32 smithi116 ceph-mon[35411]: pgmap v6172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:56:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:32 smithi114 ceph-mon[30820]: pgmap v6172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:56:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:34 smithi114 ceph-mon[30820]: pgmap v6173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:56:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:34 smithi116 ceph-mon[35411]: pgmap v6173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:56:37.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:36 smithi114 ceph-mon[30820]: pgmap v6174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:56:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:36 smithi116 ceph-mon[35411]: pgmap v6174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:56:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:38 smithi114 ceph-mon[30820]: pgmap v6175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:56:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:38 smithi116 ceph-mon[35411]: pgmap v6175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:56:41.232 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:40 smithi116 ceph-mon[35411]: pgmap v6176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:40 smithi114 ceph-mon[30820]: pgmap v6176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:42 smithi114 ceph-mon[30820]: pgmap v6177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:42 smithi116 ceph-mon[35411]: pgmap v6177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:44 smithi114 ceph-mon[30820]: pgmap v6178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:44 smithi116 ceph-mon[35411]: pgmap v6178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:46 smithi114 ceph-mon[30820]: pgmap v6179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:46 smithi116 ceph-mon[35411]: pgmap v6179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:48 smithi114 ceph-mon[30820]: pgmap v6180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:49.444 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:48 smithi116 ceph-mon[35411]: pgmap v6180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:50 smithi114 ceph-mon[30820]: pgmap v6181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:50 smithi116 ceph-mon[35411]: pgmap v6181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:56:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:52 smithi114 ceph-mon[30820]: pgmap v6182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:52 smithi116 ceph-mon[35411]: pgmap v6182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:56:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:54 smithi114 ceph-mon[30820]: pgmap v6183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:54 smithi116 ceph-mon[35411]: pgmap v6183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:56 smithi114 ceph-mon[30820]: pgmap v6184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:56 smithi116 ceph-mon[35411]: pgmap v6184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:56:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:56:58 smithi114 ceph-mon[30820]: pgmap v6185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:56:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:56:58 smithi116 ceph-mon[35411]: pgmap v6185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:00 smithi114 ceph-mon[30820]: pgmap v6186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:57:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:00 smithi116 ceph-mon[35411]: pgmap v6186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:57:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:02 smithi114 ceph-mon[30820]: pgmap v6187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:57:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:02 smithi116 ceph-mon[35411]: pgmap v6187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:57:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:04 smithi114 ceph-mon[30820]: pgmap v6188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:04 smithi116 ceph-mon[35411]: pgmap v6188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:07.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:06 smithi114 ceph-mon[30820]: pgmap v6189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:57:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:06 smithi116 ceph-mon[35411]: pgmap v6189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:57:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:08 smithi114 ceph-mon[30820]: pgmap v6190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:08 smithi116 ceph-mon[35411]: pgmap v6190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:11.232 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:10 smithi116 ceph-mon[35411]: pgmap v6191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:57:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:10 smithi114 ceph-mon[30820]: pgmap v6191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:57:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:12 smithi114 ceph-mon[30820]: pgmap v6192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:57:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:12 smithi116 ceph-mon[35411]: pgmap v6192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:57:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:14 smithi114 ceph-mon[30820]: pgmap v6193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:14 smithi116 ceph-mon[35411]: pgmap v6193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:16 smithi114 ceph-mon[30820]: pgmap v6194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:57:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:16 smithi116 ceph-mon[35411]: pgmap v6194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:57:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:18 smithi114 ceph-mon[30820]: pgmap v6195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:18 smithi116 ceph-mon[35411]: pgmap v6195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:20 smithi114 ceph-mon[30820]: pgmap v6196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:57:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:20 smithi116 ceph-mon[35411]: pgmap v6196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:57:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:22 smithi114 ceph-mon[30820]: pgmap v6197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:57:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:22 smithi116 ceph-mon[35411]: pgmap v6197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:57:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:24 smithi114 ceph-mon[30820]: pgmap v6198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:24 smithi116 ceph-mon[35411]: pgmap v6198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:26 smithi114 ceph-mon[30820]: pgmap v6199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:57:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:26 smithi116 ceph-mon[35411]: pgmap v6199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:57:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: pgmap v6200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:57:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:57:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:57:29.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: pgmap v6200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T09:57:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:29 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:57:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:29 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:57:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:29 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T09:57:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:29 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T09:57:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:30 smithi114 ceph-mon[30820]: pgmap v6201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:57:31.222+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:57:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:57:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:57:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:30 smithi116 ceph-mon[35411]: pgmap v6201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:57:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:57:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:57:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:57:32.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-mon[30820]: pgmap v6202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:57:32.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-mon[30820]: pgmap v6203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:57:32.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:31 smithi116 ceph-mon[35411]: pgmap v6202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:31 smithi116 ceph-mon[35411]: pgmap v6203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:57:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:57:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:34 smithi114 ceph-mon[30820]: pgmap v6204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:57:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:34 smithi116 ceph-mon[35411]: pgmap v6204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:57:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:36 smithi114 ceph-mon[30820]: pgmap v6205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:57:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:36 smithi116 ceph-mon[35411]: pgmap v6205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T09:57:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:38 smithi114 ceph-mon[30820]: pgmap v6206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:57:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:38 smithi116 ceph-mon[35411]: pgmap v6206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:57:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:40 smithi114 ceph-mon[30820]: pgmap v6207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:57:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:40 smithi116 ceph-mon[35411]: pgmap v6207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:57:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:42 smithi114 ceph-mon[30820]: pgmap v6208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:57:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:42 smithi116 ceph-mon[35411]: pgmap v6208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:57:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:44 smithi114 ceph-mon[30820]: pgmap v6209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:44 smithi116 ceph-mon[35411]: pgmap v6209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:46 smithi114 ceph-mon[30820]: pgmap v6210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:57:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:46 smithi116 ceph-mon[35411]: pgmap v6210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:57:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:48 smithi114 ceph-mon[30820]: pgmap v6211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:57:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:48 smithi116 ceph-mon[35411]: pgmap v6211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:57:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:50 smithi114 ceph-mon[30820]: pgmap v6212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:50 smithi116 ceph-mon[35411]: pgmap v6212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:52 smithi114 ceph-mon[30820]: pgmap v6213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:57:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:52 smithi116 ceph-mon[35411]: pgmap v6213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:57:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:54 smithi114 ceph-mon[30820]: pgmap v6214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:54 smithi116 ceph-mon[35411]: pgmap v6214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:57:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:56 smithi114 ceph-mon[30820]: pgmap v6215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:57:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:56 smithi116 ceph-mon[35411]: pgmap v6215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:57:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:57:58 smithi114 ceph-mon[30820]: pgmap v6216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:57:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:57:58 smithi116 ceph-mon[35411]: pgmap v6216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:58:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:00 smithi114 ceph-mon[30820]: pgmap v6217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:00 smithi116 ceph-mon[35411]: pgmap v6217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:02 smithi114 ceph-mon[30820]: pgmap v6218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:02 smithi116 ceph-mon[35411]: pgmap v6218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:04 smithi114 ceph-mon[30820]: pgmap v6219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:58:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:04 smithi116 ceph-mon[35411]: pgmap v6219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:58:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:06 smithi114 ceph-mon[30820]: pgmap v6220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:06 smithi116 ceph-mon[35411]: pgmap v6220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:08 smithi114 ceph-mon[30820]: pgmap v6221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T09:58:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:08 smithi116 ceph-mon[35411]: pgmap v6221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T09:58:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:10 smithi114 ceph-mon[30820]: pgmap v6222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:58:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:10 smithi116 ceph-mon[35411]: pgmap v6222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T09:58:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:12 smithi114 ceph-mon[30820]: pgmap v6223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:12 smithi116 ceph-mon[35411]: pgmap v6223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:14 smithi114 ceph-mon[30820]: pgmap v6224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:14 smithi116 ceph-mon[35411]: pgmap v6224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:16 smithi114 ceph-mon[30820]: pgmap v6225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:58:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:16 smithi116 ceph-mon[35411]: pgmap v6225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:58:18.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:18 smithi114 ceph-mon[30820]: pgmap v6226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:58:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:18 smithi116 ceph-mon[35411]: pgmap v6226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:58:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:20 smithi114 ceph-mon[30820]: pgmap v6227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:20 smithi116 ceph-mon[35411]: pgmap v6227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:22 smithi114 ceph-mon[30820]: pgmap v6228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:22 smithi116 ceph-mon[35411]: pgmap v6228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:24 smithi114 ceph-mon[30820]: pgmap v6229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:24 smithi116 ceph-mon[35411]: pgmap v6229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:26 smithi114 ceph-mon[30820]: pgmap v6230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:58:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:26 smithi116 ceph-mon[35411]: pgmap v6230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T09:58:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:28 smithi114 ceph-mon[30820]: pgmap v6231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:58:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:28 smithi116 ceph-mon[35411]: pgmap v6231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T09:58:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:30 smithi114 ceph-mon[30820]: pgmap v6232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:30 smithi116 ceph-mon[35411]: pgmap v6232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:58:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:58:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:32 smithi114 ceph-mon[30820]: pgmap v6233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:32 smithi116 ceph-mon[35411]: pgmap v6233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-mon[30820]: pgmap v6234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:34.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:58:34.580+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:58:34.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:58:34.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:58:34.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:58:34.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:58:34.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:58:34.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:58:34.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:58:34.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:58:34.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:58:34.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:58:34.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:58:34.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:58:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:58:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:34 smithi116 ceph-mon[35411]: pgmap v6234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:58:35.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:58:35.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:58:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:58:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:58:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:58:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:58:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:36 smithi114 ceph-mon[30820]: pgmap v6235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T09:58:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:36 smithi114 ceph-mon[30820]: pgmap v6236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:58:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:58:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:58:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:58:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:36 smithi116 ceph-mon[35411]: pgmap v6235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T09:58:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:36 smithi116 ceph-mon[35411]: pgmap v6236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T09:58:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:38 smithi114 ceph-mon[30820]: pgmap v6237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:38 smithi116 ceph-mon[35411]: pgmap v6237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:40 smithi114 ceph-mon[30820]: pgmap v6238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:40 smithi116 ceph-mon[35411]: pgmap v6238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:42 smithi116 ceph-mon[35411]: pgmap v6239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:42 smithi114 ceph-mon[30820]: pgmap v6239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:44 smithi116 ceph-mon[35411]: pgmap v6240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:44 smithi114 ceph-mon[30820]: pgmap v6240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:58:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:46 smithi116 ceph-mon[35411]: pgmap v6241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:58:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:46 smithi114 ceph-mon[30820]: pgmap v6241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:58:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:48 smithi114 ceph-mon[30820]: pgmap v6242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:58:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:48 smithi116 ceph-mon[35411]: pgmap v6242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:58:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:50 smithi114 ceph-mon[30820]: pgmap v6243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:50 smithi116 ceph-mon[35411]: pgmap v6243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:52 smithi114 ceph-mon[30820]: pgmap v6244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:52 smithi116 ceph-mon[35411]: pgmap v6244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:58:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:54 smithi114 ceph-mon[30820]: pgmap v6245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:54 smithi116 ceph-mon[35411]: pgmap v6245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:56 smithi114 ceph-mon[30820]: pgmap v6246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:58:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:56 smithi116 ceph-mon[35411]: pgmap v6246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:58:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:57 smithi114 ceph-mon[30820]: pgmap v6247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:58:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:57 smithi116 ceph-mon[35411]: pgmap v6247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:58:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:58:59 smithi114 ceph-mon[30820]: pgmap v6248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:58:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:58:59 smithi116 ceph-mon[35411]: pgmap v6248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:01.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:01 smithi114 ceph-mon[30820]: pgmap v6249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:01 smithi116 ceph-mon[35411]: pgmap v6249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:03 smithi114 ceph-mon[30820]: pgmap v6250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:03 smithi116 ceph-mon[35411]: pgmap v6250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:05 smithi114 ceph-mon[30820]: pgmap v6251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:05 smithi116 ceph-mon[35411]: pgmap v6251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:07 smithi114 ceph-mon[30820]: pgmap v6252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:07 smithi116 ceph-mon[35411]: pgmap v6252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:09 smithi114 ceph-mon[30820]: pgmap v6253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:09 smithi116 ceph-mon[35411]: pgmap v6253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:11 smithi114 ceph-mon[30820]: pgmap v6254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:11 smithi116 ceph-mon[35411]: pgmap v6254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:13 smithi114 ceph-mon[30820]: pgmap v6255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:13 smithi116 ceph-mon[35411]: pgmap v6255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:15 smithi114 ceph-mon[30820]: pgmap v6256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:15 smithi116 ceph-mon[35411]: pgmap v6256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:17 smithi114 ceph-mon[30820]: pgmap v6257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:17 smithi116 ceph-mon[35411]: pgmap v6257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:19 smithi114 ceph-mon[30820]: pgmap v6258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:19 smithi116 ceph-mon[35411]: pgmap v6258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:21 smithi114 ceph-mon[30820]: pgmap v6259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:21 smithi116 ceph-mon[35411]: pgmap v6259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:23 smithi114 ceph-mon[30820]: pgmap v6260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:23 smithi116 ceph-mon[35411]: pgmap v6260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:25 smithi114 ceph-mon[30820]: pgmap v6261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:25 smithi116 ceph-mon[35411]: pgmap v6261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:28.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:27 smithi114 ceph-mon[30820]: pgmap v6262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:27 smithi116 ceph-mon[35411]: pgmap v6262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:29 smithi114 ceph-mon[30820]: pgmap v6263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:29 smithi116 ceph-mon[35411]: pgmap v6263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:31 smithi114 ceph-mon[30820]: pgmap v6264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:31 smithi116 ceph-mon[35411]: pgmap v6264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:33 smithi114 ceph-mon[30820]: pgmap v6265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:33 smithi116 ceph-mon[35411]: pgmap v6265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:59:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T09:59:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:35 smithi114 ceph-mon[30820]: pgmap v6266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:35 smithi116 ceph-mon[35411]: pgmap v6266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:37 smithi116 ceph-mon[35411]: pgmap v6267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-mon[30820]: pgmap v6267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T09:59:37.941+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:59:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T09:59:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:59:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T09:59:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:59:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:59:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:59:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:59:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:59:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 09:59:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:59:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:38 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T09:59:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:59:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:38 smithi116 ceph-mon[35411]: pgmap v6268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:59:39.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:38 smithi116 ceph-mon[35411]: pgmap v6269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:59:39.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:59:39.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:38 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:59:39.319 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T09:59:39.319 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T09:59:39.319 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T09:59:39.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:38 smithi114 ceph-mon[30820]: pgmap v6268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:38 smithi114 ceph-mon[30820]: pgmap v6269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:59:39.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T09:59:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:41 smithi114 ceph-mon[30820]: pgmap v6270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:59:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:41 smithi116 ceph-mon[35411]: pgmap v6270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T09:59:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:43 smithi114 ceph-mon[30820]: pgmap v6271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:59:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:43 smithi116 ceph-mon[35411]: pgmap v6271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:59:45.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:45 smithi114 ceph-mon[30820]: pgmap v6272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:59:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:45 smithi116 ceph-mon[35411]: pgmap v6272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T09:59:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:47 smithi114 ceph-mon[30820]: pgmap v6273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:59:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:47 smithi116 ceph-mon[35411]: pgmap v6273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T09:59:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:49 smithi114 ceph-mon[30820]: pgmap v6274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:59:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:49 smithi116 ceph-mon[35411]: pgmap v6274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T09:59:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:51 smithi114 ceph-mon[30820]: pgmap v6275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:51 smithi116 ceph-mon[35411]: pgmap v6275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T09:59:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:53 smithi114 ceph-mon[30820]: pgmap v6276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:53 smithi116 ceph-mon[35411]: pgmap v6276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T09:59:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:55 smithi114 ceph-mon[30820]: pgmap v6277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:55 smithi116 ceph-mon[35411]: pgmap v6277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:57 smithi114 ceph-mon[30820]: pgmap v6278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:57 smithi116 ceph-mon[35411]: pgmap v6278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T09:59:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 09:59:59 smithi114 ceph-mon[30820]: pgmap v6279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T09:59:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 09:59:59 smithi116 ceph-mon[35411]: pgmap v6279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:00:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:00:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:01 smithi114 ceph-mon[30820]: pgmap v6280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:00:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:01 smithi116 ceph-mon[35411]: pgmap v6280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:00:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:03 smithi114 ceph-mon[30820]: pgmap v6281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:00:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:03 smithi116 ceph-mon[35411]: pgmap v6281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:00:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:05 smithi114 ceph-mon[30820]: pgmap v6282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:05 smithi116 ceph-mon[35411]: pgmap v6282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:07 smithi114 ceph-mon[30820]: pgmap v6283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:00:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:07 smithi116 ceph-mon[35411]: pgmap v6283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:00:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:09 smithi114 ceph-mon[30820]: pgmap v6284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:09 smithi116 ceph-mon[35411]: pgmap v6284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:11 smithi114 ceph-mon[30820]: pgmap v6285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:00:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:11 smithi116 ceph-mon[35411]: pgmap v6285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:00:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:13 smithi114 ceph-mon[30820]: pgmap v6286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:00:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:13 smithi116 ceph-mon[35411]: pgmap v6286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:00:15.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:15 smithi114 ceph-mon[30820]: pgmap v6287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:15 smithi116 ceph-mon[35411]: pgmap v6287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:17 smithi114 ceph-mon[30820]: pgmap v6288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:00:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:17 smithi116 ceph-mon[35411]: pgmap v6288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:00:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:19 smithi114 ceph-mon[30820]: pgmap v6289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:00:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:19 smithi116 ceph-mon[35411]: pgmap v6289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:00:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:21 smithi114 ceph-mon[30820]: pgmap v6290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:00:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:21 smithi116 ceph-mon[35411]: pgmap v6290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:00:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:23 smithi114 ceph-mon[30820]: pgmap v6291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:00:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:23 smithi116 ceph-mon[35411]: pgmap v6291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:00:25.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:25 smithi114 ceph-mon[30820]: pgmap v6292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:00:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:25 smithi116 ceph-mon[35411]: pgmap v6292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:00:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:27 smithi114 ceph-mon[30820]: pgmap v6293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:00:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:27 smithi116 ceph-mon[35411]: pgmap v6293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:00:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:29 smithi114 ceph-mon[30820]: pgmap v6294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:29.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:29 smithi116 ceph-mon[35411]: pgmap v6294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:31 smithi114 ceph-mon[30820]: pgmap v6295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:00:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:31 smithi116 ceph-mon[35411]: pgmap v6295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:00:33.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:33 smithi114 ceph-mon[30820]: pgmap v6296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:00:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:33 smithi116 ceph-mon[35411]: pgmap v6296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:00:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:35 smithi114 ceph-mon[30820]: pgmap v6297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:35 smithi116 ceph-mon[35411]: pgmap v6297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:37 smithi114 ceph-mon[30820]: pgmap v6298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:00:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:37 smithi116 ceph-mon[35411]: pgmap v6298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:00:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:00:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:00:39.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:39 smithi114 ceph-mon[30820]: pgmap v6299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:39 smithi116 ceph-mon[35411]: pgmap v6299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:40 smithi114 ceph-mon[30820]: pgmap v6300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:00:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:40 smithi116 ceph-mon[35411]: pgmap v6300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:00:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:41.589 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:00:41.300+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:00:41.590 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:00:41.591 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:00:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:00:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:00:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:00:41.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:00:41.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:00:41.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-mon[30820]: pgmap v6301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:00:41.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-mon[30820]: pgmap v6302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:00:41.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:00:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:41 smithi116 ceph-mon[35411]: pgmap v6301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:41 smithi116 ceph-mon[35411]: pgmap v6302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:00:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:00:44.633 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:44 smithi116 ceph-mon[35411]: pgmap v6303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:00:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:44 smithi114 ceph-mon[30820]: pgmap v6303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:00:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:46 smithi114 ceph-mon[30820]: pgmap v6304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:00:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:46 smithi116 ceph-mon[35411]: pgmap v6304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:00:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:48 smithi114 ceph-mon[30820]: pgmap v6305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:00:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:48 smithi116 ceph-mon[35411]: pgmap v6305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:00:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:50 smithi114 ceph-mon[30820]: pgmap v6306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:00:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:50 smithi116 ceph-mon[35411]: pgmap v6306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:00:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:52 smithi114 ceph-mon[30820]: pgmap v6307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:00:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:52 smithi116 ceph-mon[35411]: pgmap v6307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:00:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:54 smithi114 ceph-mon[30820]: pgmap v6308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:54 smithi116 ceph-mon[35411]: pgmap v6308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:00:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:56 smithi114 ceph-mon[30820]: pgmap v6309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:00:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:56 smithi116 ceph-mon[35411]: pgmap v6309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:00:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:00:57 smithi114 ceph-mon[30820]: pgmap v6310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:00:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:00:57 smithi116 ceph-mon[35411]: pgmap v6310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:00 smithi114 ceph-mon[30820]: pgmap v6311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:00 smithi116 ceph-mon[35411]: pgmap v6311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:02 smithi114 ceph-mon[30820]: pgmap v6312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:01:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:02 smithi116 ceph-mon[35411]: pgmap v6312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:01:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:04 smithi114 ceph-mon[30820]: pgmap v6313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:04 smithi116 ceph-mon[35411]: pgmap v6313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:06.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:06 smithi114 ceph-mon[30820]: pgmap v6314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:01:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:06 smithi116 ceph-mon[35411]: pgmap v6314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:01:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:08 smithi114 ceph-mon[30820]: pgmap v6315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:08 smithi116 ceph-mon[35411]: pgmap v6315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:10 smithi114 ceph-mon[30820]: pgmap v6316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:10 smithi116 ceph-mon[35411]: pgmap v6316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:12 smithi114 ceph-mon[30820]: pgmap v6317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:01:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:12 smithi116 ceph-mon[35411]: pgmap v6317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:01:14.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:14 smithi114 ceph-mon[30820]: pgmap v6318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:14 smithi116 ceph-mon[35411]: pgmap v6318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:16 smithi114 ceph-mon[30820]: pgmap v6319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:01:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:16 smithi116 ceph-mon[35411]: pgmap v6319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:01:18.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:18 smithi114 ceph-mon[30820]: pgmap v6320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:18 smithi116 ceph-mon[35411]: pgmap v6320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:20 smithi114 ceph-mon[30820]: pgmap v6321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:20 smithi116 ceph-mon[35411]: pgmap v6321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:22 smithi114 ceph-mon[30820]: pgmap v6322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:01:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:22 smithi116 ceph-mon[35411]: pgmap v6322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:01:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:24 smithi114 ceph-mon[30820]: pgmap v6323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:24 smithi116 ceph-mon[35411]: pgmap v6323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:26 smithi114 ceph-mon[30820]: pgmap v6324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:01:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:26 smithi116 ceph-mon[35411]: pgmap v6324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:01:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:28 smithi116 ceph-mon[35411]: pgmap v6325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:28 smithi114 ceph-mon[30820]: pgmap v6325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:30.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:30 smithi116 ceph-mon[35411]: pgmap v6326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:01:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:30 smithi114 ceph-mon[30820]: pgmap v6326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:01:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:32 smithi116 ceph-mon[35411]: pgmap v6327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:01:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:32 smithi114 ceph-mon[30820]: pgmap v6327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:01:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:34 smithi114 ceph-mon[30820]: pgmap v6328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:34 smithi116 ceph-mon[35411]: pgmap v6328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:36 smithi114 ceph-mon[30820]: pgmap v6329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:01:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:36 smithi116 ceph-mon[35411]: pgmap v6329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:01:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:38 smithi114 ceph-mon[30820]: pgmap v6330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:38 smithi116 ceph-mon[35411]: pgmap v6330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:01:40.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:40 smithi114 ceph-mon[30820]: pgmap v6331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:01:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:40 smithi116 ceph-mon[35411]: pgmap v6331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:01:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:01:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:01:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:42 smithi114 ceph-mon[30820]: pgmap v6332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:01:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:42 smithi116 ceph-mon[35411]: pgmap v6332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:01:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:44.732 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:44 smithi116 ceph-mon[35411]: pgmap v6333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-mon[30820]: pgmap v6333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:01:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:01:44.661+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:01:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:01:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:01:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:01:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:01:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:01:45.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:01:45.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:01:45.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:01:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:01:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:01:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:01:46.928 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:46 smithi114 ceph-mon[30820]: pgmap v6334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:01:46.929 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:46 smithi114 ceph-mon[30820]: pgmap v6335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:01:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:01:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:46 smithi116 ceph-mon[35411]: pgmap v6334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:01:46.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:46 smithi116 ceph-mon[35411]: pgmap v6335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:01:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:48 smithi114 ceph-mon[30820]: pgmap v6336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:01:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:48 smithi116 ceph-mon[35411]: pgmap v6336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:01:50.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:50 smithi114 ceph-mon[30820]: pgmap v6337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:01:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:50 smithi116 ceph-mon[35411]: pgmap v6337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:01:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:52 smithi114 ceph-mon[30820]: pgmap v6338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:01:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:52 smithi116 ceph-mon[35411]: pgmap v6338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:01:54.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:54 smithi114 ceph-mon[30820]: pgmap v6339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:01:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:54 smithi116 ceph-mon[35411]: pgmap v6339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:01:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:56 smithi114 ceph-mon[30820]: pgmap v6340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:01:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:56 smithi116 ceph-mon[35411]: pgmap v6340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:01:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:01:58 smithi114 ceph-mon[30820]: pgmap v6341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:01:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:01:58 smithi116 ceph-mon[35411]: pgmap v6341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:00 smithi114 ceph-mon[30820]: pgmap v6342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:00 smithi116 ceph-mon[35411]: pgmap v6342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:01.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:01 smithi114 ceph-mon[30820]: pgmap v6343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:01 smithi116 ceph-mon[35411]: pgmap v6343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:03 smithi114 ceph-mon[30820]: pgmap v6344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:03 smithi116 ceph-mon[35411]: pgmap v6344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:05 smithi114 ceph-mon[30820]: pgmap v6345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:05 smithi116 ceph-mon[35411]: pgmap v6345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:07 smithi114 ceph-mon[30820]: pgmap v6346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:07 smithi116 ceph-mon[35411]: pgmap v6346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:09 smithi114 ceph-mon[30820]: pgmap v6347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:09 smithi116 ceph-mon[35411]: pgmap v6347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:11 smithi114 ceph-mon[30820]: pgmap v6348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:11 smithi116 ceph-mon[35411]: pgmap v6348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:13 smithi114 ceph-mon[30820]: pgmap v6349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:13 smithi116 ceph-mon[35411]: pgmap v6349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:15 smithi114 ceph-mon[30820]: pgmap v6350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:15 smithi116 ceph-mon[35411]: pgmap v6350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:17 smithi114 ceph-mon[30820]: pgmap v6351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:17 smithi116 ceph-mon[35411]: pgmap v6351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:19 smithi114 ceph-mon[30820]: pgmap v6352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:19 smithi116 ceph-mon[35411]: pgmap v6352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:21 smithi114 ceph-mon[30820]: pgmap v6353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:21 smithi116 ceph-mon[35411]: pgmap v6353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:23 smithi114 ceph-mon[30820]: pgmap v6354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:23 smithi116 ceph-mon[35411]: pgmap v6354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:25 smithi114 ceph-mon[30820]: pgmap v6355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:25 smithi116 ceph-mon[35411]: pgmap v6355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:27 smithi114 ceph-mon[30820]: pgmap v6356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:27 smithi116 ceph-mon[35411]: pgmap v6356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:29 smithi114 ceph-mon[30820]: pgmap v6357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:29 smithi116 ceph-mon[35411]: pgmap v6357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:31 smithi114 ceph-mon[30820]: pgmap v6358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:31 smithi116 ceph-mon[35411]: pgmap v6358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:33 smithi114 ceph-mon[30820]: pgmap v6359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:33 smithi116 ceph-mon[35411]: pgmap v6359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:35 smithi114 ceph-mon[30820]: pgmap v6360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:35 smithi116 ceph-mon[35411]: pgmap v6360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:38.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:37 smithi114 ceph-mon[30820]: pgmap v6361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:37 smithi116 ceph-mon[35411]: pgmap v6361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:39 smithi114 ceph-mon[30820]: pgmap v6362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:39 smithi116 ceph-mon[35411]: pgmap v6362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:41 smithi114 ceph-mon[30820]: pgmap v6363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:41 smithi116 ceph-mon[35411]: pgmap v6363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:02:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:43 smithi114 ceph-mon[30820]: pgmap v6364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:43 smithi116 ceph-mon[35411]: pgmap v6364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:02:45.080 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:02:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:02:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:45 smithi114 ceph-mon[30820]: pgmap v6365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:45 smithi116 ceph-mon[35411]: pgmap v6365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:02:48.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:47 smithi114 ceph-mon[30820]: pgmap v6366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:02:48.021+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:02:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:02:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:02:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:02:48.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:02:48.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:02:48.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:02:48.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:02:48.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:02:48.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:02:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:47 smithi116 ceph-mon[35411]: pgmap v6366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:02:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:02:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:02:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:02:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:02:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-mon[30820]: pgmap v6367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:02:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-mon[30820]: pgmap v6368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:02:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:48 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:02:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:02:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:02:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:02:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:02:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:02:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:02:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:02:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:02:49.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:02:49.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:02:49.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:02:49.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:02:49.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:02:49.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:02:49.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:48 smithi116 ceph-mon[35411]: pgmap v6367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:02:49.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:48 smithi116 ceph-mon[35411]: pgmap v6368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:02:49.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:02:51.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:51 smithi114 ceph-mon[30820]: pgmap v6369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:02:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:51 smithi116 ceph-mon[35411]: pgmap v6369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:02:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:53 smithi114 ceph-mon[30820]: pgmap v6370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:02:53.451 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:53 smithi116 ceph-mon[35411]: pgmap v6370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:02:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:55 smithi114 ceph-mon[30820]: pgmap v6371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:02:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:55 smithi116 ceph-mon[35411]: pgmap v6371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:02:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:57 smithi114 ceph-mon[30820]: pgmap v6372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:02:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:57 smithi116 ceph-mon[35411]: pgmap v6372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:02:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:02:59 smithi114 ceph-mon[30820]: pgmap v6373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:02:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:02:59 smithi116 ceph-mon[35411]: pgmap v6373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:03:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:01 smithi114 ceph-mon[30820]: pgmap v6374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:01 smithi116 ceph-mon[35411]: pgmap v6374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:03 smithi114 ceph-mon[30820]: pgmap v6375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:03 smithi116 ceph-mon[35411]: pgmap v6375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:05 smithi114 ceph-mon[30820]: pgmap v6376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:05 smithi116 ceph-mon[35411]: pgmap v6376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:07 smithi114 ceph-mon[30820]: pgmap v6377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:07 smithi116 ceph-mon[35411]: pgmap v6377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:09 smithi114 ceph-mon[30820]: pgmap v6378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:09 smithi116 ceph-mon[35411]: pgmap v6378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:11 smithi114 ceph-mon[30820]: pgmap v6379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:11 smithi116 ceph-mon[35411]: pgmap v6379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:13 smithi114 ceph-mon[30820]: pgmap v6380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:13 smithi116 ceph-mon[35411]: pgmap v6380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:15.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:15 smithi114 ceph-mon[30820]: pgmap v6381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:15 smithi116 ceph-mon[35411]: pgmap v6381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:17 smithi114 ceph-mon[30820]: pgmap v6382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:17 smithi116 ceph-mon[35411]: pgmap v6382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:19 smithi114 ceph-mon[30820]: pgmap v6383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:19 smithi116 ceph-mon[35411]: pgmap v6383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:21 smithi114 ceph-mon[30820]: pgmap v6384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:21 smithi116 ceph-mon[35411]: pgmap v6384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:23 smithi114 ceph-mon[30820]: pgmap v6385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:23 smithi116 ceph-mon[35411]: pgmap v6385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:25 smithi114 ceph-mon[30820]: pgmap v6386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:25 smithi116 ceph-mon[35411]: pgmap v6386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:27 smithi114 ceph-mon[30820]: pgmap v6387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:27 smithi116 ceph-mon[35411]: pgmap v6387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:29 smithi114 ceph-mon[30820]: pgmap v6388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:29 smithi116 ceph-mon[35411]: pgmap v6388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:31 smithi114 ceph-mon[30820]: pgmap v6389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:31 smithi116 ceph-mon[35411]: pgmap v6389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:33.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:33 smithi114 ceph-mon[30820]: pgmap v6390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:33 smithi116 ceph-mon[35411]: pgmap v6390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:35.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:35 smithi114 ceph-mon[30820]: pgmap v6391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:35 smithi116 ceph-mon[35411]: pgmap v6391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:37 smithi114 ceph-mon[30820]: pgmap v6392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:37 smithi116 ceph-mon[35411]: pgmap v6392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:39 smithi116 ceph-mon[35411]: pgmap v6393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:39 smithi114 ceph-mon[30820]: pgmap v6393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:41.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:41 smithi116 ceph-mon[35411]: pgmap v6394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:41.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:41 smithi114 ceph-mon[30820]: pgmap v6394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:43 smithi116 ceph-mon[35411]: pgmap v6395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:43 smithi114 ceph-mon[30820]: pgmap v6395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:03:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:45 smithi114 ceph-mon[30820]: pgmap v6396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:45 smithi116 ceph-mon[35411]: pgmap v6396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:47 smithi114 ceph-mon[30820]: pgmap v6397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:47 smithi116 ceph-mon[35411]: pgmap v6397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:03:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:03:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:03:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:49 smithi114 ceph-mon[30820]: pgmap v6398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:49 smithi116 ceph-mon[35411]: pgmap v6398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:03:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-mon[30820]: pgmap v6399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:51.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:03:51.380+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:03:51.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:03:51.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:03:51.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:03:51.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:03:51.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:03:51.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:03:51.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:03:51.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:03:51.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:03:51.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:03:51.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:03:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:03:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:51 smithi116 ceph-mon[35411]: pgmap v6399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:03:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:52 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:03:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:03:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:03:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:03:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:03:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:03:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:03:52.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:03:52.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:03:52.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:03:52.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:03:52.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:03:52.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:03:52.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:52 smithi114 ceph-mon[30820]: pgmap v6400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:03:52.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:52 smithi114 ceph-mon[30820]: pgmap v6401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:03:52.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:52 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:03:52.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:03:52.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:03:52.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:03:52.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:03:52.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:52 smithi116 ceph-mon[35411]: pgmap v6400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:03:52.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:52 smithi116 ceph-mon[35411]: pgmap v6401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:03:52.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:03:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:54 smithi114 ceph-mon[30820]: pgmap v6402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:03:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:54 smithi116 ceph-mon[35411]: pgmap v6402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:03:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:56 smithi114 ceph-mon[30820]: pgmap v6403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:03:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:56 smithi116 ceph-mon[35411]: pgmap v6403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:03:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:03:58 smithi114 ceph-mon[30820]: pgmap v6404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:03:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:03:58 smithi116 ceph-mon[35411]: pgmap v6404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:04:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:00 smithi114 ceph-mon[30820]: pgmap v6405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:04:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:00 smithi116 ceph-mon[35411]: pgmap v6405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:04:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:02 smithi114 ceph-mon[30820]: pgmap v6406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:04:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:02 smithi116 ceph-mon[35411]: pgmap v6406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:04:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:04 smithi114 ceph-mon[30820]: pgmap v6407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:04 smithi116 ceph-mon[35411]: pgmap v6407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:06.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:06 smithi114 ceph-mon[30820]: pgmap v6408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:06 smithi116 ceph-mon[35411]: pgmap v6408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:08.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:08 smithi114 ceph-mon[30820]: pgmap v6409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:08 smithi116 ceph-mon[35411]: pgmap v6409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:10 smithi114 ceph-mon[30820]: pgmap v6410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:10 smithi116 ceph-mon[35411]: pgmap v6410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:12.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:12 smithi114 ceph-mon[30820]: pgmap v6411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:12 smithi116 ceph-mon[35411]: pgmap v6411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:14 smithi114 ceph-mon[30820]: pgmap v6412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:14 smithi116 ceph-mon[35411]: pgmap v6412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:16.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:16 smithi114 ceph-mon[30820]: pgmap v6413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:16 smithi116 ceph-mon[35411]: pgmap v6413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:18 smithi114 ceph-mon[30820]: pgmap v6414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:18 smithi116 ceph-mon[35411]: pgmap v6414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:20 smithi114 ceph-mon[30820]: pgmap v6415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:20 smithi116 ceph-mon[35411]: pgmap v6415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:22 smithi114 ceph-mon[30820]: pgmap v6416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:22 smithi116 ceph-mon[35411]: pgmap v6416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:24 smithi114 ceph-mon[30820]: pgmap v6417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:24 smithi116 ceph-mon[35411]: pgmap v6417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:26 smithi114 ceph-mon[30820]: pgmap v6418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:26 smithi116 ceph-mon[35411]: pgmap v6418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:28 smithi114 ceph-mon[30820]: pgmap v6419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:28 smithi116 ceph-mon[35411]: pgmap v6419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:30 smithi114 ceph-mon[30820]: pgmap v6420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:30 smithi116 ceph-mon[35411]: pgmap v6420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:31 smithi114 ceph-mon[30820]: pgmap v6421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:31 smithi116 ceph-mon[35411]: pgmap v6421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:34 smithi114 ceph-mon[30820]: pgmap v6422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:34 smithi116 ceph-mon[35411]: pgmap v6422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:36 smithi114 ceph-mon[30820]: pgmap v6423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:36 smithi116 ceph-mon[35411]: pgmap v6423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:38 smithi114 ceph-mon[30820]: pgmap v6424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:38 smithi116 ceph-mon[35411]: pgmap v6424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:40 smithi114 ceph-mon[30820]: pgmap v6425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:41.000 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:40 smithi116 ceph-mon[35411]: pgmap v6425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:42 smithi114 ceph-mon[30820]: pgmap v6426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:42 smithi116 ceph-mon[35411]: pgmap v6426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:44 smithi114 ceph-mon[30820]: pgmap v6427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:44 smithi116 ceph-mon[35411]: pgmap v6427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:46.732 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:46 smithi116 ceph-mon[35411]: pgmap v6428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:46 smithi114 ceph-mon[30820]: pgmap v6428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:04:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:48 smithi114 ceph-mon[30820]: pgmap v6429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:48 smithi116 ceph-mon[35411]: pgmap v6429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:50.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:50 smithi114 ceph-mon[30820]: pgmap v6430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:50 smithi116 ceph-mon[35411]: pgmap v6430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:04:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:52 smithi114 ceph-mon[30820]: pgmap v6431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:04:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:52 smithi116 ceph-mon[35411]: pgmap v6431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:04:52.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:04:54.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-mon[30820]: pgmap v6432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:54.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:04:54.740+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:04:54.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:04:54.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:04:54.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:04:54.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:04:54.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:04:54.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:04:54.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:04:54.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:04:54.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:04:54.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:04:54.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:04:54.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:04:54.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:04:54.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:04:54.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:04:54.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:04:54.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:04:54.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:04:54.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:04:54.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:04:54.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:04:54.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:04:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:04:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:54 smithi116 ceph-mon[35411]: pgmap v6432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:04:55.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:04:55.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:04:55.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:04:55.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:04:55.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:04:55.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:04:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:04:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:04:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:04:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:04:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:04:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:04:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:04:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:04:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:04:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:04:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:04:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:04:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:04:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:04:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:04:56.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:04:56.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:04:56.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:04:56.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:04:56.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:04:56.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:04:56.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:04:56.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:04:56.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:56 smithi114 ceph-mon[30820]: pgmap v6433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:04:56.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:56 smithi114 ceph-mon[30820]: pgmap v6434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:04:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:56 smithi116 ceph-mon[35411]: pgmap v6433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:04:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:56 smithi116 ceph-mon[35411]: pgmap v6434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:04:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:04:58 smithi114 ceph-mon[30820]: pgmap v6435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:04:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:04:58 smithi116 ceph-mon[35411]: pgmap v6435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:05:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:00 smithi114 ceph-mon[30820]: pgmap v6436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:05:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:00 smithi116 ceph-mon[35411]: pgmap v6436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:05:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:02 smithi114 ceph-mon[30820]: pgmap v6437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:05:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:02 smithi116 ceph-mon[35411]: pgmap v6437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:05:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:03 smithi114 ceph-mon[30820]: pgmap v6438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:05:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:03 smithi116 ceph-mon[35411]: pgmap v6438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:05:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:05 smithi114 ceph-mon[30820]: pgmap v6439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:05:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:05 smithi116 ceph-mon[35411]: pgmap v6439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:05:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:07 smithi114 ceph-mon[30820]: pgmap v6440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:07 smithi116 ceph-mon[35411]: pgmap v6440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:09 smithi114 ceph-mon[30820]: pgmap v6441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:09 smithi116 ceph-mon[35411]: pgmap v6441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:11 smithi114 ceph-mon[30820]: pgmap v6442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:11 smithi116 ceph-mon[35411]: pgmap v6442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:13 smithi114 ceph-mon[30820]: pgmap v6443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:13 smithi116 ceph-mon[35411]: pgmap v6443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:15 smithi114 ceph-mon[30820]: pgmap v6444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:15 smithi116 ceph-mon[35411]: pgmap v6444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:17 smithi114 ceph-mon[30820]: pgmap v6445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:17 smithi116 ceph-mon[35411]: pgmap v6445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:19 smithi114 ceph-mon[30820]: pgmap v6446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:19 smithi116 ceph-mon[35411]: pgmap v6446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:21 smithi114 ceph-mon[30820]: pgmap v6447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:21 smithi116 ceph-mon[35411]: pgmap v6447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:23 smithi114 ceph-mon[30820]: pgmap v6448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:23 smithi116 ceph-mon[35411]: pgmap v6448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:25 smithi114 ceph-mon[30820]: pgmap v6449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:25 smithi116 ceph-mon[35411]: pgmap v6449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:28.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:27 smithi114 ceph-mon[30820]: pgmap v6450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:27 smithi116 ceph-mon[35411]: pgmap v6450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:29 smithi114 ceph-mon[30820]: pgmap v6451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:29 smithi116 ceph-mon[35411]: pgmap v6451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:31 smithi114 ceph-mon[30820]: pgmap v6452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:31 smithi116 ceph-mon[35411]: pgmap v6452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:33 smithi114 ceph-mon[30820]: pgmap v6453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:33 smithi116 ceph-mon[35411]: pgmap v6453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:35 smithi114 ceph-mon[30820]: pgmap v6454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:35 smithi116 ceph-mon[35411]: pgmap v6454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:38.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:37 smithi114 ceph-mon[30820]: pgmap v6455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:37 smithi116 ceph-mon[35411]: pgmap v6455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:39 smithi114 ceph-mon[30820]: pgmap v6456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:39 smithi116 ceph-mon[35411]: pgmap v6456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:41 smithi114 ceph-mon[30820]: pgmap v6457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:41 smithi116 ceph-mon[35411]: pgmap v6457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:43 smithi114 ceph-mon[30820]: pgmap v6458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:43 smithi116 ceph-mon[35411]: pgmap v6458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:45 smithi114 ceph-mon[30820]: pgmap v6459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:45 smithi116 ceph-mon[35411]: pgmap v6459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:48.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:47 smithi114 ceph-mon[30820]: pgmap v6460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:47 smithi116 ceph-mon[35411]: pgmap v6460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:49 smithi114 ceph-mon[30820]: pgmap v6461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:49 smithi116 ceph-mon[35411]: pgmap v6461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:51 smithi114 ceph-mon[30820]: pgmap v6462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:51 smithi116 ceph-mon[35411]: pgmap v6462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:05:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:53 smithi114 ceph-mon[30820]: pgmap v6463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:53 smithi116 ceph-mon[35411]: pgmap v6463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:05:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:05:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:05:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:55 smithi114 ceph-mon[30820]: pgmap v6464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:55 smithi116 ceph-mon[35411]: pgmap v6464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:05:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:57 smithi114 ceph-mon[30820]: pgmap v6465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:58.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:05:58.101+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:05:58.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:05:58.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:05:58.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:05:58.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:05:58.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:05:58.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:05:58.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:05:58.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:05:58.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:05:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:57 smithi116 ceph-mon[35411]: pgmap v6465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:05:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:05:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:05:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:05:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:05:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:05:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:05:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:05:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:05:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-mon[30820]: pgmap v6466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-mon[30820]: pgmap v6467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:05:59.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:05:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:05:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:58 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:05:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:05:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:58 smithi116 ceph-mon[35411]: pgmap v6466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:05:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:58 smithi116 ceph-mon[35411]: pgmap v6467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:05:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:05:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:06:01.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:01 smithi114 ceph-mon[30820]: pgmap v6468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:06:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:01 smithi116 ceph-mon[35411]: pgmap v6468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:06:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:03 smithi114 ceph-mon[30820]: pgmap v6469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:06:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:03 smithi116 ceph-mon[35411]: pgmap v6469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:06:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:05 smithi116 ceph-mon[35411]: pgmap v6470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:06:05.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:05 smithi114 ceph-mon[30820]: pgmap v6470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:06:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:07 smithi116 ceph-mon[35411]: pgmap v6471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:06:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:07 smithi114 ceph-mon[30820]: pgmap v6471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:06:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:09 smithi116 ceph-mon[35411]: pgmap v6472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:06:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:09 smithi114 ceph-mon[30820]: pgmap v6472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:06:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:11 smithi114 ceph-mon[30820]: pgmap v6473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:11 smithi116 ceph-mon[35411]: pgmap v6473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:13.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:13 smithi114 ceph-mon[30820]: pgmap v6474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:13.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:13 smithi116 ceph-mon[35411]: pgmap v6474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:15.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:15 smithi114 ceph-mon[30820]: pgmap v6475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:15 smithi116 ceph-mon[35411]: pgmap v6475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:17.482 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:17 smithi116 ceph-mon[35411]: pgmap v6476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:17 smithi114 ceph-mon[30820]: pgmap v6476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:19 smithi114 ceph-mon[30820]: pgmap v6477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:19 smithi116 ceph-mon[35411]: pgmap v6477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:21 smithi114 ceph-mon[30820]: pgmap v6478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:21 smithi116 ceph-mon[35411]: pgmap v6478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:23 smithi114 ceph-mon[30820]: pgmap v6479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:23 smithi116 ceph-mon[35411]: pgmap v6479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:25 smithi114 ceph-mon[30820]: pgmap v6480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:25 smithi116 ceph-mon[35411]: pgmap v6480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:27 smithi114 ceph-mon[30820]: pgmap v6481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:27 smithi116 ceph-mon[35411]: pgmap v6481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:29.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:29 smithi114 ceph-mon[30820]: pgmap v6482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:29 smithi116 ceph-mon[35411]: pgmap v6482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:31 smithi114 ceph-mon[30820]: pgmap v6483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:31 smithi116 ceph-mon[35411]: pgmap v6483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:33 smithi114 ceph-mon[30820]: pgmap v6484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:33 smithi116 ceph-mon[35411]: pgmap v6484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:35.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:35 smithi114 ceph-mon[30820]: pgmap v6485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:35.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:35 smithi116 ceph-mon[35411]: pgmap v6485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:37 smithi114 ceph-mon[30820]: pgmap v6486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:37 smithi116 ceph-mon[35411]: pgmap v6486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:39 smithi114 ceph-mon[30820]: pgmap v6487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:39 smithi116 ceph-mon[35411]: pgmap v6487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:41 smithi114 ceph-mon[30820]: pgmap v6488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:41 smithi116 ceph-mon[35411]: pgmap v6488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:43.477 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:43 smithi114 ceph-mon[30820]: pgmap v6489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:43 smithi116 ceph-mon[35411]: pgmap v6489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:45 smithi114 ceph-mon[30820]: pgmap v6490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:45 smithi116 ceph-mon[35411]: pgmap v6490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:47 smithi114 ceph-mon[30820]: pgmap v6491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:47 smithi116 ceph-mon[35411]: pgmap v6491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:49 smithi114 ceph-mon[30820]: pgmap v6492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:49 smithi116 ceph-mon[35411]: pgmap v6492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:51 smithi114 ceph-mon[30820]: pgmap v6493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:51 smithi116 ceph-mon[35411]: pgmap v6493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:06:53.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:53 smithi114 ceph-mon[30820]: pgmap v6494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:53.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:53 smithi116 ceph-mon[35411]: pgmap v6494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:06:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:55 smithi114 ceph-mon[30820]: pgmap v6495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:55.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:55 smithi116 ceph-mon[35411]: pgmap v6495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:57 smithi114 ceph-mon[30820]: pgmap v6496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:57 smithi116 ceph-mon[35411]: pgmap v6496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:06:58.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:06:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:06:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:06:59 smithi114 ceph-mon[30820]: pgmap v6497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:06:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:06:59 smithi116 ceph-mon[35411]: pgmap v6497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-mon[30820]: pgmap v6498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:07:01.460+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:07:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:07:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:07:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:07:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:07:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:07:01.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:07:01.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:07:01.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:07:01.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:07:01.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:07:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:07:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:01 smithi116 ceph-mon[35411]: pgmap v6498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:02 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:07:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:07:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:07:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:07:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:07:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:07:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:07:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:07:02.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:07:02.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:07:02.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:07:02.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:07:02.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:02 smithi114 ceph-mon[30820]: pgmap v6499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:07:02.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:02 smithi114 ceph-mon[30820]: pgmap v6500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:02.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:02 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:07:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:07:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:07:02.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:02 smithi116 ceph-mon[35411]: pgmap v6499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:07:02.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:02 smithi116 ceph-mon[35411]: pgmap v6500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:02.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:07:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:04 smithi114 ceph-mon[30820]: pgmap v6501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:04 smithi116 ceph-mon[35411]: pgmap v6501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:06.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:06 smithi114 ceph-mon[30820]: pgmap v6502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:06 smithi116 ceph-mon[35411]: pgmap v6502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:08.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:08 smithi114 ceph-mon[30820]: pgmap v6503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:08.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:08 smithi116 ceph-mon[35411]: pgmap v6503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:10.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:10 smithi114 ceph-mon[30820]: pgmap v6504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:10 smithi116 ceph-mon[35411]: pgmap v6504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:07:12.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:12 smithi114 ceph-mon[30820]: pgmap v6505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:07:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:12 smithi116 ceph-mon[35411]: pgmap v6505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:07:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:14 smithi114 ceph-mon[30820]: pgmap v6506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:14 smithi116 ceph-mon[35411]: pgmap v6506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:15.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:15 smithi114 ceph-mon[30820]: pgmap v6507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:15 smithi116 ceph-mon[35411]: pgmap v6507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:18 smithi114 ceph-mon[30820]: pgmap v6508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:18 smithi116 ceph-mon[35411]: pgmap v6508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:20 smithi114 ceph-mon[30820]: pgmap v6509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:20 smithi116 ceph-mon[35411]: pgmap v6509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:22.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:22 smithi114 ceph-mon[30820]: pgmap v6510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:07:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:22 smithi116 ceph-mon[35411]: pgmap v6510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:07:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:24 smithi114 ceph-mon[30820]: pgmap v6511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:24 smithi116 ceph-mon[35411]: pgmap v6511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:26 smithi114 ceph-mon[30820]: pgmap v6512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:26 smithi116 ceph-mon[35411]: pgmap v6512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:27 smithi114 ceph-mon[30820]: pgmap v6513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:27 smithi116 ceph-mon[35411]: pgmap v6513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:30 smithi114 ceph-mon[30820]: pgmap v6514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:30.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:30 smithi116 ceph-mon[35411]: pgmap v6514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:32.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:32 smithi114 ceph-mon[30820]: pgmap v6515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:07:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:32 smithi116 ceph-mon[35411]: pgmap v6515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:07:34.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:34 smithi114 ceph-mon[30820]: pgmap v6516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:34 smithi116 ceph-mon[35411]: pgmap v6516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:36 smithi114 ceph-mon[30820]: pgmap v6517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:36 smithi116 ceph-mon[35411]: pgmap v6517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:37.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:37 smithi114 ceph-mon[30820]: pgmap v6518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:37 smithi116 ceph-mon[35411]: pgmap v6518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:40 smithi114 ceph-mon[30820]: pgmap v6519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:40 smithi116 ceph-mon[35411]: pgmap v6519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:42 smithi114 ceph-mon[30820]: pgmap v6520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:07:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:42 smithi116 ceph-mon[35411]: pgmap v6520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:07:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:44 smithi114 ceph-mon[30820]: pgmap v6521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:44.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:44 smithi116 ceph-mon[35411]: pgmap v6521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:46 smithi114 ceph-mon[30820]: pgmap v6522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:46 smithi116 ceph-mon[35411]: pgmap v6522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:47 smithi114 ceph-mon[30820]: pgmap v6523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:47 smithi116 ceph-mon[35411]: pgmap v6523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:50.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:50 smithi114 ceph-mon[30820]: pgmap v6524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:50 smithi116 ceph-mon[35411]: pgmap v6524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:07:52.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:52 smithi114 ceph-mon[30820]: pgmap v6525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:07:52.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:52 smithi116 ceph-mon[35411]: pgmap v6525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:07:54.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:54 smithi114 ceph-mon[30820]: pgmap v6526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:54 smithi116 ceph-mon[35411]: pgmap v6526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:55.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:55 smithi114 ceph-mon[30820]: pgmap v6527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:55 smithi116 ceph-mon[35411]: pgmap v6527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:07:58.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:07:58 smithi114 ceph-mon[30820]: pgmap v6528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:07:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:07:58 smithi116 ceph-mon[35411]: pgmap v6528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:00 smithi114 ceph-mon[30820]: pgmap v6529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:08:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:00 smithi116 ceph-mon[35411]: pgmap v6529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: pgmap v6530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: pgmap v6530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:02.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:08:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:03 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:08:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:03 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:08:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:03 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:08:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:03 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:08:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:03 smithi114 ceph-mon[30820]: pgmap v6531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:08:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:03 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:08:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:03 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:08:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:03 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:08:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:03 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:08:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:03 smithi116 ceph-mon[35411]: pgmap v6531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:08:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:08:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:08:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:08:04.821+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:08:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:08:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:08:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:08:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:08:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:08:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:08:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:05 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:08:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:08:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:08:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:08:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:08:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:08:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:08:06.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:08:06.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:08:06.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:08:06.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:08:06.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:05 smithi114 ceph-mon[30820]: pgmap v6532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:08:06.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:05 smithi114 ceph-mon[30820]: pgmap v6533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T10:08:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:05 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:08:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:08:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:08:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:08:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:08:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:08:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:08:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:08:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:08:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:08:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:08:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:08:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:08:06.184 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:08:06.184 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:08:06.184 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:05 smithi116 ceph-mon[35411]: pgmap v6532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:08:06.184 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:05 smithi116 ceph-mon[35411]: pgmap v6533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T10:08:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:07 smithi114 ceph-mon[30820]: pgmap v6534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 6 op/s 2024-09-19T10:08:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:07 smithi116 ceph-mon[35411]: pgmap v6534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 6 op/s 2024-09-19T10:08:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:09 smithi114 ceph-mon[30820]: pgmap v6535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 4 op/s 2024-09-19T10:08:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:09 smithi116 ceph-mon[35411]: pgmap v6535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 4 op/s 2024-09-19T10:08:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:11 smithi114 ceph-mon[30820]: pgmap v6536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:08:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:11 smithi116 ceph-mon[35411]: pgmap v6536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:08:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:13 smithi114 ceph-mon[30820]: pgmap v6537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:08:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:13 smithi116 ceph-mon[35411]: pgmap v6537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:08:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:15 smithi114 ceph-mon[30820]: pgmap v6538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:08:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:15 smithi116 ceph-mon[35411]: pgmap v6538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:08:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:17 smithi114 ceph-mon[30820]: pgmap v6539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:17 smithi116 ceph-mon[35411]: pgmap v6539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:19 smithi114 ceph-mon[30820]: pgmap v6540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:19 smithi116 ceph-mon[35411]: pgmap v6540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:21 smithi114 ceph-mon[30820]: pgmap v6541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:08:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:21 smithi116 ceph-mon[35411]: pgmap v6541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:08:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:23 smithi116 ceph-mon[35411]: pgmap v6542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:23 smithi114 ceph-mon[30820]: pgmap v6542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:25 smithi116 ceph-mon[35411]: pgmap v6543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:08:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:25 smithi114 ceph-mon[30820]: pgmap v6543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:08:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:27 smithi116 ceph-mon[35411]: pgmap v6544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:27 smithi114 ceph-mon[30820]: pgmap v6544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:29 smithi114 ceph-mon[30820]: pgmap v6545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:29 smithi116 ceph-mon[35411]: pgmap v6545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:31 smithi114 ceph-mon[30820]: pgmap v6546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:08:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:31 smithi116 ceph-mon[35411]: pgmap v6546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:08:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:33 smithi114 ceph-mon[30820]: pgmap v6547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:33 smithi116 ceph-mon[35411]: pgmap v6547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:35 smithi114 ceph-mon[30820]: pgmap v6548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T10:08:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:35 smithi116 ceph-mon[35411]: pgmap v6548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T10:08:38.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:37 smithi114 ceph-mon[30820]: pgmap v6549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 5 op/s 2024-09-19T10:08:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:37 smithi116 ceph-mon[35411]: pgmap v6549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 5 op/s 2024-09-19T10:08:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:39 smithi114 ceph-mon[30820]: pgmap v6550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:39 smithi116 ceph-mon[35411]: pgmap v6550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:42.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:41 smithi114 ceph-mon[30820]: pgmap v6551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:08:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:41 smithi116 ceph-mon[35411]: pgmap v6551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:08:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:43 smithi114 ceph-mon[30820]: pgmap v6552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:43 smithi116 ceph-mon[35411]: pgmap v6552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:46.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:45 smithi114 ceph-mon[30820]: pgmap v6553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:08:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:45 smithi116 ceph-mon[35411]: pgmap v6553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:47 smithi114 ceph-mon[30820]: pgmap v6554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:47 smithi116 ceph-mon[35411]: pgmap v6554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:49 smithi114 ceph-mon[30820]: pgmap v6555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:49 smithi116 ceph-mon[35411]: pgmap v6555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:52.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:51 smithi116 ceph-mon[35411]: pgmap v6556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:08:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:51 smithi114 ceph-mon[30820]: pgmap v6556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:08:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:53 smithi114 ceph-mon[30820]: pgmap v6557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:53 smithi116 ceph-mon[35411]: pgmap v6557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:08:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:55 smithi114 ceph-mon[30820]: pgmap v6558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:08:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:55 smithi116 ceph-mon[35411]: pgmap v6558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:08:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:57 smithi114 ceph-mon[30820]: pgmap v6559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:08:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:57 smithi116 ceph-mon[35411]: pgmap v6559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:09:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:08:59 smithi114 ceph-mon[30820]: pgmap v6560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:08:59 smithi116 ceph-mon[35411]: pgmap v6560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:01 smithi114 ceph-mon[30820]: pgmap v6561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:01 smithi116 ceph-mon[35411]: pgmap v6561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:03 smithi114 ceph-mon[30820]: pgmap v6562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:03 smithi116 ceph-mon[35411]: pgmap v6562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:09:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:09:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:06 smithi114 ceph-mon[30820]: pgmap v6563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:09:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:06 smithi116 ceph-mon[35411]: pgmap v6563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:09:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-mon[30820]: pgmap v6564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:09:08.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:09:08.181+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:09:08.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:09:08.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:09:08.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:09:08.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:09:08.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:09:08.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:09:08.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:09:08.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:09:08.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:09:08.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:09:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:09:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:08 smithi116 ceph-mon[35411]: pgmap v6564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:09 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:09:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:09:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:09:09.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:09:09.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:09 smithi114 ceph-mon[30820]: pgmap v6565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:09:09.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:09 smithi114 ceph-mon[30820]: pgmap v6566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:09:09.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:09:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:09 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:09:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:09:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:09 smithi116 ceph-mon[35411]: pgmap v6565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:09 smithi116 ceph-mon[35411]: pgmap v6566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:09:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:09:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:11 smithi114 ceph-mon[30820]: pgmap v6567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:09:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:11 smithi116 ceph-mon[35411]: pgmap v6567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:09:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:12 smithi114 ceph-mon[30820]: pgmap v6568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:09:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:12 smithi116 ceph-mon[35411]: pgmap v6568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:09:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:15 smithi114 ceph-mon[30820]: pgmap v6569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:09:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:15 smithi116 ceph-mon[35411]: pgmap v6569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:09:17.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:17 smithi114 ceph-mon[30820]: pgmap v6570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:09:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:17 smithi116 ceph-mon[35411]: pgmap v6570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:09:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:19 smithi114 ceph-mon[30820]: pgmap v6571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:09:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:19 smithi116 ceph-mon[35411]: pgmap v6571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:09:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:21 smithi114 ceph-mon[30820]: pgmap v6572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:09:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:21 smithi116 ceph-mon[35411]: pgmap v6572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:09:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:22 smithi114 ceph-mon[30820]: pgmap v6573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:09:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:22 smithi116 ceph-mon[35411]: pgmap v6573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:09:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:25 smithi114 ceph-mon[30820]: pgmap v6574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:25 smithi116 ceph-mon[35411]: pgmap v6574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:27 smithi114 ceph-mon[30820]: pgmap v6575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:27 smithi116 ceph-mon[35411]: pgmap v6575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:29.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:29 smithi114 ceph-mon[30820]: pgmap v6576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:29 smithi116 ceph-mon[35411]: pgmap v6576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:31 smithi114 ceph-mon[30820]: pgmap v6577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:09:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:31 smithi116 ceph-mon[35411]: pgmap v6577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:09:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:33 smithi114 ceph-mon[30820]: pgmap v6578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:09:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:33 smithi116 ceph-mon[35411]: pgmap v6578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:09:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:35 smithi114 ceph-mon[30820]: pgmap v6579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:35 smithi116 ceph-mon[35411]: pgmap v6579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:37 smithi114 ceph-mon[30820]: pgmap v6580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:37 smithi116 ceph-mon[35411]: pgmap v6580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:39 smithi114 ceph-mon[30820]: pgmap v6581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:39 smithi116 ceph-mon[35411]: pgmap v6581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:41 smithi114 ceph-mon[30820]: pgmap v6582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:09:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:41 smithi116 ceph-mon[35411]: pgmap v6582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:09:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:43 smithi114 ceph-mon[30820]: pgmap v6583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:09:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:43 smithi116 ceph-mon[35411]: pgmap v6583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:09:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:45 smithi114 ceph-mon[30820]: pgmap v6584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:45 smithi116 ceph-mon[35411]: pgmap v6584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:47 smithi114 ceph-mon[30820]: pgmap v6585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:47 smithi116 ceph-mon[35411]: pgmap v6585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:49 smithi114 ceph-mon[30820]: pgmap v6586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:49 smithi116 ceph-mon[35411]: pgmap v6586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:51 smithi114 ceph-mon[30820]: pgmap v6587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:09:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:51 smithi116 ceph-mon[35411]: pgmap v6587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:09:53.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:53 smithi114 ceph-mon[30820]: pgmap v6588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:09:53.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:53 smithi116 ceph-mon[35411]: pgmap v6588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:09:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:55 smithi114 ceph-mon[30820]: pgmap v6589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:55 smithi116 ceph-mon[35411]: pgmap v6589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:57 smithi114 ceph-mon[30820]: pgmap v6590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:57 smithi116 ceph-mon[35411]: pgmap v6590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:09:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:09:59 smithi114 ceph-mon[30820]: pgmap v6591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:09:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:09:59 smithi116 ceph-mon[35411]: pgmap v6591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:10:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:10:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:01 smithi114 ceph-mon[30820]: pgmap v6592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:10:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:01 smithi116 ceph-mon[35411]: pgmap v6592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:10:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:03 smithi114 ceph-mon[30820]: pgmap v6593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:10:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:03 smithi116 ceph-mon[35411]: pgmap v6593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:10:05.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:05 smithi114 ceph-mon[30820]: pgmap v6594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:05 smithi116 ceph-mon[35411]: pgmap v6594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:07 smithi114 ceph-mon[30820]: pgmap v6595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:07 smithi116 ceph-mon[35411]: pgmap v6595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:09 smithi114 ceph-mon[30820]: pgmap v6596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:10:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:09 smithi116 ceph-mon[35411]: pgmap v6596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:10:11.543 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-mon[30820]: pgmap v6597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:10:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:11 smithi116 ceph-mon[35411]: pgmap v6597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:10:11.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:10:11.540+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:10:11.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:10:11.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:10:11.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:10:11.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:10:11.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:10:11.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:10:11.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:10:11.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:10:11.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:10:11.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:10:11.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:10:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:12 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:10:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:10:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:10:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:10:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:10:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:10:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:12 smithi114 ceph-mon[30820]: pgmap v6598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:10:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:12 smithi114 ceph-mon[30820]: pgmap v6599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:10:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:10:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:12 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:10:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:10:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:10:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:10:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:10:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:12 smithi116 ceph-mon[35411]: pgmap v6598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:12 smithi116 ceph-mon[35411]: pgmap v6599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:10:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:10:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:13 smithi114 ceph-mon[30820]: pgmap v6600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:10:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:13 smithi116 ceph-mon[35411]: pgmap v6600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:10:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:15 smithi114 ceph-mon[30820]: pgmap v6601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:10:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:15 smithi116 ceph-mon[35411]: pgmap v6601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:10:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:17 smithi114 ceph-mon[30820]: pgmap v6602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:10:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:17 smithi116 ceph-mon[35411]: pgmap v6602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:10:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:19 smithi114 ceph-mon[30820]: pgmap v6603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:10:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:19 smithi116 ceph-mon[35411]: pgmap v6603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:10:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:21 smithi114 ceph-mon[30820]: pgmap v6604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:10:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:21 smithi116 ceph-mon[35411]: pgmap v6604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:10:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:23 smithi114 ceph-mon[30820]: pgmap v6605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:23 smithi116 ceph-mon[35411]: pgmap v6605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:25 smithi114 ceph-mon[30820]: pgmap v6606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:25 smithi116 ceph-mon[35411]: pgmap v6606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:27 smithi114 ceph-mon[30820]: pgmap v6607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:10:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:27 smithi116 ceph-mon[35411]: pgmap v6607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:10:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:29 smithi114 ceph-mon[30820]: pgmap v6608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:10:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:29 smithi116 ceph-mon[35411]: pgmap v6608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:10:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:31 smithi114 ceph-mon[30820]: pgmap v6609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:10:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:31 smithi116 ceph-mon[35411]: pgmap v6609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:10:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:33 smithi114 ceph-mon[30820]: pgmap v6610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:10:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:33 smithi116 ceph-mon[35411]: pgmap v6610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:10:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:35 smithi114 ceph-mon[30820]: pgmap v6611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:35 smithi116 ceph-mon[35411]: pgmap v6611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:37 smithi114 ceph-mon[30820]: pgmap v6612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:37 smithi116 ceph-mon[35411]: pgmap v6612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:39 smithi114 ceph-mon[30820]: pgmap v6613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:10:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:39 smithi116 ceph-mon[35411]: pgmap v6613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:10:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:41 smithi114 ceph-mon[30820]: pgmap v6614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:10:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:41 smithi116 ceph-mon[35411]: pgmap v6614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:10:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:43 smithi114 ceph-mon[30820]: pgmap v6615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:43 smithi116 ceph-mon[35411]: pgmap v6615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:45 smithi114 ceph-mon[30820]: pgmap v6616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:45 smithi116 ceph-mon[35411]: pgmap v6616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:47 smithi114 ceph-mon[30820]: pgmap v6617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:47 smithi116 ceph-mon[35411]: pgmap v6617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:49 smithi114 ceph-mon[30820]: pgmap v6618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:10:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:49 smithi116 ceph-mon[35411]: pgmap v6618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:10:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:51 smithi114 ceph-mon[30820]: pgmap v6619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:10:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:51 smithi116 ceph-mon[35411]: pgmap v6619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:10:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:53 smithi114 ceph-mon[30820]: pgmap v6620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:53 smithi116 ceph-mon[35411]: pgmap v6620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:55 smithi114 ceph-mon[30820]: pgmap v6621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:55 smithi116 ceph-mon[35411]: pgmap v6621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:10:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:57 smithi114 ceph-mon[30820]: pgmap v6622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:57 smithi116 ceph-mon[35411]: pgmap v6622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:10:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:10:59 smithi114 ceph-mon[30820]: pgmap v6623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:10:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:10:59 smithi116 ceph-mon[35411]: pgmap v6623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:01 smithi114 ceph-mon[30820]: pgmap v6624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:01 smithi116 ceph-mon[35411]: pgmap v6624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:03 smithi114 ceph-mon[30820]: pgmap v6625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:03 smithi116 ceph-mon[35411]: pgmap v6625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:05.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:05 smithi114 ceph-mon[30820]: pgmap v6626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:11:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:05 smithi116 ceph-mon[35411]: pgmap v6626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:11:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:07 smithi114 ceph-mon[30820]: pgmap v6627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:07 smithi116 ceph-mon[35411]: pgmap v6627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:09 smithi114 ceph-mon[30820]: pgmap v6628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:09 smithi116 ceph-mon[35411]: pgmap v6628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:11 smithi114 ceph-mon[30820]: pgmap v6629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:11 smithi116 ceph-mon[35411]: pgmap v6629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:11:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:11:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-mon[30820]: pgmap v6630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:14 smithi116 ceph-mon[35411]: pgmap v6630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:15.161 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:11:14.900+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:11:15.161 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:11:15.161 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:11:15.162 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:11:15.163 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:11:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:11:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:11:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:11:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:11:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:11:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:11:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:16 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:11:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:11:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:11:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:11:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:11:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:11:16.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:11:16.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:11:16.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:11:16.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:11:16.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:11:16.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:16 smithi114 ceph-mon[30820]: pgmap v6631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:11:16.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:16 smithi114 ceph-mon[30820]: pgmap v6632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:16 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:11:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:11:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:11:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:16 smithi116 ceph-mon[35411]: pgmap v6631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:11:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:16 smithi116 ceph-mon[35411]: pgmap v6632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:11:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:18 smithi114 ceph-mon[30820]: pgmap v6633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:11:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:18 smithi116 ceph-mon[35411]: pgmap v6633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:11:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:20 smithi116 ceph-mon[35411]: pgmap v6634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:11:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:20 smithi114 ceph-mon[30820]: pgmap v6634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:11:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:22 smithi116 ceph-mon[35411]: pgmap v6635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:11:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:22 smithi114 ceph-mon[30820]: pgmap v6635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:11:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:24 smithi116 ceph-mon[35411]: pgmap v6636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:11:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:24 smithi114 ceph-mon[30820]: pgmap v6636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:11:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:26 smithi114 ceph-mon[30820]: pgmap v6637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:11:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:26 smithi116 ceph-mon[35411]: pgmap v6637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:11:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:28 smithi114 ceph-mon[30820]: pgmap v6638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:28 smithi116 ceph-mon[35411]: pgmap v6638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:30 smithi114 ceph-mon[30820]: pgmap v6639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:30 smithi116 ceph-mon[35411]: pgmap v6639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:32 smithi114 ceph-mon[30820]: pgmap v6640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:11:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:32 smithi116 ceph-mon[35411]: pgmap v6640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:11:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:34 smithi114 ceph-mon[30820]: pgmap v6641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:34 smithi116 ceph-mon[35411]: pgmap v6641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:36 smithi114 ceph-mon[30820]: pgmap v6642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:36 smithi116 ceph-mon[35411]: pgmap v6642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:38 smithi114 ceph-mon[30820]: pgmap v6643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:38 smithi116 ceph-mon[35411]: pgmap v6643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:40 smithi114 ceph-mon[30820]: pgmap v6644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:40 smithi116 ceph-mon[35411]: pgmap v6644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:42 smithi114 ceph-mon[30820]: pgmap v6645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:11:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:42 smithi116 ceph-mon[35411]: pgmap v6645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:11:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:44 smithi114 ceph-mon[30820]: pgmap v6646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:44 smithi116 ceph-mon[35411]: pgmap v6646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:46 smithi114 ceph-mon[30820]: pgmap v6647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:46 smithi116 ceph-mon[35411]: pgmap v6647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:48 smithi114 ceph-mon[30820]: pgmap v6648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:48 smithi116 ceph-mon[35411]: pgmap v6648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:50 smithi114 ceph-mon[30820]: pgmap v6649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:50 smithi116 ceph-mon[35411]: pgmap v6649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:52 smithi114 ceph-mon[30820]: pgmap v6650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:11:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:52 smithi116 ceph-mon[35411]: pgmap v6650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:11:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:54 smithi114 ceph-mon[30820]: pgmap v6651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:54 smithi116 ceph-mon[35411]: pgmap v6651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:11:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:56 smithi114 ceph-mon[30820]: pgmap v6652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:56 smithi116 ceph-mon[35411]: pgmap v6652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:11:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:11:58 smithi114 ceph-mon[30820]: pgmap v6653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:11:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:11:58 smithi116 ceph-mon[35411]: pgmap v6653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:00 smithi114 ceph-mon[30820]: pgmap v6654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:00 smithi116 ceph-mon[35411]: pgmap v6654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:02 smithi114 ceph-mon[30820]: pgmap v6655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:12:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:02 smithi116 ceph-mon[35411]: pgmap v6655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:12:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:04 smithi114 ceph-mon[30820]: pgmap v6656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:04 smithi116 ceph-mon[35411]: pgmap v6656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:06 smithi114 ceph-mon[30820]: pgmap v6657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:06 smithi116 ceph-mon[35411]: pgmap v6657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:08 smithi114 ceph-mon[30820]: pgmap v6658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:08 smithi116 ceph-mon[35411]: pgmap v6658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:10 smithi114 ceph-mon[30820]: pgmap v6659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:10 smithi116 ceph-mon[35411]: pgmap v6659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:12 smithi114 ceph-mon[30820]: pgmap v6660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:12:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:12 smithi116 ceph-mon[35411]: pgmap v6660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:12:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:14 smithi114 ceph-mon[30820]: pgmap v6661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:14 smithi116 ceph-mon[35411]: pgmap v6661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:12:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:12:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:16 smithi114 ceph-mon[30820]: pgmap v6662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:16 smithi116 ceph-mon[35411]: pgmap v6662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-mon[30820]: pgmap v6663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:12:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:12:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:12:18.260+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:12:18.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:12:18.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:12:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:12:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:18 smithi116 ceph-mon[35411]: pgmap v6663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:12:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:12:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:12:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:12:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:12:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:12:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:12:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:12:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:12:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:12:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:12:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:12:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:12:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:12:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:19 smithi114 ceph-mon[30820]: pgmap v6664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:12:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:19 smithi114 ceph-mon[30820]: pgmap v6665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:12:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:12:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:12:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:12:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:12:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:12:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:12:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:12:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:12:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:12:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:12:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:12:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:19 smithi116 ceph-mon[35411]: pgmap v6664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:12:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:19 smithi116 ceph-mon[35411]: pgmap v6665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:12:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:21 smithi114 ceph-mon[30820]: pgmap v6666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:12:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:21 smithi116 ceph-mon[35411]: pgmap v6666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:12:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:23 smithi114 ceph-mon[30820]: pgmap v6667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:12:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:23 smithi116 ceph-mon[35411]: pgmap v6667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:12:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:25 smithi114 ceph-mon[30820]: pgmap v6668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:12:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:25 smithi116 ceph-mon[35411]: pgmap v6668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:12:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:27 smithi114 ceph-mon[30820]: pgmap v6669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:12:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:27 smithi116 ceph-mon[35411]: pgmap v6669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:12:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:29 smithi114 ceph-mon[30820]: pgmap v6670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:12:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:29 smithi116 ceph-mon[35411]: pgmap v6670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:12:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:31 smithi114 ceph-mon[30820]: pgmap v6671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:12:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:31 smithi116 ceph-mon[35411]: pgmap v6671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:12:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:33 smithi114 ceph-mon[30820]: pgmap v6672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:12:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:33 smithi116 ceph-mon[35411]: pgmap v6672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:12:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:35 smithi114 ceph-mon[30820]: pgmap v6673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:35 smithi116 ceph-mon[35411]: pgmap v6673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:37 smithi114 ceph-mon[30820]: pgmap v6674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:37 smithi116 ceph-mon[35411]: pgmap v6674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:39 smithi114 ceph-mon[30820]: pgmap v6675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:39 smithi116 ceph-mon[35411]: pgmap v6675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:41 smithi114 ceph-mon[30820]: pgmap v6676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:12:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:41 smithi116 ceph-mon[35411]: pgmap v6676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:12:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:43 smithi114 ceph-mon[30820]: pgmap v6677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:12:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:43 smithi116 ceph-mon[35411]: pgmap v6677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:12:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:45 smithi114 ceph-mon[30820]: pgmap v6678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:45 smithi116 ceph-mon[35411]: pgmap v6678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:47 smithi114 ceph-mon[30820]: pgmap v6679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:47 smithi116 ceph-mon[35411]: pgmap v6679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:49 smithi114 ceph-mon[30820]: pgmap v6680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:49 smithi116 ceph-mon[35411]: pgmap v6680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:51 smithi114 ceph-mon[30820]: pgmap v6681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:12:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:51 smithi116 ceph-mon[35411]: pgmap v6681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:12:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:53 smithi114 ceph-mon[30820]: pgmap v6682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:12:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:53 smithi116 ceph-mon[35411]: pgmap v6682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:12:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:55 smithi114 ceph-mon[30820]: pgmap v6683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:55 smithi116 ceph-mon[35411]: pgmap v6683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:12:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:57 smithi116 ceph-mon[35411]: pgmap v6684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:57 smithi114 ceph-mon[30820]: pgmap v6684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:12:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:12:59 smithi116 ceph-mon[35411]: pgmap v6685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:12:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:12:59 smithi114 ceph-mon[30820]: pgmap v6685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:01 smithi116 ceph-mon[35411]: pgmap v6686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:13:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:01 smithi114 ceph-mon[30820]: pgmap v6686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:13:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:03 smithi114 ceph-mon[30820]: pgmap v6687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:13:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:03 smithi116 ceph-mon[35411]: pgmap v6687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:13:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:05 smithi114 ceph-mon[30820]: pgmap v6688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:13:05.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:05 smithi116 ceph-mon[35411]: pgmap v6688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:13:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:07 smithi114 ceph-mon[30820]: pgmap v6689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:13:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:07 smithi116 ceph-mon[35411]: pgmap v6689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:13:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:09 smithi114 ceph-mon[30820]: pgmap v6690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:09 smithi116 ceph-mon[35411]: pgmap v6690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:11 smithi114 ceph-mon[30820]: pgmap v6691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:13:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:11 smithi116 ceph-mon[35411]: pgmap v6691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:13:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:13 smithi114 ceph-mon[30820]: pgmap v6692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:13:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:13 smithi116 ceph-mon[35411]: pgmap v6692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:13:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:15 smithi114 ceph-mon[30820]: pgmap v6693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:13:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:15 smithi116 ceph-mon[35411]: pgmap v6693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:13:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:17 smithi114 ceph-mon[30820]: pgmap v6694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:13:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:17 smithi116 ceph-mon[35411]: pgmap v6694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:13:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:19 smithi114 ceph-mon[30820]: pgmap v6695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:13:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:19 smithi116 ceph-mon[35411]: pgmap v6695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:13:21.623 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-mon[30820]: pgmap v6696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:13:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:21 smithi116 ceph-mon[35411]: pgmap v6696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:13:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:13:21.621+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:13:21.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:13:21.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:13:21.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:13:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:13:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:13:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:13:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:13:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:13:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:13:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:13:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:23 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:13:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:13:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:13:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:23 smithi114 ceph-mon[30820]: pgmap v6697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:13:23.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:23 smithi114 ceph-mon[30820]: pgmap v6698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:23 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:13:23.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:13:23.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:13:23.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:23 smithi116 ceph-mon[35411]: pgmap v6697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:13:23.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:23 smithi116 ceph-mon[35411]: pgmap v6698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:13:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:25 smithi114 ceph-mon[30820]: pgmap v6699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:13:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:25 smithi116 ceph-mon[35411]: pgmap v6699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:13:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:27 smithi114 ceph-mon[30820]: pgmap v6700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:13:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:27 smithi116 ceph-mon[35411]: pgmap v6700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:13:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:29 smithi114 ceph-mon[30820]: pgmap v6701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:13:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:29 smithi116 ceph-mon[35411]: pgmap v6701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:13:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:31 smithi114 ceph-mon[30820]: pgmap v6702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:13:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:31 smithi116 ceph-mon[35411]: pgmap v6702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:13:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:33 smithi114 ceph-mon[30820]: pgmap v6703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:13:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:33 smithi116 ceph-mon[35411]: pgmap v6703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:13:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:35 smithi114 ceph-mon[30820]: pgmap v6704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:35 smithi116 ceph-mon[35411]: pgmap v6704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:37 smithi114 ceph-mon[30820]: pgmap v6705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:13:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:37 smithi116 ceph-mon[35411]: pgmap v6705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:13:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:39 smithi114 ceph-mon[30820]: pgmap v6706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:39 smithi116 ceph-mon[35411]: pgmap v6706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:41 smithi114 ceph-mon[30820]: pgmap v6707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:13:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:41 smithi116 ceph-mon[35411]: pgmap v6707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:13:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:43 smithi114 ceph-mon[30820]: pgmap v6708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:13:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:43 smithi116 ceph-mon[35411]: pgmap v6708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:13:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:45 smithi116 ceph-mon[35411]: pgmap v6709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:45 smithi114 ceph-mon[30820]: pgmap v6709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:47 smithi116 ceph-mon[35411]: pgmap v6710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:13:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:47 smithi114 ceph-mon[30820]: pgmap v6710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:13:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:49 smithi116 ceph-mon[35411]: pgmap v6711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:49.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:49 smithi114 ceph-mon[30820]: pgmap v6711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:51 smithi114 ceph-mon[30820]: pgmap v6712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:13:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:51 smithi116 ceph-mon[35411]: pgmap v6712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:13:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:53 smithi114 ceph-mon[30820]: pgmap v6713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:13:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:53 smithi116 ceph-mon[35411]: pgmap v6713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:13:55.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:55 smithi116 ceph-mon[35411]: pgmap v6714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:55 smithi114 ceph-mon[30820]: pgmap v6714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:57 smithi114 ceph-mon[30820]: pgmap v6715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:13:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:57 smithi116 ceph-mon[35411]: pgmap v6715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:13:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:13:59 smithi114 ceph-mon[30820]: pgmap v6716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:13:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:13:59 smithi116 ceph-mon[35411]: pgmap v6716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:01 smithi114 ceph-mon[30820]: pgmap v6717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:01 smithi116 ceph-mon[35411]: pgmap v6717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:03 smithi114 ceph-mon[30820]: pgmap v6718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:03 smithi116 ceph-mon[35411]: pgmap v6718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:05 smithi114 ceph-mon[30820]: pgmap v6719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:05 smithi116 ceph-mon[35411]: pgmap v6719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:07 smithi114 ceph-mon[30820]: pgmap v6720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:14:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:07 smithi116 ceph-mon[35411]: pgmap v6720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:14:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:09 smithi114 ceph-mon[30820]: pgmap v6721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:09 smithi116 ceph-mon[35411]: pgmap v6721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:11 smithi114 ceph-mon[30820]: pgmap v6722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:11 smithi116 ceph-mon[35411]: pgmap v6722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:13 smithi114 ceph-mon[30820]: pgmap v6723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:13 smithi116 ceph-mon[35411]: pgmap v6723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:15 smithi114 ceph-mon[30820]: pgmap v6724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:15 smithi116 ceph-mon[35411]: pgmap v6724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:17 smithi114 ceph-mon[30820]: pgmap v6725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:14:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:17 smithi116 ceph-mon[35411]: pgmap v6725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:14:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:19 smithi114 ceph-mon[30820]: pgmap v6726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:19 smithi116 ceph-mon[35411]: pgmap v6726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:21 smithi114 ceph-mon[30820]: pgmap v6727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:21 smithi116 ceph-mon[35411]: pgmap v6727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:14:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:14:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:23 smithi114 ceph-mon[30820]: pgmap v6728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:23 smithi116 ceph-mon[35411]: pgmap v6728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:25.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:14:24.981+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:14:25.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:14:25.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:14:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:14:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:14:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:14:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:14:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:14:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:14:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:14:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:14:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:14:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:25 smithi114 ceph-mon[30820]: pgmap v6729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:14:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:14:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:14:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:25 smithi116 ceph-mon[35411]: pgmap v6729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:14:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:14:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:14:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:14:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:14:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:26 smithi114 ceph-mon[30820]: pgmap v6730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:14:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:26 smithi114 ceph-mon[30820]: pgmap v6731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:26 smithi116 ceph-mon[35411]: pgmap v6730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:14:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:26 smithi116 ceph-mon[35411]: pgmap v6731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:14:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:28 smithi114 ceph-mon[30820]: pgmap v6732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:14:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:28 smithi116 ceph-mon[35411]: pgmap v6732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:14:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:30 smithi114 ceph-mon[30820]: pgmap v6733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:14:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:30 smithi116 ceph-mon[35411]: pgmap v6733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:14:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:32 smithi114 ceph-mon[30820]: pgmap v6734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:14:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:32 smithi116 ceph-mon[35411]: pgmap v6734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:14:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:34 smithi114 ceph-mon[30820]: pgmap v6735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:14:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:34 smithi116 ceph-mon[35411]: pgmap v6735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:14:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:36 smithi114 ceph-mon[30820]: pgmap v6736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:14:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:36 smithi116 ceph-mon[35411]: pgmap v6736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:14:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:38 smithi114 ceph-mon[30820]: pgmap v6737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:38 smithi116 ceph-mon[35411]: pgmap v6737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:40 smithi114 ceph-mon[30820]: pgmap v6738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:40 smithi116 ceph-mon[35411]: pgmap v6738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:42 smithi114 ceph-mon[30820]: pgmap v6739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:14:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:42 smithi116 ceph-mon[35411]: pgmap v6739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:14:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:44 smithi114 ceph-mon[30820]: pgmap v6740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:44 smithi116 ceph-mon[35411]: pgmap v6740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:46 smithi114 ceph-mon[30820]: pgmap v6741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:46 smithi116 ceph-mon[35411]: pgmap v6741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:48 smithi114 ceph-mon[30820]: pgmap v6742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:48 smithi116 ceph-mon[35411]: pgmap v6742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:50 smithi114 ceph-mon[30820]: pgmap v6743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:50 smithi116 ceph-mon[35411]: pgmap v6743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:52 smithi114 ceph-mon[30820]: pgmap v6744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:14:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:52 smithi116 ceph-mon[35411]: pgmap v6744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:14:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:54 smithi114 ceph-mon[30820]: pgmap v6745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:54 smithi116 ceph-mon[35411]: pgmap v6745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:14:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:56 smithi114 ceph-mon[30820]: pgmap v6746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:56 smithi116 ceph-mon[35411]: pgmap v6746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:14:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:14:58 smithi114 ceph-mon[30820]: pgmap v6747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:14:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:14:58 smithi116 ceph-mon[35411]: pgmap v6747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:00 smithi114 ceph-mon[30820]: pgmap v6748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:00 smithi116 ceph-mon[35411]: pgmap v6748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:02 smithi114 ceph-mon[30820]: pgmap v6749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:15:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:02 smithi116 ceph-mon[35411]: pgmap v6749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:15:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:04 smithi114 ceph-mon[30820]: pgmap v6750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:04 smithi116 ceph-mon[35411]: pgmap v6750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:06 smithi114 ceph-mon[30820]: pgmap v6751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:06 smithi116 ceph-mon[35411]: pgmap v6751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:08 smithi114 ceph-mon[30820]: pgmap v6752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:08 smithi116 ceph-mon[35411]: pgmap v6752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:10 smithi114 ceph-mon[30820]: pgmap v6753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:10 smithi116 ceph-mon[35411]: pgmap v6753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:12 smithi114 ceph-mon[30820]: pgmap v6754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:15:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:12 smithi116 ceph-mon[35411]: pgmap v6754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:15:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:14 smithi114 ceph-mon[30820]: pgmap v6755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:14 smithi116 ceph-mon[35411]: pgmap v6755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:16 smithi114 ceph-mon[30820]: pgmap v6756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:16 smithi116 ceph-mon[35411]: pgmap v6756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:18 smithi114 ceph-mon[30820]: pgmap v6757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:18 smithi116 ceph-mon[35411]: pgmap v6757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:20 smithi114 ceph-mon[30820]: pgmap v6758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:20 smithi116 ceph-mon[35411]: pgmap v6758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:22 smithi114 ceph-mon[30820]: pgmap v6759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:15:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:22 smithi116 ceph-mon[35411]: pgmap v6759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:15:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:24 smithi114 ceph-mon[30820]: pgmap v6760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:24 smithi116 ceph-mon[35411]: pgmap v6760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:15:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:15:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:26 smithi114 ceph-mon[30820]: pgmap v6761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:15:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:15:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:26 smithi116 ceph-mon[35411]: pgmap v6761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:15:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:15:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:27 smithi114 ceph-mon[30820]: pgmap v6762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:27 smithi116 ceph-mon[35411]: pgmap v6762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:28.642 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:15:28.340+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:15:28.642 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:15:28.642 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:15:28.643 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:15:28.644 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:15:28.644 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:15:28.644 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:15:28.644 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:15:28.644 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:15:28.644 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:15:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:15:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:15:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:15:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:15:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:15:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:15:28.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-mon[30820]: pgmap v6763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-mon[30820]: pgmap v6764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:15:28.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:15:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:15:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:15:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:15:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:15:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:15:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:28 smithi116 ceph-mon[35411]: pgmap v6763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:15:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:28 smithi116 ceph-mon[35411]: pgmap v6764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:15:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:15:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:31 smithi114 ceph-mon[30820]: pgmap v6765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:15:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:31 smithi116 ceph-mon[35411]: pgmap v6765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:15:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:33 smithi114 ceph-mon[30820]: pgmap v6766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:15:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:33 smithi116 ceph-mon[35411]: pgmap v6766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:15:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:35 smithi114 ceph-mon[30820]: pgmap v6767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:15:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:35 smithi116 ceph-mon[35411]: pgmap v6767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:15:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:37 smithi114 ceph-mon[30820]: pgmap v6768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:15:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:37 smithi116 ceph-mon[35411]: pgmap v6768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:15:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:39 smithi116 ceph-mon[35411]: pgmap v6769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:15:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:39 smithi114 ceph-mon[30820]: pgmap v6769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:15:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:41 smithi116 ceph-mon[35411]: pgmap v6770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:15:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:41 smithi114 ceph-mon[30820]: pgmap v6770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:15:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:43 smithi116 ceph-mon[35411]: pgmap v6771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:15:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:43 smithi114 ceph-mon[30820]: pgmap v6771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:15:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:45 smithi116 ceph-mon[35411]: pgmap v6772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:45 smithi114 ceph-mon[30820]: pgmap v6772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:47 smithi114 ceph-mon[30820]: pgmap v6773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:47 smithi116 ceph-mon[35411]: pgmap v6773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:49 smithi114 ceph-mon[30820]: pgmap v6774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:49 smithi116 ceph-mon[35411]: pgmap v6774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:51.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:51 smithi114 ceph-mon[30820]: pgmap v6775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:15:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:51 smithi116 ceph-mon[35411]: pgmap v6775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:15:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:53 smithi114 ceph-mon[30820]: pgmap v6776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:15:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:53 smithi116 ceph-mon[35411]: pgmap v6776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:15:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:55 smithi114 ceph-mon[30820]: pgmap v6777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:55 smithi116 ceph-mon[35411]: pgmap v6777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:15:57.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:57 smithi116 ceph-mon[35411]: pgmap v6778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:57 smithi114 ceph-mon[30820]: pgmap v6778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:15:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:15:59 smithi114 ceph-mon[30820]: pgmap v6779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:15:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:15:59 smithi116 ceph-mon[35411]: pgmap v6779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:01 smithi114 ceph-mon[30820]: pgmap v6780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:16:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:01 smithi116 ceph-mon[35411]: pgmap v6780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:16:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:03 smithi114 ceph-mon[30820]: pgmap v6781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:16:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:03 smithi116 ceph-mon[35411]: pgmap v6781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:16:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:05 smithi114 ceph-mon[30820]: pgmap v6782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:16:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:05 smithi116 ceph-mon[35411]: pgmap v6782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:16:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:07 smithi114 ceph-mon[30820]: pgmap v6783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:16:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:07 smithi116 ceph-mon[35411]: pgmap v6783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:16:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:09 smithi114 ceph-mon[30820]: pgmap v6784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:09 smithi116 ceph-mon[35411]: pgmap v6784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:11 smithi114 ceph-mon[30820]: pgmap v6785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:16:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:11 smithi116 ceph-mon[35411]: pgmap v6785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:16:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:13 smithi114 ceph-mon[30820]: pgmap v6786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:16:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:13 smithi116 ceph-mon[35411]: pgmap v6786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:16:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:15 smithi114 ceph-mon[30820]: pgmap v6787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:16:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:15 smithi116 ceph-mon[35411]: pgmap v6787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:16:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:17 smithi114 ceph-mon[30820]: pgmap v6788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:16:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:17 smithi116 ceph-mon[35411]: pgmap v6788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:16:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:19 smithi114 ceph-mon[30820]: pgmap v6789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:19 smithi116 ceph-mon[35411]: pgmap v6789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:21 smithi114 ceph-mon[30820]: pgmap v6790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:16:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:21 smithi116 ceph-mon[35411]: pgmap v6790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:16:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:23 smithi114 ceph-mon[30820]: pgmap v6791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:16:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:23 smithi116 ceph-mon[35411]: pgmap v6791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:16:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:25 smithi114 ceph-mon[30820]: pgmap v6792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:16:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:25 smithi116 ceph-mon[35411]: pgmap v6792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:16:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:27 smithi114 ceph-mon[30820]: pgmap v6793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:16:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:27 smithi116 ceph-mon[35411]: pgmap v6793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:16:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:16:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:16:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:29 smithi114 ceph-mon[30820]: pgmap v6794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:16:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:16:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:29 smithi116 ceph-mon[35411]: pgmap v6794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:16:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:16:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-mon[30820]: pgmap v6795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:16:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:16:31.700+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:16:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:16:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:16:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:16:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:16:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:16:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:16:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:16:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:16:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:16:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:16:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:16:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:31 smithi116 ceph-mon[35411]: pgmap v6795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:16:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:16:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:16:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:16:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:16:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:16:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:33 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:16:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:16:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:16:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:16:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:33 smithi114 ceph-mon[30820]: pgmap v6796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:16:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:33 smithi114 ceph-mon[30820]: pgmap v6797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:16:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:33 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:16:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:16:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:16:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:16:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:16:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:16:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:16:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:16:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:16:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:16:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:16:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:33 smithi116 ceph-mon[35411]: pgmap v6796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:16:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:33 smithi116 ceph-mon[35411]: pgmap v6797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:16:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:35 smithi114 ceph-mon[30820]: pgmap v6798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:16:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:35 smithi116 ceph-mon[35411]: pgmap v6798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:16:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:37 smithi114 ceph-mon[30820]: pgmap v6799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:16:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:37 smithi116 ceph-mon[35411]: pgmap v6799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:16:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:39 smithi114 ceph-mon[30820]: pgmap v6800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:16:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:39 smithi116 ceph-mon[35411]: pgmap v6800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:16:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:41 smithi114 ceph-mon[30820]: pgmap v6801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:16:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:41 smithi116 ceph-mon[35411]: pgmap v6801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:16:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:43 smithi114 ceph-mon[30820]: pgmap v6802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:16:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:43 smithi116 ceph-mon[35411]: pgmap v6802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:16:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:45 smithi114 ceph-mon[30820]: pgmap v6803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:45 smithi116 ceph-mon[35411]: pgmap v6803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:47 smithi114 ceph-mon[30820]: pgmap v6804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:16:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:47 smithi116 ceph-mon[35411]: pgmap v6804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:16:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:49 smithi114 ceph-mon[30820]: pgmap v6805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:49 smithi116 ceph-mon[35411]: pgmap v6805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:51 smithi114 ceph-mon[30820]: pgmap v6806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:16:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:51 smithi116 ceph-mon[35411]: pgmap v6806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:16:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:53 smithi114 ceph-mon[30820]: pgmap v6807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:16:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:53 smithi116 ceph-mon[35411]: pgmap v6807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:16:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:55 smithi114 ceph-mon[30820]: pgmap v6808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:55 smithi116 ceph-mon[35411]: pgmap v6808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:57 smithi114 ceph-mon[30820]: pgmap v6809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:16:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:57 smithi116 ceph-mon[35411]: pgmap v6809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:16:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:16:59 smithi114 ceph-mon[30820]: pgmap v6810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:16:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:16:59 smithi116 ceph-mon[35411]: pgmap v6810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:01 smithi114 ceph-mon[30820]: pgmap v6811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:01 smithi116 ceph-mon[35411]: pgmap v6811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:03 smithi114 ceph-mon[30820]: pgmap v6812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:03 smithi116 ceph-mon[35411]: pgmap v6812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:05 smithi114 ceph-mon[30820]: pgmap v6813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:05 smithi116 ceph-mon[35411]: pgmap v6813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:07 smithi114 ceph-mon[30820]: pgmap v6814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:17:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:07 smithi116 ceph-mon[35411]: pgmap v6814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:17:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:09 smithi114 ceph-mon[30820]: pgmap v6815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:09 smithi116 ceph-mon[35411]: pgmap v6815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:11 smithi114 ceph-mon[30820]: pgmap v6816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:11 smithi116 ceph-mon[35411]: pgmap v6816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:13 smithi114 ceph-mon[30820]: pgmap v6817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:13 smithi116 ceph-mon[35411]: pgmap v6817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:15 smithi114 ceph-mon[30820]: pgmap v6818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:15 smithi116 ceph-mon[35411]: pgmap v6818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:17 smithi114 ceph-mon[30820]: pgmap v6819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:17:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:17 smithi116 ceph-mon[35411]: pgmap v6819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:17:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:19 smithi114 ceph-mon[30820]: pgmap v6820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:19 smithi116 ceph-mon[35411]: pgmap v6820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:21 smithi114 ceph-mon[30820]: pgmap v6821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:21 smithi116 ceph-mon[35411]: pgmap v6821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:23 smithi114 ceph-mon[30820]: pgmap v6822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:23 smithi116 ceph-mon[35411]: pgmap v6822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:25 smithi114 ceph-mon[30820]: pgmap v6823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:25 smithi116 ceph-mon[35411]: pgmap v6823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:27 smithi114 ceph-mon[30820]: pgmap v6824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:17:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:27 smithi116 ceph-mon[35411]: pgmap v6824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:17:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:29 smithi114 ceph-mon[30820]: pgmap v6825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:29 smithi116 ceph-mon[35411]: pgmap v6825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:31 smithi114 ceph-mon[30820]: pgmap v6826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:31 smithi116 ceph-mon[35411]: pgmap v6826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:17:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:17:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:33 smithi114 ceph-mon[30820]: pgmap v6827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:33 smithi116 ceph-mon[35411]: pgmap v6827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:17:34.997+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:17:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:17:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:17:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:17:35.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:17:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:17:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:17:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:17:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:17:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:17:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:17:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:35 smithi114 ceph-mon[30820]: pgmap v6828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:17:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:17:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:17:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:35 smithi116 ceph-mon[35411]: pgmap v6828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:17:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:17:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:17:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:36 smithi114 ceph-mon[30820]: pgmap v6829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T10:17:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:36 smithi114 ceph-mon[30820]: pgmap v6830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:17:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:36 smithi116 ceph-mon[35411]: pgmap v6829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T10:17:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:36 smithi116 ceph-mon[35411]: pgmap v6830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:17:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:38 smithi114 ceph-mon[30820]: pgmap v6831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:17:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:38 smithi116 ceph-mon[35411]: pgmap v6831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:17:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:39 smithi114 ceph-mon[30820]: pgmap v6832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:17:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:39 smithi116 ceph-mon[35411]: pgmap v6832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:17:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:42 smithi114 ceph-mon[30820]: pgmap v6833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:17:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:42 smithi116 ceph-mon[35411]: pgmap v6833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:17:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:44 smithi114 ceph-mon[30820]: pgmap v6834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:17:44.608 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:44 smithi116 ceph-mon[35411]: pgmap v6834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:17:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:46 smithi114 ceph-mon[30820]: pgmap v6835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:17:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:46 smithi116 ceph-mon[35411]: pgmap v6835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:17:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:48 smithi114 ceph-mon[30820]: pgmap v6836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:48 smithi116 ceph-mon[35411]: pgmap v6836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:50 smithi114 ceph-mon[30820]: pgmap v6837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:50 smithi116 ceph-mon[35411]: pgmap v6837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:52 smithi114 ceph-mon[30820]: pgmap v6838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:17:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:52 smithi116 ceph-mon[35411]: pgmap v6838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:17:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:54 smithi114 ceph-mon[30820]: pgmap v6839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:54 smithi116 ceph-mon[35411]: pgmap v6839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:17:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:56 smithi114 ceph-mon[30820]: pgmap v6840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:56 smithi116 ceph-mon[35411]: pgmap v6840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:17:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:17:58 smithi114 ceph-mon[30820]: pgmap v6841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:17:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:17:58 smithi116 ceph-mon[35411]: pgmap v6841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:00 smithi114 ceph-mon[30820]: pgmap v6842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:00 smithi116 ceph-mon[35411]: pgmap v6842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:02 smithi114 ceph-mon[30820]: pgmap v6843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:02 smithi116 ceph-mon[35411]: pgmap v6843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:04 smithi114 ceph-mon[30820]: pgmap v6844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:04 smithi116 ceph-mon[35411]: pgmap v6844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:06 smithi114 ceph-mon[30820]: pgmap v6845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:06 smithi116 ceph-mon[35411]: pgmap v6845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:08 smithi114 ceph-mon[30820]: pgmap v6846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:08 smithi116 ceph-mon[35411]: pgmap v6846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:10 smithi114 ceph-mon[30820]: pgmap v6847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:10 smithi116 ceph-mon[35411]: pgmap v6847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:12 smithi114 ceph-mon[30820]: pgmap v6848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:12 smithi116 ceph-mon[35411]: pgmap v6848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:14 smithi114 ceph-mon[30820]: pgmap v6849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:14 smithi116 ceph-mon[35411]: pgmap v6849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:16 smithi114 ceph-mon[30820]: pgmap v6850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:16 smithi116 ceph-mon[35411]: pgmap v6850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:18 smithi114 ceph-mon[30820]: pgmap v6851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:18 smithi116 ceph-mon[35411]: pgmap v6851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:20 smithi114 ceph-mon[30820]: pgmap v6852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:20 smithi116 ceph-mon[35411]: pgmap v6852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:22 smithi114 ceph-mon[30820]: pgmap v6853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:22 smithi116 ceph-mon[35411]: pgmap v6853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:24 smithi114 ceph-mon[30820]: pgmap v6854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:24 smithi116 ceph-mon[35411]: pgmap v6854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:26 smithi114 ceph-mon[30820]: pgmap v6855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:26 smithi116 ceph-mon[35411]: pgmap v6855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:28 smithi114 ceph-mon[30820]: pgmap v6856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:28 smithi116 ceph-mon[35411]: pgmap v6856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:30 smithi114 ceph-mon[30820]: pgmap v6857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:30 smithi116 ceph-mon[35411]: pgmap v6857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:32 smithi114 ceph-mon[30820]: pgmap v6858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:32 smithi116 ceph-mon[35411]: pgmap v6858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:34 smithi114 ceph-mon[30820]: pgmap v6859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:34 smithi116 ceph-mon[35411]: pgmap v6859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:18:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:18:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: pgmap v6860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:18:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:18:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: pgmap v6860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:18:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:37 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:18:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:37 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:18:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:37 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:18:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:37 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:18:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:38 smithi116 ceph-mon[35411]: pgmap v6861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-mon[30820]: pgmap v6861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:18:38.356+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:18:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:18:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:18:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:18:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:18:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:18:38.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:18:38.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:18:38.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:18:38.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:18:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:18:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:18:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:18:39.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:39 smithi114 ceph-mon[30820]: pgmap v6862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:18:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:39 smithi114 ceph-mon[30820]: pgmap v6863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:18:39.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:18:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:18:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:18:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:18:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:18:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:18:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:39 smithi116 ceph-mon[35411]: pgmap v6862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:39 smithi116 ceph-mon[35411]: pgmap v6863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:18:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:18:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:41 smithi116 ceph-mon[35411]: pgmap v6864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:18:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:41 smithi114 ceph-mon[30820]: pgmap v6864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:18:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:43 smithi116 ceph-mon[35411]: pgmap v6865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:18:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:43 smithi114 ceph-mon[30820]: pgmap v6865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:18:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:45 smithi116 ceph-mon[35411]: pgmap v6866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:18:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:45 smithi114 ceph-mon[30820]: pgmap v6866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:18:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:47 smithi114 ceph-mon[30820]: pgmap v6867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:18:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:47 smithi116 ceph-mon[35411]: pgmap v6867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:18:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:49 smithi114 ceph-mon[30820]: pgmap v6868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:18:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:49 smithi116 ceph-mon[35411]: pgmap v6868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:18:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:51 smithi114 ceph-mon[30820]: pgmap v6869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:51 smithi116 ceph-mon[35411]: pgmap v6869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:18:53.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:53 smithi114 ceph-mon[30820]: pgmap v6870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:53 smithi116 ceph-mon[35411]: pgmap v6870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:55 smithi114 ceph-mon[30820]: pgmap v6871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:55 smithi116 ceph-mon[35411]: pgmap v6871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:18:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:57 smithi114 ceph-mon[30820]: pgmap v6872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:57 smithi116 ceph-mon[35411]: pgmap v6872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:18:59.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:18:59 smithi116 ceph-mon[35411]: pgmap v6873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:18:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:18:59 smithi114 ceph-mon[30820]: pgmap v6873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:01 smithi114 ceph-mon[30820]: pgmap v6874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:01 smithi116 ceph-mon[35411]: pgmap v6874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:03 smithi114 ceph-mon[30820]: pgmap v6875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:03 smithi116 ceph-mon[35411]: pgmap v6875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:05 smithi114 ceph-mon[30820]: pgmap v6876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:19:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:05 smithi116 ceph-mon[35411]: pgmap v6876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:19:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:07 smithi114 ceph-mon[30820]: pgmap v6877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:19:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:07 smithi116 ceph-mon[35411]: pgmap v6877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:19:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:09 smithi114 ceph-mon[30820]: pgmap v6878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:09 smithi116 ceph-mon[35411]: pgmap v6878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:11 smithi114 ceph-mon[30820]: pgmap v6879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:11 smithi116 ceph-mon[35411]: pgmap v6879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:13 smithi114 ceph-mon[30820]: pgmap v6880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:13 smithi116 ceph-mon[35411]: pgmap v6880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:15 smithi114 ceph-mon[30820]: pgmap v6881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:19:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:15 smithi116 ceph-mon[35411]: pgmap v6881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:19:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:17 smithi114 ceph-mon[30820]: pgmap v6882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:19:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:17 smithi116 ceph-mon[35411]: pgmap v6882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:19:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:19 smithi114 ceph-mon[30820]: pgmap v6883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:19 smithi116 ceph-mon[35411]: pgmap v6883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:21 smithi114 ceph-mon[30820]: pgmap v6884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:21 smithi116 ceph-mon[35411]: pgmap v6884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:23 smithi114 ceph-mon[30820]: pgmap v6885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:23 smithi116 ceph-mon[35411]: pgmap v6885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:25 smithi114 ceph-mon[30820]: pgmap v6886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:19:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:25 smithi116 ceph-mon[35411]: pgmap v6886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:19:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:27 smithi114 ceph-mon[30820]: pgmap v6887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:19:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:27 smithi116 ceph-mon[35411]: pgmap v6887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:19:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:29 smithi114 ceph-mon[30820]: pgmap v6888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:29 smithi116 ceph-mon[35411]: pgmap v6888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:31 smithi114 ceph-mon[30820]: pgmap v6889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:31 smithi116 ceph-mon[35411]: pgmap v6889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:33 smithi114 ceph-mon[30820]: pgmap v6890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:33 smithi116 ceph-mon[35411]: pgmap v6890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:35 smithi114 ceph-mon[30820]: pgmap v6891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:19:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:35 smithi116 ceph-mon[35411]: pgmap v6891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:19:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:37 smithi114 ceph-mon[30820]: pgmap v6892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:19:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:37 smithi116 ceph-mon[35411]: pgmap v6892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:19:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:19:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:19:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:39 smithi114 ceph-mon[30820]: pgmap v6893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:39 smithi116 ceph-mon[35411]: pgmap v6893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-mon[30820]: pgmap v6894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:41.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:19:41.716+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:19:41.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:19:41.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:19:41.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:19:41.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:19:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:19:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:19:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:19:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:19:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:19:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:19:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:41 smithi116 ceph-mon[35411]: pgmap v6894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:19:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:19:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:19:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:19:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:19:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:19:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:43 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:43 smithi114 ceph-mon[30820]: pgmap v6895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:19:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:43 smithi114 ceph-mon[30820]: pgmap v6896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:19:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:43 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:19:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:19:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:19:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:19:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:19:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:19:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:19:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:19:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:19:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:19:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:43 smithi116 ceph-mon[35411]: pgmap v6895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:19:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:43 smithi116 ceph-mon[35411]: pgmap v6896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:19:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:45 smithi114 ceph-mon[30820]: pgmap v6897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:19:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:45 smithi116 ceph-mon[35411]: pgmap v6897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:19:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:47 smithi114 ceph-mon[30820]: pgmap v6898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:19:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:47 smithi116 ceph-mon[35411]: pgmap v6898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:19:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:49 smithi114 ceph-mon[30820]: pgmap v6899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:19:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:49 smithi116 ceph-mon[35411]: pgmap v6899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:19:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:51 smithi114 ceph-mon[30820]: pgmap v6900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:19:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:51 smithi116 ceph-mon[35411]: pgmap v6900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:19:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:53 smithi114 ceph-mon[30820]: pgmap v6901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:19:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:53 smithi116 ceph-mon[35411]: pgmap v6901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:19:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:55 smithi114 ceph-mon[30820]: pgmap v6902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:55 smithi116 ceph-mon[35411]: pgmap v6902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:57 smithi114 ceph-mon[30820]: pgmap v6903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:57 smithi116 ceph-mon[35411]: pgmap v6903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:19:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:19:59 smithi114 ceph-mon[30820]: pgmap v6904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:19:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:19:59 smithi116 ceph-mon[35411]: pgmap v6904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:20:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:20:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:01 smithi114 ceph-mon[30820]: pgmap v6905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:01 smithi116 ceph-mon[35411]: pgmap v6905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:03 smithi114 ceph-mon[30820]: pgmap v6906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:03 smithi116 ceph-mon[35411]: pgmap v6906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:05 smithi114 ceph-mon[30820]: pgmap v6907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:05 smithi116 ceph-mon[35411]: pgmap v6907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:07 smithi114 ceph-mon[30820]: pgmap v6908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:20:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:07 smithi116 ceph-mon[35411]: pgmap v6908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:20:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:09 smithi114 ceph-mon[30820]: pgmap v6909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:09 smithi116 ceph-mon[35411]: pgmap v6909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:11 smithi114 ceph-mon[30820]: pgmap v6910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:11 smithi116 ceph-mon[35411]: pgmap v6910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:13 smithi114 ceph-mon[30820]: pgmap v6911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:13 smithi116 ceph-mon[35411]: pgmap v6911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:15 smithi114 ceph-mon[30820]: pgmap v6912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:15 smithi116 ceph-mon[35411]: pgmap v6912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:17 smithi114 ceph-mon[30820]: pgmap v6913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:20:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:17 smithi116 ceph-mon[35411]: pgmap v6913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:20:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:19 smithi114 ceph-mon[30820]: pgmap v6914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:19 smithi116 ceph-mon[35411]: pgmap v6914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:21 smithi114 ceph-mon[30820]: pgmap v6915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:21 smithi116 ceph-mon[35411]: pgmap v6915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:23 smithi114 ceph-mon[30820]: pgmap v6916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:23 smithi116 ceph-mon[35411]: pgmap v6916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:25 smithi114 ceph-mon[30820]: pgmap v6917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:25 smithi116 ceph-mon[35411]: pgmap v6917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:27 smithi114 ceph-mon[30820]: pgmap v6918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:20:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:27 smithi116 ceph-mon[35411]: pgmap v6918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:20:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:29 smithi114 ceph-mon[30820]: pgmap v6919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:29 smithi116 ceph-mon[35411]: pgmap v6919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:31 smithi114 ceph-mon[30820]: pgmap v6920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:31 smithi116 ceph-mon[35411]: pgmap v6920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:33 smithi114 ceph-mon[30820]: pgmap v6921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:33 smithi116 ceph-mon[35411]: pgmap v6921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:35.897 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:35 smithi114 ceph-mon[30820]: pgmap v6922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:35 smithi116 ceph-mon[35411]: pgmap v6922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:37 smithi114 ceph-mon[30820]: pgmap v6923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:20:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:37 smithi116 ceph-mon[35411]: pgmap v6923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:20:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:39 smithi114 ceph-mon[30820]: pgmap v6924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:39 smithi116 ceph-mon[35411]: pgmap v6924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:41 smithi114 ceph-mon[30820]: pgmap v6925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:41 smithi116 ceph-mon[35411]: pgmap v6925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:20:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:20:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:20:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:43 smithi114 ceph-mon[30820]: pgmap v6926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:43 smithi116 ceph-mon[35411]: pgmap v6926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:20:45.076+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:20:45.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:20:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:20:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:45 smithi116 ceph-mon[35411]: pgmap v6927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:20:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:20:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:20:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-mon[30820]: pgmap v6927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:20:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:20:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:20:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:20:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:20:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:46 smithi116 ceph-mon[35411]: pgmap v6928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:20:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:46 smithi116 ceph-mon[35411]: pgmap v6929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:20:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:20:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:20:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:46 smithi114 ceph-mon[30820]: pgmap v6928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:20:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:46 smithi114 ceph-mon[30820]: pgmap v6929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:20:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:48 smithi116 ceph-mon[35411]: pgmap v6930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:20:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:48 smithi114 ceph-mon[30820]: pgmap v6930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:20:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:50 smithi114 ceph-mon[30820]: pgmap v6931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:20:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:50 smithi116 ceph-mon[35411]: pgmap v6931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:20:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:52 smithi114 ceph-mon[30820]: pgmap v6932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:20:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:52 smithi116 ceph-mon[35411]: pgmap v6932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:20:54.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:53 smithi114 ceph-mon[30820]: pgmap v6933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:20:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:53 smithi116 ceph-mon[35411]: pgmap v6933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:20:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:56 smithi114 ceph-mon[30820]: pgmap v6934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:20:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:56 smithi116 ceph-mon[35411]: pgmap v6934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:20:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:20:58 smithi114 ceph-mon[30820]: pgmap v6935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:20:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:20:58 smithi116 ceph-mon[35411]: pgmap v6935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:00 smithi114 ceph-mon[30820]: pgmap v6936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:21:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:00 smithi116 ceph-mon[35411]: pgmap v6936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:21:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:02 smithi114 ceph-mon[30820]: pgmap v6937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:02 smithi116 ceph-mon[35411]: pgmap v6937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:04 smithi114 ceph-mon[30820]: pgmap v6938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:04 smithi116 ceph-mon[35411]: pgmap v6938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:06 smithi114 ceph-mon[30820]: pgmap v6939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:21:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:06 smithi116 ceph-mon[35411]: pgmap v6939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:21:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:08 smithi114 ceph-mon[30820]: pgmap v6940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:08 smithi116 ceph-mon[35411]: pgmap v6940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:10 smithi114 ceph-mon[30820]: pgmap v6941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:10 smithi116 ceph-mon[35411]: pgmap v6941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:12 smithi116 ceph-mon[35411]: pgmap v6942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:12 smithi114 ceph-mon[30820]: pgmap v6942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:14 smithi116 ceph-mon[35411]: pgmap v6943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:14 smithi114 ceph-mon[30820]: pgmap v6943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:16 smithi116 ceph-mon[35411]: pgmap v6944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:21:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:16 smithi114 ceph-mon[30820]: pgmap v6944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:21:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:18 smithi114 ceph-mon[30820]: pgmap v6945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:18 smithi116 ceph-mon[35411]: pgmap v6945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:20 smithi114 ceph-mon[30820]: pgmap v6946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:20 smithi116 ceph-mon[35411]: pgmap v6946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:22 smithi114 ceph-mon[30820]: pgmap v6947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:22 smithi116 ceph-mon[35411]: pgmap v6947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:24.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:24 smithi114 ceph-mon[30820]: pgmap v6948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:24 smithi116 ceph-mon[35411]: pgmap v6948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:26 smithi114 ceph-mon[30820]: pgmap v6949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:21:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:26 smithi116 ceph-mon[35411]: pgmap v6949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:21:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:28 smithi114 ceph-mon[30820]: pgmap v6950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:28 smithi116 ceph-mon[35411]: pgmap v6950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:30.480 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:30 smithi116 ceph-mon[35411]: pgmap v6951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:30 smithi114 ceph-mon[30820]: pgmap v6951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:32 smithi114 ceph-mon[30820]: pgmap v6952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:32 smithi116 ceph-mon[35411]: pgmap v6952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:34 smithi114 ceph-mon[30820]: pgmap v6953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:34 smithi116 ceph-mon[35411]: pgmap v6953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:36 smithi114 ceph-mon[30820]: pgmap v6954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:21:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:36 smithi116 ceph-mon[35411]: pgmap v6954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:21:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:38 smithi114 ceph-mon[30820]: pgmap v6955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:38 smithi116 ceph-mon[35411]: pgmap v6955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:40 smithi114 ceph-mon[30820]: pgmap v6956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:21:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:40 smithi116 ceph-mon[35411]: pgmap v6956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:21:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:42 smithi114 ceph-mon[30820]: pgmap v6957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:42 smithi116 ceph-mon[35411]: pgmap v6957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:44 smithi114 ceph-mon[30820]: pgmap v6958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:21:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:44 smithi116 ceph-mon[35411]: pgmap v6958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:21:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:21:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:21:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:46 smithi114 ceph-mon[30820]: pgmap v6959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:21:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:46 smithi116 ceph-mon[35411]: pgmap v6959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:21:47.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:47.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:47.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:47.659 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:48.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:21:48.372+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:21:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:21:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:21:48.793 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:48 smithi116 ceph-mon[35411]: pgmap v6960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:49.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:48 smithi114 ceph-mon[30820]: pgmap v6960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:21:49.904 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:21:49.905 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:21:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:21:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:21:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:21:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:21:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:21:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:21:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:21:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:21:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:21:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:21:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:21:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:21:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:21:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:21:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:21:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:21:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:21:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:21:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:21:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:21:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:21:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:21:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:21:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:50 smithi116 ceph-mon[35411]: pgmap v6961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:50 smithi114 ceph-mon[30820]: pgmap v6961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:21:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:52 smithi116 ceph-mon[35411]: pgmap v6962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:52 smithi114 ceph-mon[30820]: pgmap v6962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:21:54.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:54.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:54.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:21:54.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:21:54.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:53 smithi114 ceph-mon[30820]: pgmap v6963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:21:54.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:53 smithi114 ceph-mon[30820]: pgmap v6964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:21:54.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:21:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:21:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:53 smithi116 ceph-mon[35411]: pgmap v6963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:21:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:53 smithi116 ceph-mon[35411]: pgmap v6964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:21:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:21:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:55 smithi114 ceph-mon[30820]: pgmap v6965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:21:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:55 smithi116 ceph-mon[35411]: pgmap v6965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:21:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:57 smithi114 ceph-mon[30820]: pgmap v6966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:21:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:57 smithi116 ceph-mon[35411]: pgmap v6966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:22:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:21:59 smithi114 ceph-mon[30820]: pgmap v6967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:22:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:21:59 smithi116 ceph-mon[35411]: pgmap v6967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:22:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:01 smithi114 ceph-mon[30820]: pgmap v6968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:22:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:01 smithi116 ceph-mon[35411]: pgmap v6968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:22:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:03 smithi114 ceph-mon[30820]: pgmap v6969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:22:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:03 smithi116 ceph-mon[35411]: pgmap v6969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:22:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:05 smithi116 ceph-mon[35411]: pgmap v6970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:05 smithi114 ceph-mon[30820]: pgmap v6970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:07 smithi116 ceph-mon[35411]: pgmap v6971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:07 smithi114 ceph-mon[30820]: pgmap v6971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:09 smithi116 ceph-mon[35411]: pgmap v6972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:09 smithi114 ceph-mon[30820]: pgmap v6972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:12.215 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:11 smithi116 ceph-mon[35411]: pgmap v6973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:11 smithi114 ceph-mon[30820]: pgmap v6973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:13 smithi114 ceph-mon[30820]: pgmap v6974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:13 smithi116 ceph-mon[35411]: pgmap v6974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:15 smithi114 ceph-mon[30820]: pgmap v6975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:15 smithi116 ceph-mon[35411]: pgmap v6975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:17 smithi114 ceph-mon[30820]: pgmap v6976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:17 smithi116 ceph-mon[35411]: pgmap v6976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:19 smithi114 ceph-mon[30820]: pgmap v6977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:19 smithi116 ceph-mon[35411]: pgmap v6977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:21 smithi114 ceph-mon[30820]: pgmap v6978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:21 smithi116 ceph-mon[35411]: pgmap v6978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:24.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:23 smithi114 ceph-mon[30820]: pgmap v6979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:23 smithi116 ceph-mon[35411]: pgmap v6979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:25 smithi114 ceph-mon[30820]: pgmap v6980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:25 smithi116 ceph-mon[35411]: pgmap v6980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:27 smithi114 ceph-mon[30820]: pgmap v6981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:27 smithi116 ceph-mon[35411]: pgmap v6981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:30.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:29 smithi116 ceph-mon[35411]: pgmap v6982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:29 smithi114 ceph-mon[30820]: pgmap v6982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:31 smithi114 ceph-mon[30820]: pgmap v6983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:31 smithi116 ceph-mon[35411]: pgmap v6983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:33 smithi114 ceph-mon[30820]: pgmap v6984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:33 smithi116 ceph-mon[35411]: pgmap v6984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:35 smithi114 ceph-mon[30820]: pgmap v6985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:35 smithi116 ceph-mon[35411]: pgmap v6985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:37 smithi114 ceph-mon[30820]: pgmap v6986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:37 smithi116 ceph-mon[35411]: pgmap v6986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:39 smithi114 ceph-mon[30820]: pgmap v6987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:39 smithi116 ceph-mon[35411]: pgmap v6987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:41 smithi114 ceph-mon[30820]: pgmap v6988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:41 smithi116 ceph-mon[35411]: pgmap v6988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:43 smithi114 ceph-mon[30820]: pgmap v6989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:43 smithi116 ceph-mon[35411]: pgmap v6989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:45 smithi114 ceph-mon[30820]: pgmap v6990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:45 smithi116 ceph-mon[35411]: pgmap v6990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:47 smithi114 ceph-mon[30820]: pgmap v6991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:47 smithi116 ceph-mon[35411]: pgmap v6991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:22:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:50 smithi114 ceph-mon[30820]: pgmap v6992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:50 smithi116 ceph-mon[35411]: pgmap v6992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:52.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:52 smithi114 ceph-mon[30820]: pgmap v6993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:52 smithi116 ceph-mon[35411]: pgmap v6993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:22:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:22:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:22:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:54 smithi114 ceph-mon[30820]: pgmap v6994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:54 smithi116 ceph-mon[35411]: pgmap v6994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:22:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-mon[30820]: pgmap v6995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:22:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:22:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:56.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:22:56.148+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:22:56.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:22:56.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:22:56.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:22:56.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:22:56.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:22:56.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:22:56.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:22:56.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:22:56.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:22:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:22:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:56 smithi116 ceph-mon[35411]: pgmap v6995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:22:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:22:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:22:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:22:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:57 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:22:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:22:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:57 smithi114 ceph-mon[30820]: pgmap v6996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:22:57.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:57 smithi114 ceph-mon[30820]: pgmap v6997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:57 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:22:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:57 smithi116 ceph-mon[35411]: pgmap v6996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:22:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:57 smithi116 ceph-mon[35411]: pgmap v6997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:22:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:22:59 smithi114 ceph-mon[30820]: pgmap v6998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:22:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:22:59 smithi116 ceph-mon[35411]: pgmap v6998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:23:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:01 smithi114 ceph-mon[30820]: pgmap v6999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:23:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:01 smithi116 ceph-mon[35411]: pgmap v6999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:23:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:03 smithi114 ceph-mon[30820]: pgmap v7000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:23:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:03 smithi116 ceph-mon[35411]: pgmap v7000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:23:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:05 smithi114 ceph-mon[30820]: pgmap v7001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:23:05.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:05 smithi116 ceph-mon[35411]: pgmap v7001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:23:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:07 smithi114 ceph-mon[30820]: pgmap v7002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:23:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:07 smithi116 ceph-mon[35411]: pgmap v7002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:23:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:09 smithi114 ceph-mon[30820]: pgmap v7003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:09 smithi116 ceph-mon[35411]: pgmap v7003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:11 smithi114 ceph-mon[30820]: pgmap v7004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:11 smithi116 ceph-mon[35411]: pgmap v7004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:13 smithi114 ceph-mon[30820]: pgmap v7005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:13 smithi116 ceph-mon[35411]: pgmap v7005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:15 smithi114 ceph-mon[30820]: pgmap v7006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:15 smithi116 ceph-mon[35411]: pgmap v7006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:17 smithi114 ceph-mon[30820]: pgmap v7007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:17 smithi116 ceph-mon[35411]: pgmap v7007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:19 smithi114 ceph-mon[30820]: pgmap v7008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:19 smithi116 ceph-mon[35411]: pgmap v7008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:21 smithi114 ceph-mon[30820]: pgmap v7009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:21 smithi116 ceph-mon[35411]: pgmap v7009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:23 smithi114 ceph-mon[30820]: pgmap v7010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:23 smithi116 ceph-mon[35411]: pgmap v7010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:25 smithi114 ceph-mon[30820]: pgmap v7011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:25 smithi116 ceph-mon[35411]: pgmap v7011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:27 smithi114 ceph-mon[30820]: pgmap v7012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:27.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:27 smithi116 ceph-mon[35411]: pgmap v7012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:29 smithi114 ceph-mon[30820]: pgmap v7013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:29 smithi116 ceph-mon[35411]: pgmap v7013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:31 smithi114 ceph-mon[30820]: pgmap v7014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:31 smithi116 ceph-mon[35411]: pgmap v7014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:33 smithi114 ceph-mon[30820]: pgmap v7015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:33 smithi116 ceph-mon[35411]: pgmap v7015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:35 smithi114 ceph-mon[30820]: pgmap v7016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:35.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:35 smithi116 ceph-mon[35411]: pgmap v7016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:37 smithi114 ceph-mon[30820]: pgmap v7017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:37 smithi116 ceph-mon[35411]: pgmap v7017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:39 smithi114 ceph-mon[30820]: pgmap v7018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:39 smithi116 ceph-mon[35411]: pgmap v7018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:41 smithi114 ceph-mon[30820]: pgmap v7019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:41 smithi116 ceph-mon[35411]: pgmap v7019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:43.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:43 smithi114 ceph-mon[30820]: pgmap v7020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:43 smithi116 ceph-mon[35411]: pgmap v7020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:45 smithi114 ceph-mon[30820]: pgmap v7021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:45 smithi116 ceph-mon[35411]: pgmap v7021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:47 smithi114 ceph-mon[30820]: pgmap v7022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:47 smithi116 ceph-mon[35411]: pgmap v7022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:49 smithi114 ceph-mon[30820]: pgmap v7023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:49 smithi116 ceph-mon[35411]: pgmap v7023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:51 smithi114 ceph-mon[30820]: pgmap v7024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:51 smithi116 ceph-mon[35411]: pgmap v7024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:23:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:53 smithi114 ceph-mon[30820]: pgmap v7025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:53 smithi116 ceph-mon[35411]: pgmap v7025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:23:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:55 smithi114 ceph-mon[30820]: pgmap v7026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:55 smithi116 ceph-mon[35411]: pgmap v7026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:23:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:23:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:57 smithi114 ceph-mon[30820]: pgmap v7027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:57 smithi116 ceph-mon[35411]: pgmap v7027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:23:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-mon[30820]: pgmap v7028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:23:59.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:23:59.510+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:23:59.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:23:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:23:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:23:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:23:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:23:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:23:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:23:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:23:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:23:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:23:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:23:59 smithi116 ceph-mon[35411]: pgmap v7028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:00 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:00 smithi114 ceph-mon[30820]: pgmap v7029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:24:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:00 smithi114 ceph-mon[30820]: pgmap v7030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:00.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:24:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:00 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:24:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:24:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:24:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:24:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:24:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:24:00.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:00 smithi116 ceph-mon[35411]: pgmap v7029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:24:00.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:00 smithi116 ceph-mon[35411]: pgmap v7030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:00.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:24:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:02 smithi114 ceph-mon[30820]: pgmap v7031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:02 smithi116 ceph-mon[35411]: pgmap v7031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:04 smithi114 ceph-mon[30820]: pgmap v7032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:04 smithi116 ceph-mon[35411]: pgmap v7032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:06 smithi114 ceph-mon[30820]: pgmap v7033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:06 smithi116 ceph-mon[35411]: pgmap v7033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:08 smithi114 ceph-mon[30820]: pgmap v7034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:08 smithi116 ceph-mon[35411]: pgmap v7034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:24:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:10 smithi114 ceph-mon[30820]: pgmap v7035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:24:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:10 smithi116 ceph-mon[35411]: pgmap v7035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:24:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:12 smithi114 ceph-mon[30820]: pgmap v7036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:12 smithi116 ceph-mon[35411]: pgmap v7036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:14 smithi114 ceph-mon[30820]: pgmap v7037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:14 smithi116 ceph-mon[35411]: pgmap v7037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:16 smithi114 ceph-mon[30820]: pgmap v7038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:16 smithi116 ceph-mon[35411]: pgmap v7038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:18 smithi114 ceph-mon[30820]: pgmap v7039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:18 smithi116 ceph-mon[35411]: pgmap v7039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:20 smithi114 ceph-mon[30820]: pgmap v7040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:24:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:20 smithi116 ceph-mon[35411]: pgmap v7040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:24:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:22 smithi114 ceph-mon[30820]: pgmap v7041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:22 smithi116 ceph-mon[35411]: pgmap v7041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:24 smithi114 ceph-mon[30820]: pgmap v7042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:24 smithi116 ceph-mon[35411]: pgmap v7042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:26 smithi114 ceph-mon[30820]: pgmap v7043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:26 smithi116 ceph-mon[35411]: pgmap v7043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:28 smithi114 ceph-mon[30820]: pgmap v7044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:28 smithi116 ceph-mon[35411]: pgmap v7044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:30 smithi114 ceph-mon[30820]: pgmap v7045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:24:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:30 smithi116 ceph-mon[35411]: pgmap v7045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:24:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:32 smithi114 ceph-mon[30820]: pgmap v7046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:32 smithi116 ceph-mon[35411]: pgmap v7046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:34 smithi114 ceph-mon[30820]: pgmap v7047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:34 smithi116 ceph-mon[35411]: pgmap v7047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:36 smithi114 ceph-mon[30820]: pgmap v7048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:36 smithi116 ceph-mon[35411]: pgmap v7048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:38 smithi114 ceph-mon[30820]: pgmap v7049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:38 smithi116 ceph-mon[35411]: pgmap v7049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:40 smithi114 ceph-mon[30820]: pgmap v7050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:24:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:40 smithi116 ceph-mon[35411]: pgmap v7050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:24:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:42 smithi114 ceph-mon[30820]: pgmap v7051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:42 smithi116 ceph-mon[35411]: pgmap v7051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:44 smithi114 ceph-mon[30820]: pgmap v7052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:44 smithi116 ceph-mon[35411]: pgmap v7052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:46 smithi114 ceph-mon[30820]: pgmap v7053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:46 smithi116 ceph-mon[35411]: pgmap v7053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:48 smithi114 ceph-mon[30820]: pgmap v7054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:48 smithi116 ceph-mon[35411]: pgmap v7054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:50 smithi114 ceph-mon[30820]: pgmap v7055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:24:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:50 smithi116 ceph-mon[35411]: pgmap v7055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:24:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:52 smithi114 ceph-mon[30820]: pgmap v7056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:52 smithi116 ceph-mon[35411]: pgmap v7056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:24:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:54 smithi116 ceph-mon[35411]: pgmap v7057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:54 smithi114 ceph-mon[30820]: pgmap v7057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:56 smithi116 ceph-mon[35411]: pgmap v7058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:56.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:56 smithi114 ceph-mon[30820]: pgmap v7058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:24:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:58 smithi116 ceph-mon[35411]: pgmap v7059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:24:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:58 smithi114 ceph-mon[30820]: pgmap v7059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:24:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:25:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:24:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:25:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:00 smithi114 ceph-mon[30820]: pgmap v7060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:00 smithi116 ceph-mon[35411]: pgmap v7060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:02.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:02 smithi116 ceph-mon[35411]: pgmap v7061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-mon[30820]: pgmap v7061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:03.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:25:02.868+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:25:03.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:25:03.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:25:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:25:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:25:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:25:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:25:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:25:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:25:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:25:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:04 smithi114 ceph-mon[30820]: pgmap v7062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:25:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:04 smithi114 ceph-mon[30820]: pgmap v7063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:25:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:25:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:25:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:25:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:25:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:25:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:25:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:25:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:25:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:25:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:04 smithi116 ceph-mon[35411]: pgmap v7062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:25:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:04 smithi116 ceph-mon[35411]: pgmap v7063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:25:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:06 smithi114 ceph-mon[30820]: pgmap v7064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:25:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:06 smithi116 ceph-mon[35411]: pgmap v7064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:25:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:08 smithi114 ceph-mon[30820]: pgmap v7065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:25:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:08 smithi116 ceph-mon[35411]: pgmap v7065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:25:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:10 smithi114 ceph-mon[30820]: pgmap v7066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:25:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:10 smithi116 ceph-mon[35411]: pgmap v7066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:25:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:12 smithi114 ceph-mon[30820]: pgmap v7067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:25:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:12 smithi116 ceph-mon[35411]: pgmap v7067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:25:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:14 smithi114 ceph-mon[30820]: pgmap v7068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:25:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:14 smithi116 ceph-mon[35411]: pgmap v7068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:25:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:16 smithi114 ceph-mon[30820]: pgmap v7069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:16 smithi116 ceph-mon[35411]: pgmap v7069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:18 smithi114 ceph-mon[30820]: pgmap v7070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:18 smithi116 ceph-mon[35411]: pgmap v7070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:20 smithi114 ceph-mon[30820]: pgmap v7071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:20 smithi116 ceph-mon[35411]: pgmap v7071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:22 smithi114 ceph-mon[30820]: pgmap v7072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:25:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:22 smithi116 ceph-mon[35411]: pgmap v7072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:25:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:24 smithi114 ceph-mon[30820]: pgmap v7073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:24 smithi116 ceph-mon[35411]: pgmap v7073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:26 smithi114 ceph-mon[30820]: pgmap v7074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:26 smithi116 ceph-mon[35411]: pgmap v7074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:27 smithi114 ceph-mon[30820]: pgmap v7075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:27 smithi116 ceph-mon[35411]: pgmap v7075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:29 smithi114 ceph-mon[30820]: pgmap v7076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:29 smithi116 ceph-mon[35411]: pgmap v7076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:32.230 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:31 smithi116 ceph-mon[35411]: pgmap v7077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:25:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:31 smithi114 ceph-mon[30820]: pgmap v7077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:25:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:33 smithi114 ceph-mon[30820]: pgmap v7078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:33 smithi116 ceph-mon[35411]: pgmap v7078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:35 smithi114 ceph-mon[30820]: pgmap v7079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:35 smithi116 ceph-mon[35411]: pgmap v7079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:37 smithi114 ceph-mon[30820]: pgmap v7080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:37 smithi116 ceph-mon[35411]: pgmap v7080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:39 smithi114 ceph-mon[30820]: pgmap v7081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:39 smithi116 ceph-mon[35411]: pgmap v7081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:41 smithi114 ceph-mon[30820]: pgmap v7082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:25:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:41 smithi116 ceph-mon[35411]: pgmap v7082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:25:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:44 smithi114 ceph-mon[30820]: pgmap v7083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:44 smithi116 ceph-mon[35411]: pgmap v7083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:46 smithi114 ceph-mon[30820]: pgmap v7084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:46 smithi116 ceph-mon[35411]: pgmap v7084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:48 smithi114 ceph-mon[30820]: pgmap v7085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:48 smithi116 ceph-mon[35411]: pgmap v7085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:50 smithi114 ceph-mon[30820]: pgmap v7086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:50 smithi116 ceph-mon[35411]: pgmap v7086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:52 smithi114 ceph-mon[30820]: pgmap v7087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:25:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:52 smithi116 ceph-mon[35411]: pgmap v7087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:25:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:54 smithi114 ceph-mon[30820]: pgmap v7088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:54 smithi116 ceph-mon[35411]: pgmap v7088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:25:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:56 smithi114 ceph-mon[30820]: pgmap v7089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:56 smithi116 ceph-mon[35411]: pgmap v7089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:25:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:25:58 smithi114 ceph-mon[30820]: pgmap v7090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:25:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:25:58 smithi116 ceph-mon[35411]: pgmap v7090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:00 smithi114 ceph-mon[30820]: pgmap v7091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:00 smithi116 ceph-mon[35411]: pgmap v7091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:02 smithi114 ceph-mon[30820]: pgmap v7092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:26:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:02 smithi116 ceph-mon[35411]: pgmap v7092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:26:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:26:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:26:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:04 smithi114 ceph-mon[30820]: pgmap v7093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:04 smithi116 ceph-mon[35411]: pgmap v7093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-mon[30820]: pgmap v7094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:26:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:26:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:26:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:26:06.228+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:26:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:26:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:26:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:26:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:26:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:26:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:26:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:26:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:26:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:26:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:26:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:26:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:06 smithi116 ceph-mon[35411]: pgmap v7094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:26:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:26:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:26:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:26:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:26:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:26:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:26:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:26:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:26:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:26:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:26:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:26:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:26:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:07 smithi114 ceph-mon[30820]: pgmap v7095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:26:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:07 smithi114 ceph-mon[30820]: pgmap v7096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:26:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:26:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:26:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:26:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:26:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:26:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:26:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:26:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:26:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:26:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:07 smithi116 ceph-mon[35411]: pgmap v7095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:26:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:07 smithi116 ceph-mon[35411]: pgmap v7096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:26:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:09 smithi114 ceph-mon[30820]: pgmap v7097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:26:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:09 smithi116 ceph-mon[35411]: pgmap v7097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:26:11.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:11 smithi114 ceph-mon[30820]: pgmap v7098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:26:11.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:11 smithi116 ceph-mon[35411]: pgmap v7098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:26:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:13 smithi114 ceph-mon[30820]: pgmap v7099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:26:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:13 smithi116 ceph-mon[35411]: pgmap v7099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:26:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:15 smithi114 ceph-mon[30820]: pgmap v7100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:26:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:15 smithi116 ceph-mon[35411]: pgmap v7100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:26:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:17 smithi114 ceph-mon[30820]: pgmap v7101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:26:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:17 smithi116 ceph-mon[35411]: pgmap v7101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:26:19.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:19 smithi114 ceph-mon[30820]: pgmap v7102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:19 smithi116 ceph-mon[35411]: pgmap v7102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:21 smithi114 ceph-mon[30820]: pgmap v7103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:26:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:21 smithi116 ceph-mon[35411]: pgmap v7103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:26:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:23 smithi114 ceph-mon[30820]: pgmap v7104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:26:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:23 smithi116 ceph-mon[35411]: pgmap v7104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:26:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:25 smithi114 ceph-mon[30820]: pgmap v7105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:25 smithi116 ceph-mon[35411]: pgmap v7105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:27 smithi114 ceph-mon[30820]: pgmap v7106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:27 smithi116 ceph-mon[35411]: pgmap v7106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:29 smithi114 ceph-mon[30820]: pgmap v7107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:29 smithi116 ceph-mon[35411]: pgmap v7107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:31 smithi114 ceph-mon[30820]: pgmap v7108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:26:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:31 smithi116 ceph-mon[35411]: pgmap v7108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:26:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:33 smithi114 ceph-mon[30820]: pgmap v7109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:26:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:33 smithi116 ceph-mon[35411]: pgmap v7109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:26:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:35 smithi114 ceph-mon[30820]: pgmap v7110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:35 smithi116 ceph-mon[35411]: pgmap v7110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:37 smithi114 ceph-mon[30820]: pgmap v7111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:37 smithi116 ceph-mon[35411]: pgmap v7111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:39 smithi114 ceph-mon[30820]: pgmap v7112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:39 smithi116 ceph-mon[35411]: pgmap v7112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:41 smithi114 ceph-mon[30820]: pgmap v7113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:26:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:41 smithi116 ceph-mon[35411]: pgmap v7113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:26:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:43 smithi114 ceph-mon[30820]: pgmap v7114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:26:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:43 smithi116 ceph-mon[35411]: pgmap v7114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:26:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:45 smithi114 ceph-mon[30820]: pgmap v7115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:45 smithi116 ceph-mon[35411]: pgmap v7115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:47 smithi114 ceph-mon[30820]: pgmap v7116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:47 smithi116 ceph-mon[35411]: pgmap v7116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:49 smithi114 ceph-mon[30820]: pgmap v7117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:49 smithi116 ceph-mon[35411]: pgmap v7117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:51 smithi114 ceph-mon[30820]: pgmap v7118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:26:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:51 smithi116 ceph-mon[35411]: pgmap v7118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:26:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:53 smithi114 ceph-mon[30820]: pgmap v7119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:26:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:53 smithi116 ceph-mon[35411]: pgmap v7119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:26:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:55 smithi114 ceph-mon[30820]: pgmap v7120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:55 smithi116 ceph-mon[35411]: pgmap v7120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:26:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:57 smithi114 ceph-mon[30820]: pgmap v7121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:57 smithi116 ceph-mon[35411]: pgmap v7121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:26:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:26:59 smithi114 ceph-mon[30820]: pgmap v7122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:26:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:26:59 smithi116 ceph-mon[35411]: pgmap v7122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:01 smithi114 ceph-mon[30820]: pgmap v7123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:27:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:01 smithi116 ceph-mon[35411]: pgmap v7123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:27:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:03 smithi114 ceph-mon[30820]: pgmap v7124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:27:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:03 smithi116 ceph-mon[35411]: pgmap v7124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:27:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:05 smithi114 ceph-mon[30820]: pgmap v7125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:27:05.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:05 smithi116 ceph-mon[35411]: pgmap v7125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:27:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:27:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:27:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:07 smithi114 ceph-mon[30820]: pgmap v7126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:07 smithi116 ceph-mon[35411]: pgmap v7126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-mon[30820]: pgmap v7127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:27:09.588+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:27:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:27:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:27:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:27:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:09 smithi116 ceph-mon[35411]: pgmap v7127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:10 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:27:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:27:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:27:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:10 smithi114 ceph-mon[30820]: pgmap v7128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:10 smithi114 ceph-mon[30820]: pgmap v7129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:27:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:27:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:10 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:27:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:27:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:27:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:27:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:27:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:27:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:27:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:27:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:27:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:27:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:27:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:27:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:10 smithi116 ceph-mon[35411]: pgmap v7128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:27:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:10 smithi116 ceph-mon[35411]: pgmap v7129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:27:10.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:27:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:12 smithi114 ceph-mon[30820]: pgmap v7130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:27:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:12 smithi116 ceph-mon[35411]: pgmap v7130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:27:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:14 smithi114 ceph-mon[30820]: pgmap v7131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:27:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:14 smithi116 ceph-mon[35411]: pgmap v7131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:27:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:16 smithi114 ceph-mon[30820]: pgmap v7132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:27:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:16 smithi116 ceph-mon[35411]: pgmap v7132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:27:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:18 smithi114 ceph-mon[30820]: pgmap v7133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:27:18.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:18 smithi116 ceph-mon[35411]: pgmap v7133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:27:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:20 smithi114 ceph-mon[30820]: pgmap v7134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:27:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:20 smithi116 ceph-mon[35411]: pgmap v7134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:27:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:22 smithi116 ceph-mon[35411]: pgmap v7135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:22 smithi114 ceph-mon[30820]: pgmap v7135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:24 smithi116 ceph-mon[35411]: pgmap v7136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:24 smithi114 ceph-mon[30820]: pgmap v7136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:26 smithi116 ceph-mon[35411]: pgmap v7137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:27:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:26 smithi114 ceph-mon[30820]: pgmap v7137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:27:28.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:28 smithi114 ceph-mon[30820]: pgmap v7138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:28 smithi116 ceph-mon[35411]: pgmap v7138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:30 smithi114 ceph-mon[30820]: pgmap v7139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:27:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:30 smithi116 ceph-mon[35411]: pgmap v7139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:27:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:32 smithi114 ceph-mon[30820]: pgmap v7140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:32 smithi116 ceph-mon[35411]: pgmap v7140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:34.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:34 smithi116 ceph-mon[35411]: pgmap v7141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:34 smithi114 ceph-mon[30820]: pgmap v7141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:36 smithi114 ceph-mon[30820]: pgmap v7142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:27:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:36 smithi116 ceph-mon[35411]: pgmap v7142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:27:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:38 smithi114 ceph-mon[30820]: pgmap v7143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:38 smithi116 ceph-mon[35411]: pgmap v7143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:40 smithi114 ceph-mon[30820]: pgmap v7144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:27:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:40 smithi116 ceph-mon[35411]: pgmap v7144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:27:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:42 smithi114 ceph-mon[30820]: pgmap v7145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:42 smithi116 ceph-mon[35411]: pgmap v7145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:44 smithi114 ceph-mon[30820]: pgmap v7146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:44 smithi116 ceph-mon[35411]: pgmap v7146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:46 smithi114 ceph-mon[30820]: pgmap v7147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:27:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:46 smithi116 ceph-mon[35411]: pgmap v7147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:27:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:48 smithi114 ceph-mon[30820]: pgmap v7148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:48 smithi116 ceph-mon[35411]: pgmap v7148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:50 smithi114 ceph-mon[30820]: pgmap v7149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:27:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:50 smithi116 ceph-mon[35411]: pgmap v7149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:27:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:52 smithi114 ceph-mon[30820]: pgmap v7150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:52 smithi116 ceph-mon[35411]: pgmap v7150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:27:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:54 smithi114 ceph-mon[30820]: pgmap v7151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:54 smithi116 ceph-mon[35411]: pgmap v7151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:56 smithi114 ceph-mon[30820]: pgmap v7152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:27:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:56 smithi116 ceph-mon[35411]: pgmap v7152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:27:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:27:58 smithi114 ceph-mon[30820]: pgmap v7153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:27:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:27:58 smithi116 ceph-mon[35411]: pgmap v7153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:00 smithi114 ceph-mon[30820]: pgmap v7154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:00 smithi116 ceph-mon[35411]: pgmap v7154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:02 smithi114 ceph-mon[30820]: pgmap v7155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:02 smithi116 ceph-mon[35411]: pgmap v7155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:04 smithi114 ceph-mon[30820]: pgmap v7156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:05.210 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:04 smithi116 ceph-mon[35411]: pgmap v7156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:06 smithi114 ceph-mon[30820]: pgmap v7157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:28:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:06 smithi116 ceph-mon[35411]: pgmap v7157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:28:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:08 smithi114 ceph-mon[30820]: pgmap v7158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:08 smithi116 ceph-mon[35411]: pgmap v7158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:28:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:28:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:10 smithi114 ceph-mon[30820]: pgmap v7159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:10 smithi116 ceph-mon[35411]: pgmap v7159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-mon[30820]: pgmap v7160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:13.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:28:12.948+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:28:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:28:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:28:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:28:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:12 smithi116 ceph-mon[35411]: pgmap v7160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:13 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:28:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:28:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:28:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:28:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:28:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:13 smithi114 ceph-mon[30820]: pgmap v7161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:28:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:13 smithi114 ceph-mon[30820]: pgmap v7162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:28:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:28:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:13 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:28:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:28:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:28:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:28:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:28:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:28:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:28:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:28:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:28:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:28:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:28:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:28:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:13 smithi116 ceph-mon[35411]: pgmap v7161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:28:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:13 smithi116 ceph-mon[35411]: pgmap v7162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:28:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:28:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:16 smithi114 ceph-mon[30820]: pgmap v7163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:28:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:16 smithi116 ceph-mon[35411]: pgmap v7163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:28:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:18 smithi114 ceph-mon[30820]: pgmap v7164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:28:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:18 smithi116 ceph-mon[35411]: pgmap v7164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:28:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:20 smithi114 ceph-mon[30820]: pgmap v7165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:28:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:20 smithi116 ceph-mon[35411]: pgmap v7165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:28:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:22 smithi114 ceph-mon[30820]: pgmap v7166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:28:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:22 smithi116 ceph-mon[35411]: pgmap v7166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:28:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:24 smithi114 ceph-mon[30820]: pgmap v7167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:28:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:24 smithi116 ceph-mon[35411]: pgmap v7167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:28:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:26 smithi114 ceph-mon[30820]: pgmap v7168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:26 smithi116 ceph-mon[35411]: pgmap v7168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:28 smithi114 ceph-mon[30820]: pgmap v7169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:28 smithi116 ceph-mon[35411]: pgmap v7169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:30 smithi114 ceph-mon[30820]: pgmap v7170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:30 smithi116 ceph-mon[35411]: pgmap v7170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:32 smithi114 ceph-mon[30820]: pgmap v7171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:28:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:32 smithi116 ceph-mon[35411]: pgmap v7171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:28:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:34 smithi114 ceph-mon[30820]: pgmap v7172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:34 smithi116 ceph-mon[35411]: pgmap v7172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:36 smithi114 ceph-mon[30820]: pgmap v7173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:36 smithi116 ceph-mon[35411]: pgmap v7173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:38 smithi114 ceph-mon[30820]: pgmap v7174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:38 smithi116 ceph-mon[35411]: pgmap v7174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:40 smithi114 ceph-mon[30820]: pgmap v7175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:40 smithi116 ceph-mon[35411]: pgmap v7175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:42 smithi114 ceph-mon[30820]: pgmap v7176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:28:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:42 smithi116 ceph-mon[35411]: pgmap v7176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:28:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:44 smithi114 ceph-mon[30820]: pgmap v7177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:44 smithi116 ceph-mon[35411]: pgmap v7177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:46 smithi114 ceph-mon[30820]: pgmap v7178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:46 smithi116 ceph-mon[35411]: pgmap v7178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:48 smithi114 ceph-mon[30820]: pgmap v7179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:48 smithi116 ceph-mon[35411]: pgmap v7179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:50 smithi114 ceph-mon[30820]: pgmap v7180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:50 smithi116 ceph-mon[35411]: pgmap v7180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:52 smithi114 ceph-mon[30820]: pgmap v7181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:28:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:52 smithi116 ceph-mon[35411]: pgmap v7181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:28:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:54 smithi114 ceph-mon[30820]: pgmap v7182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:54 smithi116 ceph-mon[35411]: pgmap v7182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:28:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:56 smithi114 ceph-mon[30820]: pgmap v7183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:56 smithi116 ceph-mon[35411]: pgmap v7183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:28:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:28:58 smithi114 ceph-mon[30820]: pgmap v7184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:28:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:28:58 smithi116 ceph-mon[35411]: pgmap v7184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:00 smithi114 ceph-mon[30820]: pgmap v7185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:00 smithi116 ceph-mon[35411]: pgmap v7185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:02 smithi114 ceph-mon[30820]: pgmap v7186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:29:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:02 smithi116 ceph-mon[35411]: pgmap v7186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:29:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:04 smithi114 ceph-mon[30820]: pgmap v7187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:04 smithi116 ceph-mon[35411]: pgmap v7187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:06 smithi114 ceph-mon[30820]: pgmap v7188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:06 smithi116 ceph-mon[35411]: pgmap v7188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:08 smithi114 ceph-mon[30820]: pgmap v7189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:08 smithi116 ceph-mon[35411]: pgmap v7189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:10 smithi114 ceph-mon[30820]: pgmap v7190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:10 smithi116 ceph-mon[35411]: pgmap v7190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:12 smithi114 ceph-mon[30820]: pgmap v7191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:29:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:12 smithi116 ceph-mon[35411]: pgmap v7191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:29:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:29:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:29:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: pgmap v7192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:29:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:29:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: pgmap v7192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:29:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:29:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:29:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:15 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:29:15.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:15 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:29:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:15 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:29:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:15 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:29:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-mon[30820]: pgmap v7193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:29:16.309+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:29:16.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:29:16.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:29:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:29:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:16 smithi116 ceph-mon[35411]: pgmap v7193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:29:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:29:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:29:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:29:17.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:29:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:17 smithi114 ceph-mon[30820]: pgmap v7194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:17 smithi114 ceph-mon[30820]: pgmap v7195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:29:17.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:29:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:29:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:29:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:29:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:29:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:29:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:29:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:29:17.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:29:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:29:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:29:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:29:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:29:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:17 smithi116 ceph-mon[35411]: pgmap v7194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:29:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:17 smithi116 ceph-mon[35411]: pgmap v7195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:29:17.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:29:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:19 smithi114 ceph-mon[30820]: pgmap v7196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:29:19.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:19 smithi116 ceph-mon[35411]: pgmap v7196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:29:21.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:21 smithi114 ceph-mon[30820]: pgmap v7197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:29:21.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:21 smithi116 ceph-mon[35411]: pgmap v7197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:29:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:23 smithi114 ceph-mon[30820]: pgmap v7198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:29:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:23 smithi116 ceph-mon[35411]: pgmap v7198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:29:25.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:25 smithi114 ceph-mon[30820]: pgmap v7199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:29:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:25 smithi116 ceph-mon[35411]: pgmap v7199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:29:27.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:27 smithi114 ceph-mon[30820]: pgmap v7200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:29:27.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:27 smithi116 ceph-mon[35411]: pgmap v7200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:29:29.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:29 smithi114 ceph-mon[30820]: pgmap v7201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:29 smithi116 ceph-mon[35411]: pgmap v7201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:31 smithi114 ceph-mon[30820]: pgmap v7202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:29:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:31 smithi116 ceph-mon[35411]: pgmap v7202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:29:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:33 smithi114 ceph-mon[30820]: pgmap v7203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:29:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:33 smithi116 ceph-mon[35411]: pgmap v7203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:29:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:35 smithi114 ceph-mon[30820]: pgmap v7204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:35 smithi116 ceph-mon[35411]: pgmap v7204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:37 smithi114 ceph-mon[30820]: pgmap v7205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:37 smithi116 ceph-mon[35411]: pgmap v7205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:39 smithi114 ceph-mon[30820]: pgmap v7206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:39 smithi116 ceph-mon[35411]: pgmap v7206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:41 smithi114 ceph-mon[30820]: pgmap v7207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:29:41.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:41 smithi116 ceph-mon[35411]: pgmap v7207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:29:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:43 smithi114 ceph-mon[30820]: pgmap v7208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:29:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:43 smithi116 ceph-mon[35411]: pgmap v7208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:29:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:45 smithi116 ceph-mon[35411]: pgmap v7209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:45 smithi114 ceph-mon[30820]: pgmap v7209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:47 smithi116 ceph-mon[35411]: pgmap v7210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:47 smithi114 ceph-mon[30820]: pgmap v7210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:49 smithi116 ceph-mon[35411]: pgmap v7211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:49 smithi114 ceph-mon[30820]: pgmap v7211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:51 smithi114 ceph-mon[30820]: pgmap v7212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:29:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:51 smithi116 ceph-mon[35411]: pgmap v7212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:29:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:53 smithi114 ceph-mon[30820]: pgmap v7213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:29:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:53 smithi116 ceph-mon[35411]: pgmap v7213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:29:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:55 smithi114 ceph-mon[30820]: pgmap v7214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:55 smithi116 ceph-mon[35411]: pgmap v7214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:29:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:57 smithi114 ceph-mon[30820]: pgmap v7215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:57 smithi116 ceph-mon[35411]: pgmap v7215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:29:59.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:29:59 smithi114 ceph-mon[30820]: pgmap v7216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:29:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:29:59 smithi116 ceph-mon[35411]: pgmap v7216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:30:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:30:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:01 smithi114 ceph-mon[30820]: pgmap v7217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:30:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:01 smithi116 ceph-mon[35411]: pgmap v7217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:30:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:03 smithi114 ceph-mon[30820]: pgmap v7218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:30:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:03 smithi116 ceph-mon[35411]: pgmap v7218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:30:05.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:05 smithi116 ceph-mon[35411]: pgmap v7219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:30:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:05 smithi114 ceph-mon[30820]: pgmap v7219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:30:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:07 smithi114 ceph-mon[30820]: pgmap v7220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:07 smithi116 ceph-mon[35411]: pgmap v7220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:09 smithi114 ceph-mon[30820]: pgmap v7221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:30:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:09 smithi116 ceph-mon[35411]: pgmap v7221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:30:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:11 smithi114 ceph-mon[30820]: pgmap v7222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:30:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:11 smithi116 ceph-mon[35411]: pgmap v7222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:30:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:13 smithi114 ceph-mon[30820]: pgmap v7223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T10:30:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:13 smithi116 ceph-mon[35411]: pgmap v7223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T10:30:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:15 smithi114 ceph-mon[30820]: pgmap v7224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:30:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:15 smithi116 ceph-mon[35411]: pgmap v7224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:30:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:30:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:30:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:17 smithi114 ceph-mon[30820]: pgmap v7225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:17 smithi116 ceph-mon[35411]: pgmap v7225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-mon[30820]: pgmap v7226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:19.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:30:19.669+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:30:19.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:30:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:30:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:30:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:30:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:19 smithi116 ceph-mon[35411]: pgmap v7226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:20 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:30:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:30:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:30:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:30:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:30:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:20 smithi114 ceph-mon[30820]: pgmap v7227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:20 smithi114 ceph-mon[30820]: pgmap v7228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:30:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:30:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:20 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:30:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:30:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:30:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:30:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:30:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:30:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:30:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:30:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:30:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:30:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:30:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:20 smithi116 ceph-mon[35411]: pgmap v7227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:30:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:20 smithi116 ceph-mon[35411]: pgmap v7228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:30:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:30:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:22 smithi114 ceph-mon[30820]: pgmap v7229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:30:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:22 smithi116 ceph-mon[35411]: pgmap v7229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:30:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:24 smithi114 ceph-mon[30820]: pgmap v7230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:30:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:24 smithi116 ceph-mon[35411]: pgmap v7230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:30:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:26 smithi114 ceph-mon[30820]: pgmap v7231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:30:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:26 smithi116 ceph-mon[35411]: pgmap v7231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:30:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:28 smithi114 ceph-mon[30820]: pgmap v7232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:30:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:28 smithi116 ceph-mon[35411]: pgmap v7232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:30:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:30 smithi114 ceph-mon[30820]: pgmap v7233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:30:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:30 smithi116 ceph-mon[35411]: pgmap v7233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:30:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:32 smithi114 ceph-mon[30820]: pgmap v7234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:32 smithi116 ceph-mon[35411]: pgmap v7234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:34 smithi114 ceph-mon[30820]: pgmap v7235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:34 smithi116 ceph-mon[35411]: pgmap v7235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:36 smithi114 ceph-mon[30820]: pgmap v7236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:30:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:36 smithi116 ceph-mon[35411]: pgmap v7236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:30:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:38 smithi114 ceph-mon[30820]: pgmap v7237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:38 smithi116 ceph-mon[35411]: pgmap v7237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:40 smithi114 ceph-mon[30820]: pgmap v7238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:30:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:40 smithi116 ceph-mon[35411]: pgmap v7238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:30:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:42 smithi114 ceph-mon[30820]: pgmap v7239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:42 smithi116 ceph-mon[35411]: pgmap v7239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:44 smithi114 ceph-mon[30820]: pgmap v7240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:44 smithi116 ceph-mon[35411]: pgmap v7240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:46 smithi114 ceph-mon[30820]: pgmap v7241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:30:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:46 smithi116 ceph-mon[35411]: pgmap v7241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:30:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:48 smithi114 ceph-mon[30820]: pgmap v7242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:48 smithi116 ceph-mon[35411]: pgmap v7242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:50 smithi114 ceph-mon[30820]: pgmap v7243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:30:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:50 smithi116 ceph-mon[35411]: pgmap v7243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:30:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:52 smithi114 ceph-mon[30820]: pgmap v7244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:52 smithi116 ceph-mon[35411]: pgmap v7244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:30:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:54 smithi114 ceph-mon[30820]: pgmap v7245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:54 smithi116 ceph-mon[35411]: pgmap v7245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:56 smithi114 ceph-mon[30820]: pgmap v7246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:30:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:56 smithi116 ceph-mon[35411]: pgmap v7246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:30:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:30:57 smithi114 ceph-mon[30820]: pgmap v7247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:30:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:30:57 smithi116 ceph-mon[35411]: pgmap v7247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:00 smithi114 ceph-mon[30820]: pgmap v7248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:00 smithi116 ceph-mon[35411]: pgmap v7248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:02 smithi114 ceph-mon[30820]: pgmap v7249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:02 smithi116 ceph-mon[35411]: pgmap v7249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:04 smithi114 ceph-mon[30820]: pgmap v7250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:04 smithi116 ceph-mon[35411]: pgmap v7250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:06 smithi114 ceph-mon[30820]: pgmap v7251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:31:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:06 smithi116 ceph-mon[35411]: pgmap v7251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:31:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:08 smithi114 ceph-mon[30820]: pgmap v7252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:08 smithi116 ceph-mon[35411]: pgmap v7252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:10 smithi114 ceph-mon[30820]: pgmap v7253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:10 smithi116 ceph-mon[35411]: pgmap v7253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:12 smithi114 ceph-mon[30820]: pgmap v7254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:12 smithi116 ceph-mon[35411]: pgmap v7254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:14 smithi114 ceph-mon[30820]: pgmap v7255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:14 smithi116 ceph-mon[35411]: pgmap v7255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:16 smithi114 ceph-mon[30820]: pgmap v7256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:31:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:16 smithi116 ceph-mon[35411]: pgmap v7256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:31:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:18 smithi114 ceph-mon[30820]: pgmap v7257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:18 smithi116 ceph-mon[35411]: pgmap v7257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:20 smithi114 ceph-mon[30820]: pgmap v7258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:31:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:20 smithi116 ceph-mon[35411]: pgmap v7258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:31:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:22 smithi114 ceph-mon[30820]: pgmap v7259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:31:23.028+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:31:23.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:31:23.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:31:23.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:31:23.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:31:23.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:31:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:22 smithi116 ceph-mon[35411]: pgmap v7259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:31:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:31:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:31:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:31:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:31:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:31:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:31:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:31:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:31:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:31:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:31:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:31:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:31:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:31:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:31:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:31:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:24 smithi114 ceph-mon[30820]: pgmap v7260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:31:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:24 smithi114 ceph-mon[30820]: pgmap v7261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:31:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:31:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:31:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:31:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:31:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:31:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:31:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:31:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:31:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:31:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:31:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:24 smithi116 ceph-mon[35411]: pgmap v7260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:31:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:24 smithi116 ceph-mon[35411]: pgmap v7261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:31:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:26 smithi114 ceph-mon[30820]: pgmap v7262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:31:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:26 smithi116 ceph-mon[35411]: pgmap v7262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:31:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:28 smithi114 ceph-mon[30820]: pgmap v7263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:31:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:28 smithi116 ceph-mon[35411]: pgmap v7263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:31:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:29 smithi114 ceph-mon[30820]: pgmap v7264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:31:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:29 smithi116 ceph-mon[35411]: pgmap v7264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:31:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:32 smithi114 ceph-mon[30820]: pgmap v7265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:31:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:32 smithi116 ceph-mon[35411]: pgmap v7265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:31:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:34 smithi114 ceph-mon[30820]: pgmap v7266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:31:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:34 smithi116 ceph-mon[35411]: pgmap v7266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:31:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:36 smithi114 ceph-mon[30820]: pgmap v7267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:36 smithi116 ceph-mon[35411]: pgmap v7267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:38 smithi114 ceph-mon[30820]: pgmap v7268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:38 smithi116 ceph-mon[35411]: pgmap v7268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:40 smithi114 ceph-mon[30820]: pgmap v7269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:40 smithi116 ceph-mon[35411]: pgmap v7269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:42 smithi114 ceph-mon[30820]: pgmap v7270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:31:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:42 smithi116 ceph-mon[35411]: pgmap v7270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:31:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:44 smithi114 ceph-mon[30820]: pgmap v7271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:44 smithi116 ceph-mon[35411]: pgmap v7271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:46 smithi114 ceph-mon[30820]: pgmap v7272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:46 smithi116 ceph-mon[35411]: pgmap v7272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:48 smithi114 ceph-mon[30820]: pgmap v7273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:48 smithi116 ceph-mon[35411]: pgmap v7273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:50 smithi114 ceph-mon[30820]: pgmap v7274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:50 smithi116 ceph-mon[35411]: pgmap v7274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:52 smithi114 ceph-mon[30820]: pgmap v7275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:31:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:52 smithi116 ceph-mon[35411]: pgmap v7275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:31:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:54 smithi114 ceph-mon[30820]: pgmap v7276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:54 smithi116 ceph-mon[35411]: pgmap v7276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:31:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:56 smithi114 ceph-mon[30820]: pgmap v7277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:56 smithi116 ceph-mon[35411]: pgmap v7277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:31:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:31:58 smithi114 ceph-mon[30820]: pgmap v7278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:31:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:31:58 smithi116 ceph-mon[35411]: pgmap v7278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:00 smithi114 ceph-mon[30820]: pgmap v7279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:00 smithi116 ceph-mon[35411]: pgmap v7279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:02 smithi114 ceph-mon[30820]: pgmap v7280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:02 smithi116 ceph-mon[35411]: pgmap v7280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:04 smithi114 ceph-mon[30820]: pgmap v7281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:04 smithi116 ceph-mon[35411]: pgmap v7281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:06 smithi114 ceph-mon[30820]: pgmap v7282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:32:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:06 smithi116 ceph-mon[35411]: pgmap v7282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:32:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:08 smithi114 ceph-mon[30820]: pgmap v7283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:08 smithi116 ceph-mon[35411]: pgmap v7283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:10 smithi114 ceph-mon[30820]: pgmap v7284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:10 smithi116 ceph-mon[35411]: pgmap v7284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:12 smithi114 ceph-mon[30820]: pgmap v7285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:12 smithi116 ceph-mon[35411]: pgmap v7285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:14 smithi114 ceph-mon[30820]: pgmap v7286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:14 smithi116 ceph-mon[35411]: pgmap v7286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:16 smithi114 ceph-mon[30820]: pgmap v7287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:32:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:16 smithi116 ceph-mon[35411]: pgmap v7287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:32:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:18 smithi114 ceph-mon[30820]: pgmap v7288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:18 smithi116 ceph-mon[35411]: pgmap v7288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:20 smithi114 ceph-mon[30820]: pgmap v7289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:20.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:20 smithi116 ceph-mon[35411]: pgmap v7289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:22 smithi114 ceph-mon[30820]: pgmap v7290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:22 smithi116 ceph-mon[35411]: pgmap v7290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:32:23.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:32:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:24 smithi114 ceph-mon[30820]: pgmap v7291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:32:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:24 smithi116 ceph-mon[35411]: pgmap v7291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:32:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-mon[30820]: pgmap v7292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:32:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:32:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:32:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:32:26.388+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:32:26.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:32:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:32:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:32:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:26 smithi116 ceph-mon[35411]: pgmap v7292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:32:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:32:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:32:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:32:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:32:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:32:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:32:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:32:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:32:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:32:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:32:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:32:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:32:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:32:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:27 smithi114 ceph-mon[30820]: pgmap v7293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T10:32:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:27 smithi114 ceph-mon[30820]: pgmap v7294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:32:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:27 smithi116 ceph-mon[35411]: pgmap v7293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T10:32:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:27 smithi116 ceph-mon[35411]: pgmap v7294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:32:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:29 smithi114 ceph-mon[30820]: pgmap v7295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:32:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:29 smithi116 ceph-mon[35411]: pgmap v7295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:32:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:31 smithi114 ceph-mon[30820]: pgmap v7296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:32:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:31 smithi116 ceph-mon[35411]: pgmap v7296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:32:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:33 smithi114 ceph-mon[30820]: pgmap v7297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:32:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:33 smithi116 ceph-mon[35411]: pgmap v7297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:32:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:35 smithi114 ceph-mon[30820]: pgmap v7298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:32:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:35 smithi116 ceph-mon[35411]: pgmap v7298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:32:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:37 smithi114 ceph-mon[30820]: pgmap v7299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:32:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:37 smithi116 ceph-mon[35411]: pgmap v7299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:32:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:39 smithi114 ceph-mon[30820]: pgmap v7300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:39 smithi116 ceph-mon[35411]: pgmap v7300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:41 smithi114 ceph-mon[30820]: pgmap v7301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:41 smithi116 ceph-mon[35411]: pgmap v7301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:43 smithi114 ceph-mon[30820]: pgmap v7302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:43 smithi116 ceph-mon[35411]: pgmap v7302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:45 smithi114 ceph-mon[30820]: pgmap v7303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:32:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:45 smithi116 ceph-mon[35411]: pgmap v7303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:32:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:47 smithi114 ceph-mon[30820]: pgmap v7304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:47 smithi116 ceph-mon[35411]: pgmap v7304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:49 smithi114 ceph-mon[30820]: pgmap v7305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:49 smithi116 ceph-mon[35411]: pgmap v7305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:51 smithi114 ceph-mon[30820]: pgmap v7306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:51 smithi116 ceph-mon[35411]: pgmap v7306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:32:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:53 smithi114 ceph-mon[30820]: pgmap v7307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:53 smithi116 ceph-mon[35411]: pgmap v7307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:32:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:55 smithi116 ceph-mon[35411]: pgmap v7308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:32:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:55 smithi114 ceph-mon[30820]: pgmap v7308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:32:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:57 smithi116 ceph-mon[35411]: pgmap v7309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:57 smithi114 ceph-mon[30820]: pgmap v7309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:32:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:32:59 smithi116 ceph-mon[35411]: pgmap v7310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:32:59 smithi114 ceph-mon[30820]: pgmap v7310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:01.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:01 smithi114 ceph-mon[30820]: pgmap v7311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:01 smithi116 ceph-mon[35411]: pgmap v7311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:03.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:03 smithi114 ceph-mon[30820]: pgmap v7312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:03 smithi116 ceph-mon[35411]: pgmap v7312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:05 smithi114 ceph-mon[30820]: pgmap v7313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:33:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:05 smithi116 ceph-mon[35411]: pgmap v7313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:33:07.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:07 smithi116 ceph-mon[35411]: pgmap v7314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:07 smithi114 ceph-mon[30820]: pgmap v7314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:09 smithi114 ceph-mon[30820]: pgmap v7315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:09 smithi116 ceph-mon[35411]: pgmap v7315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:11 smithi114 ceph-mon[30820]: pgmap v7316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:11 smithi116 ceph-mon[35411]: pgmap v7316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:13 smithi114 ceph-mon[30820]: pgmap v7317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:13 smithi116 ceph-mon[35411]: pgmap v7317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:15 smithi114 ceph-mon[30820]: pgmap v7318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:33:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:15 smithi116 ceph-mon[35411]: pgmap v7318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:33:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:17 smithi114 ceph-mon[30820]: pgmap v7319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:17 smithi116 ceph-mon[35411]: pgmap v7319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:19 smithi114 ceph-mon[30820]: pgmap v7320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:19 smithi116 ceph-mon[35411]: pgmap v7320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:21 smithi114 ceph-mon[30820]: pgmap v7321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:21 smithi116 ceph-mon[35411]: pgmap v7321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:23 smithi114 ceph-mon[30820]: pgmap v7322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:23 smithi116 ceph-mon[35411]: pgmap v7322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:25 smithi114 ceph-mon[30820]: pgmap v7323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:33:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:25 smithi116 ceph-mon[35411]: pgmap v7323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:33:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:27 smithi114 ceph-mon[30820]: pgmap v7324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:33:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:27 smithi116 ceph-mon[35411]: pgmap v7324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:33:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-mon[30820]: pgmap v7325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:29.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:33:29.684+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:33:29.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:33:29.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:33:29.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:33:29.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:33:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:33:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:29 smithi116 ceph-mon[35411]: pgmap v7325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:30 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:33:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:33:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:33:30.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:33:30.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:33:30.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:30 smithi114 ceph-mon[30820]: pgmap v7326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T10:33:30.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:30 smithi114 ceph-mon[30820]: pgmap v7327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:33:30.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:30 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:33:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:33:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:33:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:33:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:33:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:33:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:33:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:33:30.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:33:30.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:33:30.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:33:30.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:30 smithi116 ceph-mon[35411]: pgmap v7326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T10:33:30.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:30 smithi116 ceph-mon[35411]: pgmap v7327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:33:30.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:33:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:32 smithi114 ceph-mon[30820]: pgmap v7328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:33:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:32 smithi116 ceph-mon[35411]: pgmap v7328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:33:35.063 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:34 smithi114 ceph-mon[30820]: pgmap v7329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:33:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:34 smithi116 ceph-mon[35411]: pgmap v7329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:33:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:36 smithi114 ceph-mon[30820]: pgmap v7330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:33:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:36 smithi116 ceph-mon[35411]: pgmap v7330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:33:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:38 smithi114 ceph-mon[30820]: pgmap v7331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:33:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:38 smithi116 ceph-mon[35411]: pgmap v7331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:33:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:40 smithi114 ceph-mon[30820]: pgmap v7332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:33:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:40 smithi116 ceph-mon[35411]: pgmap v7332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:33:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:42 smithi114 ceph-mon[30820]: pgmap v7333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:42 smithi116 ceph-mon[35411]: pgmap v7333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:45.151 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:44 smithi114 ceph-mon[30820]: pgmap v7334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:44 smithi116 ceph-mon[35411]: pgmap v7334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:46 smithi114 ceph-mon[30820]: pgmap v7335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:46 smithi116 ceph-mon[35411]: pgmap v7335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:48 smithi114 ceph-mon[30820]: pgmap v7336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:48 smithi116 ceph-mon[35411]: pgmap v7336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:50 smithi114 ceph-mon[30820]: pgmap v7337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:33:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:50 smithi116 ceph-mon[35411]: pgmap v7337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:33:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:52 smithi114 ceph-mon[30820]: pgmap v7338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:52 smithi116 ceph-mon[35411]: pgmap v7338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:33:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:54 smithi114 ceph-mon[30820]: pgmap v7339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:54 smithi116 ceph-mon[35411]: pgmap v7339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:56 smithi114 ceph-mon[30820]: pgmap v7340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:56 smithi116 ceph-mon[35411]: pgmap v7340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:33:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:33:58 smithi114 ceph-mon[30820]: pgmap v7341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:33:58.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:33:58 smithi116 ceph-mon[35411]: pgmap v7341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:00 smithi114 ceph-mon[30820]: pgmap v7342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:00 smithi116 ceph-mon[35411]: pgmap v7342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:02 smithi114 ceph-mon[30820]: pgmap v7343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:02 smithi116 ceph-mon[35411]: pgmap v7343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:04 smithi114 ceph-mon[30820]: pgmap v7344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:04 smithi116 ceph-mon[35411]: pgmap v7344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:06 smithi114 ceph-mon[30820]: pgmap v7345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:34:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:06 smithi116 ceph-mon[35411]: pgmap v7345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:34:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:08 smithi114 ceph-mon[30820]: pgmap v7346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:08 smithi116 ceph-mon[35411]: pgmap v7346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:10 smithi114 ceph-mon[30820]: pgmap v7347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:10 smithi116 ceph-mon[35411]: pgmap v7347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:12 smithi114 ceph-mon[30820]: pgmap v7348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:12 smithi116 ceph-mon[35411]: pgmap v7348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:14 smithi114 ceph-mon[30820]: pgmap v7349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:14 smithi116 ceph-mon[35411]: pgmap v7349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:16 smithi114 ceph-mon[30820]: pgmap v7350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:34:17.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:16 smithi116 ceph-mon[35411]: pgmap v7350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:34:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:18 smithi114 ceph-mon[30820]: pgmap v7351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:18 smithi116 ceph-mon[35411]: pgmap v7351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:20 smithi114 ceph-mon[30820]: pgmap v7352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:20 smithi116 ceph-mon[35411]: pgmap v7352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:22 smithi114 ceph-mon[30820]: pgmap v7353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:22 smithi116 ceph-mon[35411]: pgmap v7353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:24 smithi114 ceph-mon[30820]: pgmap v7354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:24 smithi116 ceph-mon[35411]: pgmap v7354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:26 smithi114 ceph-mon[30820]: pgmap v7355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:34:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:26 smithi116 ceph-mon[35411]: pgmap v7355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:34:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:28 smithi114 ceph-mon[30820]: pgmap v7356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:28 smithi116 ceph-mon[35411]: pgmap v7356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:30 smithi114 ceph-mon[30820]: pgmap v7357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:34:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:30 smithi116 ceph-mon[35411]: pgmap v7357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:34:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:32 smithi114 ceph-mon[30820]: pgmap v7358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:33.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:34:33.045+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:34:33.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:34:33.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:34:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:34:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:34:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:34:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:34:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:34:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:34:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:34:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:34:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:32 smithi116 ceph-mon[35411]: pgmap v7358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:34.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:34:34.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:34:34.072 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:34:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:34:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:34:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:34:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:34:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:34:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:34:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:34:35.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:34:35.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:34:35.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:34:35.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:34:35.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:34:35.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:34 smithi114 ceph-mon[30820]: pgmap v7359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:34:35.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:34 smithi114 ceph-mon[30820]: pgmap v7360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:34:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:34:35.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:34:35.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:34 smithi116 ceph-mon[35411]: pgmap v7359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:34:35.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:34 smithi116 ceph-mon[35411]: pgmap v7360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:34:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:36 smithi114 ceph-mon[30820]: pgmap v7361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:34:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:36 smithi116 ceph-mon[35411]: pgmap v7361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:34:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:38 smithi114 ceph-mon[30820]: pgmap v7362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:34:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:38 smithi116 ceph-mon[35411]: pgmap v7362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:34:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:40 smithi114 ceph-mon[30820]: pgmap v7363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:34:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:40 smithi116 ceph-mon[35411]: pgmap v7363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:34:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:42 smithi114 ceph-mon[30820]: pgmap v7364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:34:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:42 smithi116 ceph-mon[35411]: pgmap v7364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:34:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:43 smithi114 ceph-mon[30820]: pgmap v7365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:34:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:43 smithi116 ceph-mon[35411]: pgmap v7365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:34:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:46 smithi114 ceph-mon[30820]: pgmap v7366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:46 smithi116 ceph-mon[35411]: pgmap v7366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:48 smithi114 ceph-mon[30820]: pgmap v7367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:48 smithi116 ceph-mon[35411]: pgmap v7367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:50 smithi114 ceph-mon[30820]: pgmap v7368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:50 smithi116 ceph-mon[35411]: pgmap v7368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:52 smithi114 ceph-mon[30820]: pgmap v7369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:34:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:52 smithi116 ceph-mon[35411]: pgmap v7369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:34:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:54 smithi114 ceph-mon[30820]: pgmap v7370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:54 smithi116 ceph-mon[35411]: pgmap v7370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:34:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:56 smithi114 ceph-mon[30820]: pgmap v7371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:56.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:56 smithi116 ceph-mon[35411]: pgmap v7371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:34:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:34:58 smithi114 ceph-mon[30820]: pgmap v7372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:34:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:34:58 smithi116 ceph-mon[35411]: pgmap v7372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:00 smithi114 ceph-mon[30820]: pgmap v7373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:00 smithi116 ceph-mon[35411]: pgmap v7373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:02 smithi114 ceph-mon[30820]: pgmap v7374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:02 smithi116 ceph-mon[35411]: pgmap v7374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:04 smithi114 ceph-mon[30820]: pgmap v7375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:04 smithi116 ceph-mon[35411]: pgmap v7375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:06 smithi114 ceph-mon[30820]: pgmap v7376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:35:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:06 smithi116 ceph-mon[35411]: pgmap v7376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:35:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:08 smithi114 ceph-mon[30820]: pgmap v7377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:08 smithi116 ceph-mon[35411]: pgmap v7377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:10 smithi114 ceph-mon[30820]: pgmap v7378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:10 smithi116 ceph-mon[35411]: pgmap v7378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:12 smithi114 ceph-mon[30820]: pgmap v7379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:12.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:12 smithi116 ceph-mon[35411]: pgmap v7379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:14 smithi114 ceph-mon[30820]: pgmap v7380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:14 smithi116 ceph-mon[35411]: pgmap v7380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:16 smithi114 ceph-mon[30820]: pgmap v7381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:35:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:16 smithi116 ceph-mon[35411]: pgmap v7381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:35:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:18 smithi116 ceph-mon[35411]: pgmap v7382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:18 smithi114 ceph-mon[30820]: pgmap v7382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:20 smithi116 ceph-mon[35411]: pgmap v7383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:20 smithi114 ceph-mon[30820]: pgmap v7383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:22 smithi116 ceph-mon[35411]: pgmap v7384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:22 smithi114 ceph-mon[30820]: pgmap v7384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:24 smithi114 ceph-mon[30820]: pgmap v7385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:24 smithi116 ceph-mon[35411]: pgmap v7385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:26 smithi114 ceph-mon[30820]: pgmap v7386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:35:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:26 smithi116 ceph-mon[35411]: pgmap v7386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:35:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:28 smithi114 ceph-mon[30820]: pgmap v7387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:28 smithi116 ceph-mon[35411]: pgmap v7387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:30 smithi114 ceph-mon[30820]: pgmap v7388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:35:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:30 smithi116 ceph-mon[35411]: pgmap v7388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:35:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:32 smithi114 ceph-mon[30820]: pgmap v7389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:32 smithi116 ceph-mon[35411]: pgmap v7389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:35:33.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:35:34.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:34 smithi114 ceph-mon[30820]: pgmap v7390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:35:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:34 smithi116 ceph-mon[35411]: pgmap v7390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:35:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-mon[30820]: pgmap v7391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:35:36.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:35:36.404+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:35:36.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:35:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:35:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:35:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:35:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:35:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:35:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:35:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:35:36.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:35:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:35:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:35:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:35:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:36 smithi116 ceph-mon[35411]: pgmap v7391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:35:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:37 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:35:37.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:37 smithi114 ceph-mon[30820]: pgmap v7392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T10:35:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:37 smithi114 ceph-mon[30820]: pgmap v7393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:35:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:37 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:35:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:35:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:35:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:35:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:35:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:35:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:35:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:35:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:35:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:35:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:37 smithi116 ceph-mon[35411]: pgmap v7392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T10:35:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:37 smithi116 ceph-mon[35411]: pgmap v7393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:35:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:39 smithi114 ceph-mon[30820]: pgmap v7394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:39 smithi116 ceph-mon[35411]: pgmap v7394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:41 smithi114 ceph-mon[30820]: pgmap v7395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:41 smithi116 ceph-mon[35411]: pgmap v7395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:43 smithi114 ceph-mon[30820]: pgmap v7396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:43 smithi116 ceph-mon[35411]: pgmap v7396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:45 smithi114 ceph-mon[30820]: pgmap v7397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:45 smithi116 ceph-mon[35411]: pgmap v7397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:35:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:47 smithi114 ceph-mon[30820]: pgmap v7398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:35:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:47 smithi116 ceph-mon[35411]: pgmap v7398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:35:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:49 smithi114 ceph-mon[30820]: pgmap v7399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:49.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:49 smithi116 ceph-mon[35411]: pgmap v7399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:51 smithi114 ceph-mon[30820]: pgmap v7400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:51 smithi116 ceph-mon[35411]: pgmap v7400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:35:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:53 smithi114 ceph-mon[30820]: pgmap v7401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:53 smithi116 ceph-mon[35411]: pgmap v7401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:55 smithi114 ceph-mon[30820]: pgmap v7402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:35:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:55 smithi116 ceph-mon[35411]: pgmap v7402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:35:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:57 smithi114 ceph-mon[30820]: pgmap v7403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:57 smithi116 ceph-mon[35411]: pgmap v7403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:35:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:35:59 smithi114 ceph-mon[30820]: pgmap v7404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:35:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:35:59 smithi116 ceph-mon[35411]: pgmap v7404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:01 smithi114 ceph-mon[30820]: pgmap v7405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:01 smithi116 ceph-mon[35411]: pgmap v7405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:03 smithi114 ceph-mon[30820]: pgmap v7406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:03 smithi116 ceph-mon[35411]: pgmap v7406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:05 smithi114 ceph-mon[30820]: pgmap v7407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:36:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:05 smithi116 ceph-mon[35411]: pgmap v7407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:36:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:07 smithi114 ceph-mon[30820]: pgmap v7408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:07 smithi116 ceph-mon[35411]: pgmap v7408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:09 smithi114 ceph-mon[30820]: pgmap v7409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:09 smithi116 ceph-mon[35411]: pgmap v7409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:11 smithi114 ceph-mon[30820]: pgmap v7410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:11.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:11 smithi116 ceph-mon[35411]: pgmap v7410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:13 smithi114 ceph-mon[30820]: pgmap v7411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:13 smithi116 ceph-mon[35411]: pgmap v7411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:15 smithi114 ceph-mon[30820]: pgmap v7412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:36:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:15 smithi116 ceph-mon[35411]: pgmap v7412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:36:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:17 smithi114 ceph-mon[30820]: pgmap v7413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:17 smithi116 ceph-mon[35411]: pgmap v7413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:19 smithi114 ceph-mon[30820]: pgmap v7414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:19 smithi116 ceph-mon[35411]: pgmap v7414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:21 smithi114 ceph-mon[30820]: pgmap v7415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:21 smithi116 ceph-mon[35411]: pgmap v7415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:23 smithi114 ceph-mon[30820]: pgmap v7416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:23 smithi116 ceph-mon[35411]: pgmap v7416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:25 smithi114 ceph-mon[30820]: pgmap v7417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:36:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:25 smithi116 ceph-mon[35411]: pgmap v7417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:36:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:27 smithi114 ceph-mon[30820]: pgmap v7418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:27 smithi116 ceph-mon[35411]: pgmap v7418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:29 smithi114 ceph-mon[30820]: pgmap v7419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:29 smithi116 ceph-mon[35411]: pgmap v7419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:31 smithi114 ceph-mon[30820]: pgmap v7420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:31 smithi116 ceph-mon[35411]: pgmap v7420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:33 smithi114 ceph-mon[30820]: pgmap v7421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:33 smithi116 ceph-mon[35411]: pgmap v7421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:35 smithi114 ceph-mon[30820]: pgmap v7422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:36:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:35 smithi116 ceph-mon[35411]: pgmap v7422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:36:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:36:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:36:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:37 smithi114 ceph-mon[30820]: pgmap v7423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:37 smithi116 ceph-mon[35411]: pgmap v7423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-mon[30820]: pgmap v7424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:36:39.764+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:36:39.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:36:39.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:36:39.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:36:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:36:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:39 smithi116 ceph-mon[35411]: pgmap v7424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:40 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:36:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:36:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:36:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:36:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:36:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:36:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:40 smithi114 ceph-mon[30820]: pgmap v7425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:40 smithi114 ceph-mon[30820]: pgmap v7426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:36:40.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:40 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:36:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:36:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:36:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:36:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:36:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:40 smithi116 ceph-mon[35411]: pgmap v7425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:36:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:40 smithi116 ceph-mon[35411]: pgmap v7426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:36:40.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:36:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:42 smithi114 ceph-mon[30820]: pgmap v7427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:36:43.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:42 smithi116 ceph-mon[35411]: pgmap v7427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:36:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:44 smithi114 ceph-mon[30820]: pgmap v7428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:36:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:44 smithi116 ceph-mon[35411]: pgmap v7428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:36:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:46 smithi114 ceph-mon[30820]: pgmap v7429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:36:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:46 smithi116 ceph-mon[35411]: pgmap v7429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:36:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:48 smithi114 ceph-mon[30820]: pgmap v7430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:36:49.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:48 smithi116 ceph-mon[35411]: pgmap v7430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:36:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:50 smithi114 ceph-mon[30820]: pgmap v7431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:36:51.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:50 smithi116 ceph-mon[35411]: pgmap v7431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:36:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:52 smithi114 ceph-mon[30820]: pgmap v7432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:53.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:52 smithi116 ceph-mon[35411]: pgmap v7432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:36:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:54 smithi114 ceph-mon[30820]: pgmap v7433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:54 smithi116 ceph-mon[35411]: pgmap v7433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:56 smithi114 ceph-mon[30820]: pgmap v7434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:56 smithi116 ceph-mon[35411]: pgmap v7434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:36:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:36:58 smithi114 ceph-mon[30820]: pgmap v7435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:36:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:36:58 smithi116 ceph-mon[35411]: pgmap v7435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:00 smithi114 ceph-mon[30820]: pgmap v7436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:00 smithi116 ceph-mon[35411]: pgmap v7436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:01 smithi114 ceph-mon[30820]: pgmap v7437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:01 smithi116 ceph-mon[35411]: pgmap v7437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:04 smithi114 ceph-mon[30820]: pgmap v7438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:04 smithi116 ceph-mon[35411]: pgmap v7438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:06 smithi114 ceph-mon[30820]: pgmap v7439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:37:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:06 smithi116 ceph-mon[35411]: pgmap v7439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:37:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:08 smithi114 ceph-mon[30820]: pgmap v7440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:09.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:08 smithi116 ceph-mon[35411]: pgmap v7440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:10 smithi114 ceph-mon[30820]: pgmap v7441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:10 smithi116 ceph-mon[35411]: pgmap v7441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:12 smithi114 ceph-mon[30820]: pgmap v7442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:12 smithi116 ceph-mon[35411]: pgmap v7442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:14 smithi114 ceph-mon[30820]: pgmap v7443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:14 smithi116 ceph-mon[35411]: pgmap v7443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:16 smithi114 ceph-mon[30820]: pgmap v7444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:37:17.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:16 smithi116 ceph-mon[35411]: pgmap v7444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:37:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:18 smithi114 ceph-mon[30820]: pgmap v7445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:18 smithi116 ceph-mon[35411]: pgmap v7445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:20 smithi114 ceph-mon[30820]: pgmap v7446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:21.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:20 smithi116 ceph-mon[35411]: pgmap v7446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:21 smithi116 ceph-mon[35411]: pgmap v7447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:21 smithi114 ceph-mon[30820]: pgmap v7447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:24 smithi114 ceph-mon[30820]: pgmap v7448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:24 smithi116 ceph-mon[35411]: pgmap v7448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:26 smithi114 ceph-mon[30820]: pgmap v7449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:37:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:26 smithi116 ceph-mon[35411]: pgmap v7449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:37:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:28 smithi114 ceph-mon[30820]: pgmap v7450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:29.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:28 smithi116 ceph-mon[35411]: pgmap v7450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:30 smithi114 ceph-mon[30820]: pgmap v7451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:30 smithi116 ceph-mon[35411]: pgmap v7451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:32 smithi114 ceph-mon[30820]: pgmap v7452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:33.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:32 smithi116 ceph-mon[35411]: pgmap v7452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:34 smithi114 ceph-mon[30820]: pgmap v7453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:35.273 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:34 smithi116 ceph-mon[35411]: pgmap v7453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:36 smithi114 ceph-mon[30820]: pgmap v7454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:37:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:36 smithi116 ceph-mon[35411]: pgmap v7454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:37:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:38 smithi114 ceph-mon[30820]: pgmap v7455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:38 smithi116 ceph-mon[35411]: pgmap v7455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:37:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:39 smithi116 ceph-mon[35411]: pgmap v7456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:37:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:39 smithi114 ceph-mon[30820]: pgmap v7456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:37:43.126 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:42 smithi114 ceph-mon[30820]: pgmap v7457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:42 smithi116 ceph-mon[35411]: pgmap v7457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:43.416 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:37:43.125+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:37:43.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:37:43.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:37:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:37:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:37:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:37:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:37:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:37:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:37:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:44 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:37:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:37:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:37:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:37:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:37:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:37:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:37:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:37:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:37:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:37:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:37:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:44 smithi114 ceph-mon[30820]: pgmap v7458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:37:45.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:44 smithi114 ceph-mon[30820]: pgmap v7459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:44 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:37:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:37:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:37:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:44 smithi116 ceph-mon[35411]: pgmap v7458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:37:45.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:44 smithi116 ceph-mon[35411]: pgmap v7459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:37:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:46 smithi114 ceph-mon[30820]: pgmap v7460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:37:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:46 smithi116 ceph-mon[35411]: pgmap v7460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:37:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:48 smithi114 ceph-mon[30820]: pgmap v7461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:37:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:48 smithi116 ceph-mon[35411]: pgmap v7461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:37:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:50 smithi114 ceph-mon[30820]: pgmap v7462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:37:51.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:50 smithi116 ceph-mon[35411]: pgmap v7462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:37:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:52 smithi114 ceph-mon[30820]: pgmap v7463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:37:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:52 smithi116 ceph-mon[35411]: pgmap v7463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:37:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:53 smithi116 ceph-mon[35411]: pgmap v7464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:37:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:53 smithi114 ceph-mon[30820]: pgmap v7464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:37:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:56 smithi114 ceph-mon[30820]: pgmap v7465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:56 smithi116 ceph-mon[35411]: pgmap v7465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:37:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:37:58 smithi114 ceph-mon[30820]: pgmap v7466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:37:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:37:58 smithi116 ceph-mon[35411]: pgmap v7466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:00 smithi114 ceph-mon[30820]: pgmap v7467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:00.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:00 smithi116 ceph-mon[35411]: pgmap v7467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:02 smithi114 ceph-mon[30820]: pgmap v7468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:02 smithi116 ceph-mon[35411]: pgmap v7468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:04 smithi114 ceph-mon[30820]: pgmap v7469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:04 smithi116 ceph-mon[35411]: pgmap v7469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:06.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:06 smithi114 ceph-mon[30820]: pgmap v7470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:06 smithi116 ceph-mon[35411]: pgmap v7470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:08 smithi114 ceph-mon[30820]: pgmap v7471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:08 smithi116 ceph-mon[35411]: pgmap v7471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:10.479 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:10 smithi116 ceph-mon[35411]: pgmap v7472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:10 smithi114 ceph-mon[30820]: pgmap v7472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:12 smithi114 ceph-mon[30820]: pgmap v7473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:12.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:12 smithi116 ceph-mon[35411]: pgmap v7473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:14 smithi114 ceph-mon[30820]: pgmap v7474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:14 smithi116 ceph-mon[35411]: pgmap v7474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:16 smithi114 ceph-mon[30820]: pgmap v7475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:16 smithi116 ceph-mon[35411]: pgmap v7475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:18 smithi114 ceph-mon[30820]: pgmap v7476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:18 smithi116 ceph-mon[35411]: pgmap v7476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:20 smithi114 ceph-mon[30820]: pgmap v7477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:20 smithi116 ceph-mon[35411]: pgmap v7477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:22 smithi114 ceph-mon[30820]: pgmap v7478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:22 smithi116 ceph-mon[35411]: pgmap v7478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:24 smithi114 ceph-mon[30820]: pgmap v7479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:24.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:24 smithi116 ceph-mon[35411]: pgmap v7479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:26 smithi114 ceph-mon[30820]: pgmap v7480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:26 smithi116 ceph-mon[35411]: pgmap v7480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:28 smithi114 ceph-mon[30820]: pgmap v7481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:28 smithi116 ceph-mon[35411]: pgmap v7481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:30 smithi114 ceph-mon[30820]: pgmap v7482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:30 smithi116 ceph-mon[35411]: pgmap v7482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:32 smithi114 ceph-mon[30820]: pgmap v7483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:32.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:32 smithi116 ceph-mon[35411]: pgmap v7483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:34 smithi114 ceph-mon[30820]: pgmap v7484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:34.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:34 smithi116 ceph-mon[35411]: pgmap v7484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:36 smithi114 ceph-mon[30820]: pgmap v7485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:36 smithi116 ceph-mon[35411]: pgmap v7485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:38 smithi114 ceph-mon[30820]: pgmap v7486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:38.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:38 smithi116 ceph-mon[35411]: pgmap v7486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:38:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:40 smithi114 ceph-mon[30820]: pgmap v7487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:40.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:40 smithi116 ceph-mon[35411]: pgmap v7487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:42 smithi114 ceph-mon[30820]: pgmap v7488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:42 smithi116 ceph-mon[35411]: pgmap v7488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:38:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:38:43.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:38:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:44 smithi114 ceph-mon[30820]: pgmap v7489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:44.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:44 smithi116 ceph-mon[35411]: pgmap v7489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-mon[30820]: pgmap v7490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:46.666 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:38:46.484+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:38:46.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:38:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:38:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:38:46.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:46 smithi116 ceph-mon[35411]: pgmap v7490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:38:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:47 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:38:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:38:47.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:38:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:38:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:38:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:47 smithi114 ceph-mon[30820]: pgmap v7491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:38:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:47 smithi114 ceph-mon[30820]: pgmap v7492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:47 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:38:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:38:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:38:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:38:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:38:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:38:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:47 smithi116 ceph-mon[35411]: pgmap v7491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T10:38:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:47 smithi116 ceph-mon[35411]: pgmap v7492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:38:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:49 smithi114 ceph-mon[30820]: pgmap v7493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:49 smithi116 ceph-mon[35411]: pgmap v7493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:51 smithi114 ceph-mon[30820]: pgmap v7494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:51 smithi116 ceph-mon[35411]: pgmap v7494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:53 smithi114 ceph-mon[30820]: pgmap v7495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:53 smithi116 ceph-mon[35411]: pgmap v7495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:55 smithi114 ceph-mon[30820]: pgmap v7496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:55 smithi116 ceph-mon[35411]: pgmap v7496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:38:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:57 smithi114 ceph-mon[30820]: pgmap v7497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:38:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:57 smithi116 ceph-mon[35411]: pgmap v7497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:38:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:38:59 smithi114 ceph-mon[30820]: pgmap v7498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:38:59.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:38:59 smithi116 ceph-mon[35411]: pgmap v7498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:01 smithi114 ceph-mon[30820]: pgmap v7499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:01 smithi116 ceph-mon[35411]: pgmap v7499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:03 smithi114 ceph-mon[30820]: pgmap v7500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:03 smithi116 ceph-mon[35411]: pgmap v7500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:05 smithi114 ceph-mon[30820]: pgmap v7501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:05 smithi116 ceph-mon[35411]: pgmap v7501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:07 smithi114 ceph-mon[30820]: pgmap v7502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:07 smithi116 ceph-mon[35411]: pgmap v7502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:09 smithi114 ceph-mon[30820]: pgmap v7503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:09.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:09 smithi116 ceph-mon[35411]: pgmap v7503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:11 smithi114 ceph-mon[30820]: pgmap v7504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:11 smithi116 ceph-mon[35411]: pgmap v7504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:13 smithi114 ceph-mon[30820]: pgmap v7505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:13 smithi116 ceph-mon[35411]: pgmap v7505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:15 smithi114 ceph-mon[30820]: pgmap v7506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:15 smithi116 ceph-mon[35411]: pgmap v7506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:17 smithi114 ceph-mon[30820]: pgmap v7507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:17 smithi116 ceph-mon[35411]: pgmap v7507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:19 smithi114 ceph-mon[30820]: pgmap v7508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:19 smithi116 ceph-mon[35411]: pgmap v7508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:21 smithi114 ceph-mon[30820]: pgmap v7509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:21 smithi116 ceph-mon[35411]: pgmap v7509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:23 smithi114 ceph-mon[30820]: pgmap v7510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:23 smithi116 ceph-mon[35411]: pgmap v7510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:25 smithi114 ceph-mon[30820]: pgmap v7511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:25.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:25 smithi116 ceph-mon[35411]: pgmap v7511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:27 smithi114 ceph-mon[30820]: pgmap v7512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:27.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:27 smithi116 ceph-mon[35411]: pgmap v7512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:29 smithi114 ceph-mon[30820]: pgmap v7513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:29 smithi116 ceph-mon[35411]: pgmap v7513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:31 smithi114 ceph-mon[30820]: pgmap v7514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:31.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:31 smithi116 ceph-mon[35411]: pgmap v7514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:33 smithi114 ceph-mon[30820]: pgmap v7515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:33 smithi116 ceph-mon[35411]: pgmap v7515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:35 smithi114 ceph-mon[30820]: pgmap v7516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:35.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:35 smithi116 ceph-mon[35411]: pgmap v7516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:37 smithi114 ceph-mon[30820]: pgmap v7517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:37.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:37 smithi116 ceph-mon[35411]: pgmap v7517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:39 smithi114 ceph-mon[30820]: pgmap v7518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:39.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:39 smithi116 ceph-mon[35411]: pgmap v7518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:41 smithi114 ceph-mon[30820]: pgmap v7519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:41 smithi116 ceph-mon[35411]: pgmap v7519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:39:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:43 smithi114 ceph-mon[30820]: pgmap v7520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:43.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:43 smithi116 ceph-mon[35411]: pgmap v7520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:45 smithi114 ceph-mon[30820]: pgmap v7521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:45.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:45 smithi116 ceph-mon[35411]: pgmap v7521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:39:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:39:46.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:39:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: pgmap v7522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: pgmap v7522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:39:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:48 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:39:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:48 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:39:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:48 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:39:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:48 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:39:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:48 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:39:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:48 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:39:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:48 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:39:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:48 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:39:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:49 smithi116 ceph-mon[35411]: pgmap v7523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-mon[30820]: pgmap v7523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:39:50.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:39:49.845+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:39:50.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:39:50.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:39:50.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:39:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:39:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:39:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:50 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:39:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:50 smithi114 ceph-mon[30820]: pgmap v7524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:50 smithi114 ceph-mon[30820]: pgmap v7525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:39:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:50 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:39:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:50 smithi116 ceph-mon[35411]: pgmap v7524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:39:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:50 smithi116 ceph-mon[35411]: pgmap v7525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:39:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:39:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:51 smithi114 ceph-mon[30820]: pgmap v7526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:39:52.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:51 smithi116 ceph-mon[35411]: pgmap v7526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:39:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:53 smithi114 ceph-mon[30820]: pgmap v7527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:39:54.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:53 smithi116 ceph-mon[35411]: pgmap v7527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:39:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:55 smithi114 ceph-mon[30820]: pgmap v7528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:39:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:55 smithi116 ceph-mon[35411]: pgmap v7528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:39:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:57 smithi114 ceph-mon[30820]: pgmap v7529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:39:58.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:57 smithi116 ceph-mon[35411]: pgmap v7529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:40:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:39:59 smithi114 ceph-mon[30820]: pgmap v7530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:40:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:39:59 smithi116 ceph-mon[35411]: pgmap v7530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:40:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:40:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:40:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:01 smithi114 ceph-mon[30820]: pgmap v7531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:01 smithi116 ceph-mon[35411]: pgmap v7531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:03 smithi114 ceph-mon[30820]: pgmap v7532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:04.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:03 smithi116 ceph-mon[35411]: pgmap v7532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:05 smithi114 ceph-mon[30820]: pgmap v7533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:05 smithi116 ceph-mon[35411]: pgmap v7533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:08.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:07 smithi114 ceph-mon[30820]: pgmap v7534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:07 smithi116 ceph-mon[35411]: pgmap v7534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:09 smithi114 ceph-mon[30820]: pgmap v7535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:40:10.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:09 smithi116 ceph-mon[35411]: pgmap v7535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:40:12.229 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:11 smithi116 ceph-mon[35411]: pgmap v7536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:11 smithi114 ceph-mon[30820]: pgmap v7536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:13 smithi114 ceph-mon[30820]: pgmap v7537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:40:14.455 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:13 smithi116 ceph-mon[35411]: pgmap v7537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:40:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:15 smithi114 ceph-mon[30820]: pgmap v7538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:15 smithi116 ceph-mon[35411]: pgmap v7538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:18.328 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:17 smithi114 ceph-mon[30820]: pgmap v7539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:40:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:17 smithi116 ceph-mon[35411]: pgmap v7539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:40:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:19 smithi114 ceph-mon[30820]: pgmap v7540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:40:20.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:19 smithi116 ceph-mon[35411]: pgmap v7540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:40:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:21 smithi114 ceph-mon[30820]: pgmap v7541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:21 smithi116 ceph-mon[35411]: pgmap v7541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:23 smithi114 ceph-mon[30820]: pgmap v7542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:24.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:23 smithi116 ceph-mon[35411]: pgmap v7542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:25 smithi114 ceph-mon[30820]: pgmap v7543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:25 smithi116 ceph-mon[35411]: pgmap v7543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:27 smithi114 ceph-mon[30820]: pgmap v7544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:28.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:27 smithi116 ceph-mon[35411]: pgmap v7544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:29 smithi114 ceph-mon[30820]: pgmap v7545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:40:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:29 smithi116 ceph-mon[35411]: pgmap v7545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:40:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:31 smithi114 ceph-mon[30820]: pgmap v7546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:32.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:31 smithi116 ceph-mon[35411]: pgmap v7546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:33 smithi114 ceph-mon[30820]: pgmap v7547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:34.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:33 smithi116 ceph-mon[35411]: pgmap v7547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:35 smithi114 ceph-mon[30820]: pgmap v7548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:36.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:35 smithi116 ceph-mon[35411]: pgmap v7548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:38.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:37 smithi114 ceph-mon[30820]: pgmap v7549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:37 smithi116 ceph-mon[35411]: pgmap v7549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:39 smithi114 ceph-mon[30820]: pgmap v7550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:40:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:39 smithi116 ceph-mon[35411]: pgmap v7550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:40:42.229 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:41 smithi116 ceph-mon[35411]: pgmap v7551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:41 smithi114 ceph-mon[30820]: pgmap v7551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:43 smithi114 ceph-mon[30820]: pgmap v7552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:43 smithi116 ceph-mon[35411]: pgmap v7552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:45 smithi114 ceph-mon[30820]: pgmap v7553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:46.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:45 smithi116 ceph-mon[35411]: pgmap v7553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:40:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:47 smithi114 ceph-mon[30820]: pgmap v7554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:47 smithi116 ceph-mon[35411]: pgmap v7554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:40:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:49 smithi114 ceph-mon[30820]: pgmap v7555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:40:50.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:49 smithi116 ceph-mon[35411]: pgmap v7555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:40:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:40:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:40:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:51 smithi114 ceph-mon[30820]: pgmap v7556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:52.427 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:51 smithi116 ceph-mon[35411]: pgmap v7556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:40:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:40:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:40:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:40:53.204+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:40:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:40:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:40:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:40:53.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:40:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:40:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:40:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:40:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:40:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:40:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:54 smithi114 ceph-mon[30820]: pgmap v7557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:40:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:54 smithi114 ceph-mon[30820]: pgmap v7558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:40:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:40:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:54 smithi116 ceph-mon[35411]: pgmap v7557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:40:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:54 smithi116 ceph-mon[35411]: pgmap v7558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:40:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:56 smithi114 ceph-mon[30820]: pgmap v7559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:40:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:56 smithi116 ceph-mon[35411]: pgmap v7559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:40:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:40:58 smithi114 ceph-mon[30820]: pgmap v7560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:40:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:40:58 smithi116 ceph-mon[35411]: pgmap v7560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:41:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:00 smithi114 ceph-mon[30820]: pgmap v7561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:41:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:00 smithi116 ceph-mon[35411]: pgmap v7561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:41:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:02 smithi114 ceph-mon[30820]: pgmap v7562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:41:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:02 smithi116 ceph-mon[35411]: pgmap v7562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:41:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:04 smithi114 ceph-mon[30820]: pgmap v7563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:41:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:04 smithi116 ceph-mon[35411]: pgmap v7563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:41:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:06 smithi114 ceph-mon[30820]: pgmap v7564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:06 smithi116 ceph-mon[35411]: pgmap v7564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:08 smithi114 ceph-mon[30820]: pgmap v7565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:08 smithi116 ceph-mon[35411]: pgmap v7565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:10.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:10 smithi114 ceph-mon[30820]: pgmap v7566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:10 smithi116 ceph-mon[35411]: pgmap v7566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:12 smithi116 ceph-mon[35411]: pgmap v7567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:12 smithi114 ceph-mon[30820]: pgmap v7567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:14 smithi116 ceph-mon[35411]: pgmap v7568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:14 smithi114 ceph-mon[30820]: pgmap v7568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:16.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:16 smithi116 ceph-mon[35411]: pgmap v7569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:16 smithi114 ceph-mon[30820]: pgmap v7569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:18 smithi114 ceph-mon[30820]: pgmap v7570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:18 smithi116 ceph-mon[35411]: pgmap v7570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:20 smithi114 ceph-mon[30820]: pgmap v7571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:20.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:20 smithi116 ceph-mon[35411]: pgmap v7571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:22 smithi114 ceph-mon[30820]: pgmap v7572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:22 smithi116 ceph-mon[35411]: pgmap v7572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:24 smithi114 ceph-mon[30820]: pgmap v7573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:24.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:24 smithi116 ceph-mon[35411]: pgmap v7573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:26 smithi114 ceph-mon[30820]: pgmap v7574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:26 smithi116 ceph-mon[35411]: pgmap v7574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:28 smithi114 ceph-mon[30820]: pgmap v7575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:28.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:28 smithi116 ceph-mon[35411]: pgmap v7575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:30 smithi114 ceph-mon[30820]: pgmap v7576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:30.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:30 smithi116 ceph-mon[35411]: pgmap v7576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:32 smithi114 ceph-mon[30820]: pgmap v7577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:32 smithi116 ceph-mon[35411]: pgmap v7577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:34 smithi114 ceph-mon[30820]: pgmap v7578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:34.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:34 smithi116 ceph-mon[35411]: pgmap v7578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:36 smithi114 ceph-mon[30820]: pgmap v7579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:36.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:36 smithi116 ceph-mon[35411]: pgmap v7579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:38.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:38 smithi114 ceph-mon[30820]: pgmap v7580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:38 smithi116 ceph-mon[35411]: pgmap v7580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:40 smithi114 ceph-mon[30820]: pgmap v7581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:40.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:40 smithi116 ceph-mon[35411]: pgmap v7581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:42 smithi114 ceph-mon[30820]: pgmap v7582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:42.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:42 smithi116 ceph-mon[35411]: pgmap v7582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:44 smithi114 ceph-mon[30820]: pgmap v7583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:44 smithi116 ceph-mon[35411]: pgmap v7583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:46 smithi114 ceph-mon[30820]: pgmap v7584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T10:41:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:46 smithi116 ceph-mon[35411]: pgmap v7584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T10:41:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:48 smithi114 ceph-mon[30820]: pgmap v7585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:48.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:48 smithi116 ceph-mon[35411]: pgmap v7585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:41:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:50 smithi114 ceph-mon[30820]: pgmap v7586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:50.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:50 smithi116 ceph-mon[35411]: pgmap v7586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:52 smithi114 ceph-mon[30820]: pgmap v7587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:52.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:52 smithi116 ceph-mon[35411]: pgmap v7587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:41:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:41:53.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:41:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:54 smithi114 ceph-mon[30820]: pgmap v7588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:54.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:54 smithi116 ceph-mon[35411]: pgmap v7588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:41:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-mon[30820]: pgmap v7589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:56.916 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:41:56.564+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:41:56.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:41:56.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:41:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:41:56.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:56 smithi116 ceph-mon[35411]: pgmap v7589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:41:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:57 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:41:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:41:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:41:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:41:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:57 smithi114 ceph-mon[30820]: pgmap v7590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:57 smithi114 ceph-mon[30820]: pgmap v7591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:41:57.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:41:57.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:57 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:41:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:41:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:41:57.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:57 smithi116 ceph-mon[35411]: pgmap v7590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:41:57.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:57 smithi116 ceph-mon[35411]: pgmap v7591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:41:57.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:41:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:41:59 smithi114 ceph-mon[30820]: pgmap v7592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:41:59.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:41:59 smithi116 ceph-mon[35411]: pgmap v7592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:42:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:01 smithi114 ceph-mon[30820]: pgmap v7593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:42:01.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:01 smithi116 ceph-mon[35411]: pgmap v7593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:42:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:03 smithi114 ceph-mon[30820]: pgmap v7594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:42:03.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:03 smithi116 ceph-mon[35411]: pgmap v7594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:42:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:05 smithi114 ceph-mon[30820]: pgmap v7595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:42:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:05 smithi116 ceph-mon[35411]: pgmap v7595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:42:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:07 smithi114 ceph-mon[30820]: pgmap v7596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:42:07.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:07 smithi116 ceph-mon[35411]: pgmap v7596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:42:09.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:09 smithi114 ceph-mon[30820]: pgmap v7597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:09 smithi116 ceph-mon[35411]: pgmap v7597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:11 smithi114 ceph-mon[30820]: pgmap v7598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:11 smithi116 ceph-mon[35411]: pgmap v7598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:13.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:13 smithi116 ceph-mon[35411]: pgmap v7599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:13 smithi114 ceph-mon[30820]: pgmap v7599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:15 smithi114 ceph-mon[30820]: pgmap v7600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:42:15.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:15 smithi116 ceph-mon[35411]: pgmap v7600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:42:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:17 smithi114 ceph-mon[30820]: pgmap v7601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:17 smithi116 ceph-mon[35411]: pgmap v7601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:19 smithi114 ceph-mon[30820]: pgmap v7602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:19 smithi116 ceph-mon[35411]: pgmap v7602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:21 smithi114 ceph-mon[30820]: pgmap v7603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:21 smithi116 ceph-mon[35411]: pgmap v7603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:23 smithi114 ceph-mon[30820]: pgmap v7604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:42:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:23 smithi116 ceph-mon[35411]: pgmap v7604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:42:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:25 smithi114 ceph-mon[30820]: pgmap v7605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:42:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:25 smithi116 ceph-mon[35411]: pgmap v7605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:42:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:27 smithi114 ceph-mon[30820]: pgmap v7606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:27 smithi116 ceph-mon[35411]: pgmap v7606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:29 smithi114 ceph-mon[30820]: pgmap v7607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:42:29.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:29 smithi116 ceph-mon[35411]: pgmap v7607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:42:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:31 smithi114 ceph-mon[30820]: pgmap v7608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:31 smithi116 ceph-mon[35411]: pgmap v7608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:33 smithi114 ceph-mon[30820]: pgmap v7609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:33.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:33 smithi116 ceph-mon[35411]: pgmap v7609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:35 smithi116 ceph-mon[35411]: pgmap v7610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:42:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:35 smithi114 ceph-mon[30820]: pgmap v7610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:42:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:37 smithi116 ceph-mon[35411]: pgmap v7611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:37 smithi114 ceph-mon[30820]: pgmap v7611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:39 smithi116 ceph-mon[35411]: pgmap v7612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:39.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:39 smithi114 ceph-mon[30820]: pgmap v7612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:41 smithi114 ceph-mon[30820]: pgmap v7613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:41 smithi116 ceph-mon[35411]: pgmap v7613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:43.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:43 smithi116 ceph-mon[35411]: pgmap v7614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:43 smithi114 ceph-mon[30820]: pgmap v7614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:45 smithi114 ceph-mon[30820]: pgmap v7615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:42:46.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:45 smithi116 ceph-mon[35411]: pgmap v7615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:42:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:47 smithi114 ceph-mon[30820]: pgmap v7616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:48.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:47 smithi116 ceph-mon[35411]: pgmap v7616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:49 smithi114 ceph-mon[30820]: pgmap v7617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:49 smithi116 ceph-mon[35411]: pgmap v7617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:51 smithi114 ceph-mon[30820]: pgmap v7618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:51 smithi116 ceph-mon[35411]: pgmap v7618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:42:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:53 smithi114 ceph-mon[30820]: pgmap v7619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:54.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:53 smithi116 ceph-mon[35411]: pgmap v7619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:55 smithi114 ceph-mon[30820]: pgmap v7620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:42:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:55 smithi116 ceph-mon[35411]: pgmap v7620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:42:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:42:57.177 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:42:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:57 smithi114 ceph-mon[30820]: pgmap v7621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:57 smithi116 ceph-mon[35411]: pgmap v7621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:42:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:42:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:42:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:59 smithi116 ceph-mon[35411]: pgmap v7622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:42:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:42:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:42:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:42:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:42:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-mon[30820]: pgmap v7622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:42:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:42:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:43:00.183 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:42:59.924+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:43:00.184 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:43:00.185 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:42:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:43:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:00 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:43:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:43:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:43:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:43:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:43:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:43:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:43:00.666 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:43:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:43:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:43:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:43:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:43:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:43:00.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:00 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:43:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:43:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:43:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:43:00.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:43:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:01 smithi114 ceph-mon[30820]: pgmap v7623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:43:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:01 smithi114 ceph-mon[30820]: pgmap v7624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:43:01.677 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:01 smithi116 ceph-mon[35411]: pgmap v7623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:43:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:01 smithi116 ceph-mon[35411]: pgmap v7624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:43:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:03 smithi116 ceph-mon[35411]: pgmap v7625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:43:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:03 smithi114 ceph-mon[30820]: pgmap v7625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:43:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:05 smithi114 ceph-mon[30820]: pgmap v7626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:43:05.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:05 smithi116 ceph-mon[35411]: pgmap v7626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:43:07.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:07 smithi114 ceph-mon[30820]: pgmap v7627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:43:07.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:07 smithi116 ceph-mon[35411]: pgmap v7627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:43:09.718 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:09 smithi114 ceph-mon[30820]: pgmap v7628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:43:09.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:09 smithi116 ceph-mon[35411]: pgmap v7628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:43:11.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:11 smithi114 ceph-mon[30820]: pgmap v7629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:43:11.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:11 smithi116 ceph-mon[35411]: pgmap v7629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:43:13.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:13 smithi116 ceph-mon[35411]: pgmap v7630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:13 smithi114 ceph-mon[30820]: pgmap v7630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:15.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:15 smithi114 ceph-mon[30820]: pgmap v7631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:15.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:15 smithi116 ceph-mon[35411]: pgmap v7631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:17.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:17 smithi114 ceph-mon[30820]: pgmap v7632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:17.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:17 smithi116 ceph-mon[35411]: pgmap v7632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:19 smithi114 ceph-mon[30820]: pgmap v7633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:19 smithi116 ceph-mon[35411]: pgmap v7633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:21.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:21 smithi114 ceph-mon[30820]: pgmap v7634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:43:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:21 smithi116 ceph-mon[35411]: pgmap v7634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:43:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:23 smithi114 ceph-mon[30820]: pgmap v7635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:23.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:23 smithi116 ceph-mon[35411]: pgmap v7635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:25 smithi114 ceph-mon[30820]: pgmap v7636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:25 smithi116 ceph-mon[35411]: pgmap v7636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:27 smithi114 ceph-mon[30820]: pgmap v7637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:27 smithi116 ceph-mon[35411]: pgmap v7637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:29.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:29 smithi114 ceph-mon[30820]: pgmap v7638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:29 smithi116 ceph-mon[35411]: pgmap v7638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:31.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:31 smithi114 ceph-mon[30820]: pgmap v7639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:43:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:31 smithi116 ceph-mon[35411]: pgmap v7639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:43:33.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:33 smithi114 ceph-mon[30820]: pgmap v7640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:33 smithi116 ceph-mon[35411]: pgmap v7640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:35 smithi114 ceph-mon[30820]: pgmap v7641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:35 smithi116 ceph-mon[35411]: pgmap v7641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:37.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:37 smithi114 ceph-mon[30820]: pgmap v7642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:37 smithi116 ceph-mon[35411]: pgmap v7642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:39.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:39 smithi114 ceph-mon[30820]: pgmap v7643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:39 smithi116 ceph-mon[35411]: pgmap v7643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:41 smithi114 ceph-mon[30820]: pgmap v7644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:43:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:41 smithi116 ceph-mon[35411]: pgmap v7644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:43:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:43 smithi114 ceph-mon[30820]: pgmap v7645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:43 smithi116 ceph-mon[35411]: pgmap v7645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:45 smithi114 ceph-mon[30820]: pgmap v7646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:45 smithi116 ceph-mon[35411]: pgmap v7646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:47.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:47 smithi114 ceph-mon[30820]: pgmap v7647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:47 smithi116 ceph-mon[35411]: pgmap v7647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:49.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:49 smithi114 ceph-mon[30820]: pgmap v7648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:49 smithi116 ceph-mon[35411]: pgmap v7648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:51.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:51 smithi114 ceph-mon[30820]: pgmap v7649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:43:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:51 smithi116 ceph-mon[35411]: pgmap v7649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:43:53.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:53 smithi114 ceph-mon[30820]: pgmap v7650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:53 smithi116 ceph-mon[35411]: pgmap v7650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:43:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:55 smithi114 ceph-mon[30820]: pgmap v7651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:55.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:55 smithi116 ceph-mon[35411]: pgmap v7651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:43:57.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:57 smithi114 ceph-mon[30820]: pgmap v7652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:57.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:57 smithi116 ceph-mon[35411]: pgmap v7652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:43:59.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:43:59 smithi114 ceph-mon[30820]: pgmap v7653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:00.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:43:59 smithi116 ceph-mon[35411]: pgmap v7653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:44:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:44:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:01.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:01 smithi114 ceph-mon[30820]: pgmap v7654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:44:01.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:01 smithi116 ceph-mon[35411]: pgmap v7654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:44:03.550 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:44:03.220+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:44:03.551 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:44:03.552 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:44:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-mon[30820]: pgmap v7655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:44:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:44:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:44:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:03 smithi116 ceph-mon[35411]: pgmap v7655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:44:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:44:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:44:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:44:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:44:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:44:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:44:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:44:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:44:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:44:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:44:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:44:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:44:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:44:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:04 smithi114 ceph-mon[30820]: pgmap v7656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:44:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:04 smithi114 ceph-mon[30820]: pgmap v7657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:44:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:04 smithi116 ceph-mon[35411]: pgmap v7656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:44:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:04 smithi116 ceph-mon[35411]: pgmap v7657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:44:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:06 smithi114 ceph-mon[30820]: pgmap v7658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:44:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:06 smithi116 ceph-mon[35411]: pgmap v7658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:44:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:08 smithi114 ceph-mon[30820]: pgmap v7659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:44:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:08 smithi116 ceph-mon[35411]: pgmap v7659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:44:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:10 smithi114 ceph-mon[30820]: pgmap v7660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:44:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:10 smithi116 ceph-mon[35411]: pgmap v7660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:44:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:12 smithi114 ceph-mon[30820]: pgmap v7661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:44:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:12 smithi116 ceph-mon[35411]: pgmap v7661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:44:14.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:14 smithi114 ceph-mon[30820]: pgmap v7662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:44:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:14 smithi116 ceph-mon[35411]: pgmap v7662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:44:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:16 smithi114 ceph-mon[30820]: pgmap v7663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:16 smithi116 ceph-mon[35411]: pgmap v7663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:18 smithi114 ceph-mon[30820]: pgmap v7664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:18 smithi116 ceph-mon[35411]: pgmap v7664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:20 smithi114 ceph-mon[30820]: pgmap v7665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:20 smithi116 ceph-mon[35411]: pgmap v7665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:22 smithi114 ceph-mon[30820]: pgmap v7666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:44:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:22 smithi116 ceph-mon[35411]: pgmap v7666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:44:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:24 smithi114 ceph-mon[30820]: pgmap v7667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:24 smithi116 ceph-mon[35411]: pgmap v7667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:26 smithi114 ceph-mon[30820]: pgmap v7668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:26 smithi116 ceph-mon[35411]: pgmap v7668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:28 smithi114 ceph-mon[30820]: pgmap v7669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:28 smithi116 ceph-mon[35411]: pgmap v7669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:30 smithi114 ceph-mon[30820]: pgmap v7670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:30 smithi116 ceph-mon[35411]: pgmap v7670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:32 smithi114 ceph-mon[30820]: pgmap v7671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:44:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:32 smithi116 ceph-mon[35411]: pgmap v7671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:44:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:34 smithi114 ceph-mon[30820]: pgmap v7672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:34 smithi116 ceph-mon[35411]: pgmap v7672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:36 smithi114 ceph-mon[30820]: pgmap v7673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:36 smithi116 ceph-mon[35411]: pgmap v7673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:38 smithi114 ceph-mon[30820]: pgmap v7674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:38.927 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:38 smithi116 ceph-mon[35411]: pgmap v7674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:40 smithi114 ceph-mon[30820]: pgmap v7675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:40 smithi116 ceph-mon[35411]: pgmap v7675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:42 smithi114 ceph-mon[30820]: pgmap v7676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:44:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:42 smithi116 ceph-mon[35411]: pgmap v7676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:44:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:44 smithi114 ceph-mon[30820]: pgmap v7677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:44 smithi116 ceph-mon[35411]: pgmap v7677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:46 smithi114 ceph-mon[30820]: pgmap v7678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:46 smithi116 ceph-mon[35411]: pgmap v7678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:48 smithi114 ceph-mon[30820]: pgmap v7679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:48 smithi116 ceph-mon[35411]: pgmap v7679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:50 smithi114 ceph-mon[30820]: pgmap v7680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:50 smithi116 ceph-mon[35411]: pgmap v7680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:52 smithi114 ceph-mon[30820]: pgmap v7681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:44:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:52 smithi116 ceph-mon[35411]: pgmap v7681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:44:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:54 smithi114 ceph-mon[30820]: pgmap v7682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:54 smithi116 ceph-mon[35411]: pgmap v7682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:44:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:56 smithi114 ceph-mon[30820]: pgmap v7683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:56 smithi116 ceph-mon[35411]: pgmap v7683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:44:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:44:58 smithi114 ceph-mon[30820]: pgmap v7684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:44:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:44:58 smithi116 ceph-mon[35411]: pgmap v7684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:45:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:00 smithi114 ceph-mon[30820]: pgmap v7685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:00 smithi116 ceph-mon[35411]: pgmap v7685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:02 smithi114 ceph-mon[30820]: pgmap v7686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:02 smithi116 ceph-mon[35411]: pgmap v7686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:45:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:45:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:04 smithi116 ceph-mon[35411]: pgmap v7687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:04 smithi114 ceph-mon[30820]: pgmap v7687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-mon[30820]: pgmap v7688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:45:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:45:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:45:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:45:06.581+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:45:06.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:45:06.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:45:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:45:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:06 smithi116 ceph-mon[35411]: pgmap v7688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:45:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:45:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:45:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:45:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:45:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:45:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:45:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:45:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:45:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:07 smithi114 ceph-mon[30820]: pgmap v7689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:45:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:07 smithi114 ceph-mon[30820]: pgmap v7690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:45:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:45:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:45:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:45:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:45:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:07 smithi116 ceph-mon[35411]: pgmap v7689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:45:08.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:07 smithi116 ceph-mon[35411]: pgmap v7690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:45:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:09 smithi114 ceph-mon[30820]: pgmap v7691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:45:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:09 smithi116 ceph-mon[35411]: pgmap v7691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:45:11.968 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:11 smithi114 ceph-mon[30820]: pgmap v7692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:45:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:11 smithi116 ceph-mon[35411]: pgmap v7692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:45:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:13 smithi114 ceph-mon[30820]: pgmap v7693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:45:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:13 smithi116 ceph-mon[35411]: pgmap v7693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:45:15.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:15 smithi116 ceph-mon[35411]: pgmap v7694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:45:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:15 smithi114 ceph-mon[30820]: pgmap v7694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:45:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:17 smithi114 ceph-mon[30820]: pgmap v7695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:45:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:17 smithi116 ceph-mon[35411]: pgmap v7695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:45:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:19 smithi114 ceph-mon[30820]: pgmap v7696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:19 smithi116 ceph-mon[35411]: pgmap v7696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:21 smithi114 ceph-mon[30820]: pgmap v7697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:21 smithi116 ceph-mon[35411]: pgmap v7697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:23 smithi114 ceph-mon[30820]: pgmap v7698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:23 smithi116 ceph-mon[35411]: pgmap v7698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:25 smithi114 ceph-mon[30820]: pgmap v7699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:45:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:25 smithi116 ceph-mon[35411]: pgmap v7699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:45:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:27 smithi114 ceph-mon[30820]: pgmap v7700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:45:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:27 smithi116 ceph-mon[35411]: pgmap v7700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:45:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:29 smithi114 ceph-mon[30820]: pgmap v7701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:29 smithi116 ceph-mon[35411]: pgmap v7701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:31 smithi114 ceph-mon[30820]: pgmap v7702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:31 smithi116 ceph-mon[35411]: pgmap v7702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:33 smithi114 ceph-mon[30820]: pgmap v7703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:33 smithi116 ceph-mon[35411]: pgmap v7703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:35 smithi114 ceph-mon[30820]: pgmap v7704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:45:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:35 smithi116 ceph-mon[35411]: pgmap v7704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:45:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:37 smithi114 ceph-mon[30820]: pgmap v7705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:45:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:37 smithi116 ceph-mon[35411]: pgmap v7705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:45:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:39 smithi114 ceph-mon[30820]: pgmap v7706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:39 smithi116 ceph-mon[35411]: pgmap v7706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:41 smithi114 ceph-mon[30820]: pgmap v7707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:41 smithi116 ceph-mon[35411]: pgmap v7707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:43 smithi114 ceph-mon[30820]: pgmap v7708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:43 smithi116 ceph-mon[35411]: pgmap v7708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:45 smithi114 ceph-mon[30820]: pgmap v7709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:45:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:45 smithi116 ceph-mon[35411]: pgmap v7709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:45:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:47 smithi114 ceph-mon[30820]: pgmap v7710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:45:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:47 smithi116 ceph-mon[35411]: pgmap v7710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:45:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:49 smithi114 ceph-mon[30820]: pgmap v7711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:49 smithi116 ceph-mon[35411]: pgmap v7711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:51 smithi114 ceph-mon[30820]: pgmap v7712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:51 smithi116 ceph-mon[35411]: pgmap v7712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:45:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:53 smithi114 ceph-mon[30820]: pgmap v7713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:53 smithi116 ceph-mon[35411]: pgmap v7713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:45:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:55 smithi114 ceph-mon[30820]: pgmap v7714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:45:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:55 smithi116 ceph-mon[35411]: pgmap v7714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:45:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:57 smithi114 ceph-mon[30820]: pgmap v7715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:45:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:57 smithi116 ceph-mon[35411]: pgmap v7715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:45:59 smithi114 ceph-mon[30820]: pgmap v7716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:45:59 smithi116 ceph-mon[35411]: pgmap v7716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:01 smithi114 ceph-mon[30820]: pgmap v7717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:01 smithi116 ceph-mon[35411]: pgmap v7717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:03 smithi114 ceph-mon[30820]: pgmap v7718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:03 smithi116 ceph-mon[35411]: pgmap v7718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:05 smithi114 ceph-mon[30820]: pgmap v7719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:46:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:05 smithi116 ceph-mon[35411]: pgmap v7719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:46:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:46:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:46:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:07 smithi114 ceph-mon[30820]: pgmap v7720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:07 smithi116 ceph-mon[35411]: pgmap v7720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:46:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:46:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:46:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:46:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-mon[30820]: pgmap v7721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:10.166 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:46:09.940+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:46:10.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:46:10.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:46:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:46:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:09 smithi116 ceph-mon[35411]: pgmap v7721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:10 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:46:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:46:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:10 smithi114 ceph-mon[30820]: pgmap v7722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:10 smithi114 ceph-mon[30820]: pgmap v7723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:46:11.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:46:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:10 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:46:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:46:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:46:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:46:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:46:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:10 smithi116 ceph-mon[35411]: pgmap v7722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:10 smithi116 ceph-mon[35411]: pgmap v7723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:46:11.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:46:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:12 smithi114 ceph-mon[30820]: pgmap v7724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:46:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:12 smithi116 ceph-mon[35411]: pgmap v7724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:46:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:14 smithi114 ceph-mon[30820]: pgmap v7725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:46:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:14 smithi116 ceph-mon[35411]: pgmap v7725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:46:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:16 smithi114 ceph-mon[30820]: pgmap v7726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:46:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:16 smithi116 ceph-mon[35411]: pgmap v7726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:46:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:18 smithi114 ceph-mon[30820]: pgmap v7727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:46:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:18 smithi116 ceph-mon[35411]: pgmap v7727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:46:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:20 smithi114 ceph-mon[30820]: pgmap v7728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:46:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:20 smithi116 ceph-mon[35411]: pgmap v7728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:46:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:22 smithi114 ceph-mon[30820]: pgmap v7729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:22 smithi116 ceph-mon[35411]: pgmap v7729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:24 smithi114 ceph-mon[30820]: pgmap v7730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:24 smithi116 ceph-mon[35411]: pgmap v7730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:26 smithi114 ceph-mon[30820]: pgmap v7731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:26 smithi116 ceph-mon[35411]: pgmap v7731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:28 smithi114 ceph-mon[30820]: pgmap v7732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:28 smithi116 ceph-mon[35411]: pgmap v7732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:30 smithi114 ceph-mon[30820]: pgmap v7733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:46:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:30 smithi116 ceph-mon[35411]: pgmap v7733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:46:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:32 smithi114 ceph-mon[30820]: pgmap v7734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:32 smithi116 ceph-mon[35411]: pgmap v7734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:34 smithi114 ceph-mon[30820]: pgmap v7735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:34 smithi116 ceph-mon[35411]: pgmap v7735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:36 smithi114 ceph-mon[30820]: pgmap v7736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:36 smithi116 ceph-mon[35411]: pgmap v7736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:38 smithi114 ceph-mon[30820]: pgmap v7737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:38.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:38 smithi116 ceph-mon[35411]: pgmap v7737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:40 smithi114 ceph-mon[30820]: pgmap v7738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:46:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:40 smithi116 ceph-mon[35411]: pgmap v7738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:46:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:42 smithi114 ceph-mon[30820]: pgmap v7739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:42 smithi116 ceph-mon[35411]: pgmap v7739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:44 smithi114 ceph-mon[30820]: pgmap v7740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:44 smithi116 ceph-mon[35411]: pgmap v7740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:46 smithi114 ceph-mon[30820]: pgmap v7741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:46 smithi116 ceph-mon[35411]: pgmap v7741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:48 smithi114 ceph-mon[30820]: pgmap v7742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:48 smithi116 ceph-mon[35411]: pgmap v7742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:50 smithi114 ceph-mon[30820]: pgmap v7743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:46:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:50 smithi116 ceph-mon[35411]: pgmap v7743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:46:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:52 smithi114 ceph-mon[30820]: pgmap v7744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:52 smithi116 ceph-mon[35411]: pgmap v7744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:46:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:54 smithi114 ceph-mon[30820]: pgmap v7745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:54 smithi116 ceph-mon[35411]: pgmap v7745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:56 smithi114 ceph-mon[30820]: pgmap v7746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:56 smithi116 ceph-mon[35411]: pgmap v7746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:46:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:46:58 smithi114 ceph-mon[30820]: pgmap v7747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:46:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:46:58 smithi116 ceph-mon[35411]: pgmap v7747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:00 smithi114 ceph-mon[30820]: pgmap v7748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:00 smithi116 ceph-mon[35411]: pgmap v7748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:02 smithi114 ceph-mon[30820]: pgmap v7749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:02 smithi116 ceph-mon[35411]: pgmap v7749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:04 smithi114 ceph-mon[30820]: pgmap v7750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:04 smithi116 ceph-mon[35411]: pgmap v7750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:06 smithi114 ceph-mon[30820]: pgmap v7751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:47:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:06 smithi116 ceph-mon[35411]: pgmap v7751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:47:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:08 smithi114 ceph-mon[30820]: pgmap v7752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:08 smithi116 ceph-mon[35411]: pgmap v7752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:10 smithi114 ceph-mon[30820]: pgmap v7753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:10 smithi116 ceph-mon[35411]: pgmap v7753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:47:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:47:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:47:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:47:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:47:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:47:13.302 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-mon[30820]: pgmap v7754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:13 smithi116 ceph-mon[35411]: pgmap v7754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:47:13.301+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:47:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:47:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:47:13.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:47:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:47:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:47:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:47:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:47:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:47:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:47:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:47:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:47:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:47:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:47:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:47:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:47:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:47:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:47:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:47:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:47:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:47:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:47:13.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:47:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:47:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:47:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:47:14.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:47:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:47:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:14 smithi114 ceph-mon[30820]: pgmap v7755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:47:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:14 smithi114 ceph-mon[30820]: pgmap v7756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:47:14.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:47:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:47:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:47:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:47:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:47:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:47:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:14 smithi116 ceph-mon[35411]: pgmap v7755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:14 smithi116 ceph-mon[35411]: pgmap v7756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:47:14.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:47:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:16 smithi114 ceph-mon[30820]: pgmap v7757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:47:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:16 smithi116 ceph-mon[35411]: pgmap v7757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:47:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:18 smithi114 ceph-mon[30820]: pgmap v7758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:47:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:18 smithi116 ceph-mon[35411]: pgmap v7758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:47:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:20 smithi114 ceph-mon[30820]: pgmap v7759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:47:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:20 smithi116 ceph-mon[35411]: pgmap v7759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:47:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:22 smithi114 ceph-mon[30820]: pgmap v7760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:47:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:22 smithi116 ceph-mon[35411]: pgmap v7760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:47:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:24 smithi114 ceph-mon[30820]: pgmap v7761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:47:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:24 smithi116 ceph-mon[35411]: pgmap v7761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:47:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:26 smithi114 ceph-mon[30820]: pgmap v7762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:47:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:26 smithi116 ceph-mon[35411]: pgmap v7762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:47:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:28 smithi114 ceph-mon[30820]: pgmap v7763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:47:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:28 smithi116 ceph-mon[35411]: pgmap v7763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:47:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:30 smithi114 ceph-mon[30820]: pgmap v7764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:30 smithi116 ceph-mon[35411]: pgmap v7764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:32 smithi114 ceph-mon[30820]: pgmap v7765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:32 smithi116 ceph-mon[35411]: pgmap v7765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:34 smithi114 ceph-mon[30820]: pgmap v7766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:34 smithi116 ceph-mon[35411]: pgmap v7766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:36 smithi114 ceph-mon[30820]: pgmap v7767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:47:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:36 smithi116 ceph-mon[35411]: pgmap v7767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:47:38.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:38 smithi114 ceph-mon[30820]: pgmap v7768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:47:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:38 smithi116 ceph-mon[35411]: pgmap v7768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:47:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:40 smithi114 ceph-mon[30820]: pgmap v7769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:40 smithi116 ceph-mon[35411]: pgmap v7769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:42.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:42 smithi114 ceph-mon[30820]: pgmap v7770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:42 smithi116 ceph-mon[35411]: pgmap v7770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:44 smithi116 ceph-mon[35411]: pgmap v7771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:44 smithi114 ceph-mon[30820]: pgmap v7771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:46 smithi116 ceph-mon[35411]: pgmap v7772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:47:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:46 smithi114 ceph-mon[30820]: pgmap v7772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:47:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:48 smithi116 ceph-mon[35411]: pgmap v7773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:47:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:48 smithi114 ceph-mon[30820]: pgmap v7773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:47:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:50 smithi114 ceph-mon[30820]: pgmap v7774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:50 smithi116 ceph-mon[35411]: pgmap v7774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:47:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:52 smithi114 ceph-mon[30820]: pgmap v7775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:52 smithi116 ceph-mon[35411]: pgmap v7775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:47:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:54 smithi114 ceph-mon[30820]: pgmap v7776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:54 smithi116 ceph-mon[35411]: pgmap v7776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:47:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:56 smithi114 ceph-mon[30820]: pgmap v7777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:47:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:56 smithi116 ceph-mon[35411]: pgmap v7777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:47:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:47:58 smithi114 ceph-mon[30820]: pgmap v7778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:47:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:47:58 smithi116 ceph-mon[35411]: pgmap v7778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:48:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:00 smithi114 ceph-mon[30820]: pgmap v7779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:00 smithi116 ceph-mon[35411]: pgmap v7779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:02 smithi114 ceph-mon[30820]: pgmap v7780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:02 smithi116 ceph-mon[35411]: pgmap v7780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:04 smithi114 ceph-mon[30820]: pgmap v7781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:04 smithi116 ceph-mon[35411]: pgmap v7781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:06 smithi114 ceph-mon[30820]: pgmap v7782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:48:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:06 smithi116 ceph-mon[35411]: pgmap v7782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:48:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:08 smithi114 ceph-mon[30820]: pgmap v7783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:48:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:08 smithi116 ceph-mon[35411]: pgmap v7783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:48:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:10 smithi114 ceph-mon[30820]: pgmap v7784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:10 smithi116 ceph-mon[35411]: pgmap v7784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:12 smithi114 ceph-mon[30820]: pgmap v7785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:12 smithi116 ceph-mon[35411]: pgmap v7785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:13.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:48:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:48:14.817 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:14 smithi114 ceph-mon[30820]: pgmap v7786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:14 smithi116 ceph-mon[35411]: pgmap v7786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-mon[30820]: pgmap v7787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:48:16.660+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:48:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:48:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:48:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:48:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:16 smithi116 ceph-mon[35411]: pgmap v7787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:48:17.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:48:17.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:48:17.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:48:17.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:48:17.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:48:17.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:48:17.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:17 smithi114 ceph-mon[30820]: pgmap v7788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:48:17.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:17 smithi114 ceph-mon[30820]: pgmap v7789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:48:17.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:48:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:48:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:48:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:48:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:48:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:48:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:17 smithi116 ceph-mon[35411]: pgmap v7788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:17 smithi116 ceph-mon[35411]: pgmap v7789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:48:17.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:48:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:19 smithi114 ceph-mon[30820]: pgmap v7790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:48:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:19 smithi116 ceph-mon[35411]: pgmap v7790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:48:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:21 smithi114 ceph-mon[30820]: pgmap v7791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:48:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:21 smithi116 ceph-mon[35411]: pgmap v7791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:48:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:23 smithi114 ceph-mon[30820]: pgmap v7792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:48:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:23 smithi116 ceph-mon[35411]: pgmap v7792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:48:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:25 smithi114 ceph-mon[30820]: pgmap v7793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:48:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:25 smithi116 ceph-mon[35411]: pgmap v7793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:48:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:27 smithi114 ceph-mon[30820]: pgmap v7794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:48:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:27 smithi116 ceph-mon[35411]: pgmap v7794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:48:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:29 smithi114 ceph-mon[30820]: pgmap v7795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:29 smithi116 ceph-mon[35411]: pgmap v7795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:31 smithi114 ceph-mon[30820]: pgmap v7796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:48:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:31 smithi116 ceph-mon[35411]: pgmap v7796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:48:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:33 smithi114 ceph-mon[30820]: pgmap v7797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:33 smithi116 ceph-mon[35411]: pgmap v7797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:35 smithi114 ceph-mon[30820]: pgmap v7798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:35 smithi116 ceph-mon[35411]: pgmap v7798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:37 smithi114 ceph-mon[30820]: pgmap v7799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:37 smithi116 ceph-mon[35411]: pgmap v7799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:39 smithi114 ceph-mon[30820]: pgmap v7800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:39 smithi116 ceph-mon[35411]: pgmap v7800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:41 smithi114 ceph-mon[30820]: pgmap v7801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:48:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:41 smithi116 ceph-mon[35411]: pgmap v7801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:48:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:43 smithi114 ceph-mon[30820]: pgmap v7802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:43 smithi116 ceph-mon[35411]: pgmap v7802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:45 smithi114 ceph-mon[30820]: pgmap v7803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:45 smithi116 ceph-mon[35411]: pgmap v7803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:47 smithi114 ceph-mon[30820]: pgmap v7804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:47 smithi116 ceph-mon[35411]: pgmap v7804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:49 smithi114 ceph-mon[30820]: pgmap v7805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:49 smithi116 ceph-mon[35411]: pgmap v7805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:51 smithi114 ceph-mon[30820]: pgmap v7806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:48:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:51 smithi116 ceph-mon[35411]: pgmap v7806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:48:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:53 smithi114 ceph-mon[30820]: pgmap v7807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:53 smithi116 ceph-mon[35411]: pgmap v7807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:48:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:55 smithi114 ceph-mon[30820]: pgmap v7808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:55 smithi116 ceph-mon[35411]: pgmap v7808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:48:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:57 smithi114 ceph-mon[30820]: pgmap v7809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:48:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:57 smithi116 ceph-mon[35411]: pgmap v7809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:48:59 smithi114 ceph-mon[30820]: pgmap v7810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:48:59 smithi116 ceph-mon[35411]: pgmap v7810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:01 smithi114 ceph-mon[30820]: pgmap v7811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:01 smithi116 ceph-mon[35411]: pgmap v7811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:03 smithi114 ceph-mon[30820]: pgmap v7812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:03 smithi116 ceph-mon[35411]: pgmap v7812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:05 smithi114 ceph-mon[30820]: pgmap v7813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:49:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:05 smithi116 ceph-mon[35411]: pgmap v7813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:49:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:07 smithi114 ceph-mon[30820]: pgmap v7814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:07 smithi116 ceph-mon[35411]: pgmap v7814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:09 smithi114 ceph-mon[30820]: pgmap v7815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:09 smithi116 ceph-mon[35411]: pgmap v7815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:11 smithi114 ceph-mon[30820]: pgmap v7816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:11 smithi116 ceph-mon[35411]: pgmap v7816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:13 smithi114 ceph-mon[30820]: pgmap v7817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:13 smithi116 ceph-mon[35411]: pgmap v7817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:15 smithi114 ceph-mon[30820]: pgmap v7818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:49:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:15 smithi116 ceph-mon[35411]: pgmap v7818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:49:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:17 smithi114 ceph-mon[30820]: pgmap v7819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:49:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:17 smithi116 ceph-mon[35411]: pgmap v7819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:49:20.022 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:19 smithi114 ceph-mon[30820]: pgmap v7820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:19 smithi116 ceph-mon[35411]: pgmap v7820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:20.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:49:20.020+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:49:20.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:49:20.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:49:20.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:49:20.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:49:20.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:49:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:49:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:49:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:49:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:49:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:49:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:49:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:49:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:49:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:49:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:49:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:49:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:21 smithi114 ceph-mon[30820]: pgmap v7821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:49:22.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:21 smithi114 ceph-mon[30820]: pgmap v7822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:49:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:49:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:49:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:21 smithi116 ceph-mon[35411]: pgmap v7821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T10:49:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:21 smithi116 ceph-mon[35411]: pgmap v7822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:49:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:23 smithi114 ceph-mon[30820]: pgmap v7823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:49:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:23 smithi116 ceph-mon[35411]: pgmap v7823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:49:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:25 smithi114 ceph-mon[30820]: pgmap v7824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:49:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:25 smithi116 ceph-mon[35411]: pgmap v7824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:49:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:27 smithi114 ceph-mon[30820]: pgmap v7825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:49:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:27 smithi116 ceph-mon[35411]: pgmap v7825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:49:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:29 smithi114 ceph-mon[30820]: pgmap v7826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:49:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:29 smithi116 ceph-mon[35411]: pgmap v7826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:49:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:31 smithi114 ceph-mon[30820]: pgmap v7827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:49:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:31 smithi116 ceph-mon[35411]: pgmap v7827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:49:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:33 smithi114 ceph-mon[30820]: pgmap v7828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:33 smithi116 ceph-mon[35411]: pgmap v7828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:35 smithi114 ceph-mon[30820]: pgmap v7829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:35 smithi116 ceph-mon[35411]: pgmap v7829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:37 smithi114 ceph-mon[30820]: pgmap v7830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:37 smithi116 ceph-mon[35411]: pgmap v7830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:39 smithi114 ceph-mon[30820]: pgmap v7831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:39 smithi116 ceph-mon[35411]: pgmap v7831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:41 smithi114 ceph-mon[30820]: pgmap v7832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:49:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:41 smithi116 ceph-mon[35411]: pgmap v7832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:49:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:43 smithi114 ceph-mon[30820]: pgmap v7833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:43 smithi116 ceph-mon[35411]: pgmap v7833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:45 smithi114 ceph-mon[30820]: pgmap v7834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:45 smithi116 ceph-mon[35411]: pgmap v7834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:49:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:47 smithi114 ceph-mon[30820]: pgmap v7835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:47 smithi116 ceph-mon[35411]: pgmap v7835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:49 smithi114 ceph-mon[30820]: pgmap v7836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:49:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:49 smithi116 ceph-mon[35411]: pgmap v7836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:49:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:51 smithi114 ceph-mon[30820]: pgmap v7837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:49:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:51 smithi116 ceph-mon[35411]: pgmap v7837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:49:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:53 smithi114 ceph-mon[30820]: pgmap v7838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:53 smithi116 ceph-mon[35411]: pgmap v7838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:49:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:55 smithi114 ceph-mon[30820]: pgmap v7839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:49:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:55 smithi116 ceph-mon[35411]: pgmap v7839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:49:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:57 smithi114 ceph-mon[30820]: pgmap v7840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:49:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:57 smithi116 ceph-mon[35411]: pgmap v7840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:49:59 smithi114 ceph-mon[30820]: pgmap v7841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:49:59 smithi116 ceph-mon[35411]: pgmap v7841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:50:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T10:50:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:01 smithi114 ceph-mon[30820]: pgmap v7842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:50:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:01 smithi116 ceph-mon[35411]: pgmap v7842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:50:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:03 smithi114 ceph-mon[30820]: pgmap v7843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:03 smithi116 ceph-mon[35411]: pgmap v7843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:05 smithi114 ceph-mon[30820]: pgmap v7844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:05 smithi116 ceph-mon[35411]: pgmap v7844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:07 smithi114 ceph-mon[30820]: pgmap v7845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:07 smithi116 ceph-mon[35411]: pgmap v7845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:09 smithi114 ceph-mon[30820]: pgmap v7846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:09 smithi116 ceph-mon[35411]: pgmap v7846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:11 smithi114 ceph-mon[30820]: pgmap v7847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:50:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:11 smithi116 ceph-mon[35411]: pgmap v7847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:50:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:13 smithi114 ceph-mon[30820]: pgmap v7848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:13 smithi116 ceph-mon[35411]: pgmap v7848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:15 smithi114 ceph-mon[30820]: pgmap v7849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:15 smithi116 ceph-mon[35411]: pgmap v7849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:17 smithi114 ceph-mon[30820]: pgmap v7850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:17 smithi116 ceph-mon[35411]: pgmap v7850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:19 smithi114 ceph-mon[30820]: pgmap v7851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:50:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:19 smithi116 ceph-mon[35411]: pgmap v7851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:50:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:50:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:50:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: pgmap v7852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:50:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:50:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:50:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:50:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:21 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:50:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: pgmap v7852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T10:50:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T10:50:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T10:50:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T10:50:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T10:50:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:21 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T10:50:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:50:23.380+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:50:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:50:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:50:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:50:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:50:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:50:23.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:50:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:50:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:50:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:50:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:50:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-mon[30820]: pgmap v7853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:50:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:50:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:50:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:23 smithi116 ceph-mon[35411]: pgmap v7853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:50:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:50:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:50:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:50:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:50:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:24 smithi114 ceph-mon[30820]: pgmap v7854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:50:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:24 smithi114 ceph-mon[30820]: pgmap v7855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:50:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:50:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:50:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:50:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:50:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:50:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:50:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:50:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:50:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:50:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:24 smithi116 ceph-mon[35411]: pgmap v7854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:50:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:24 smithi116 ceph-mon[35411]: pgmap v7855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:50:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:26 smithi116 ceph-mon[35411]: pgmap v7856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:50:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:26 smithi114 ceph-mon[30820]: pgmap v7856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:50:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:28 smithi116 ceph-mon[35411]: pgmap v7857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:50:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:28 smithi114 ceph-mon[30820]: pgmap v7857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:50:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:30 smithi116 ceph-mon[35411]: pgmap v7858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:50:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:30 smithi114 ceph-mon[30820]: pgmap v7858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:50:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:32 smithi114 ceph-mon[30820]: pgmap v7859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:50:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:32 smithi116 ceph-mon[35411]: pgmap v7859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:50:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:34 smithi114 ceph-mon[30820]: pgmap v7860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:50:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:34 smithi116 ceph-mon[35411]: pgmap v7860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:50:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:36 smithi114 ceph-mon[30820]: pgmap v7861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:36 smithi116 ceph-mon[35411]: pgmap v7861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:38 smithi114 ceph-mon[30820]: pgmap v7862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:38 smithi116 ceph-mon[35411]: pgmap v7862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:41.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:40 smithi114 ceph-mon[30820]: pgmap v7863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:50:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:40 smithi116 ceph-mon[35411]: pgmap v7863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:50:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:42 smithi114 ceph-mon[30820]: pgmap v7864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:42 smithi116 ceph-mon[35411]: pgmap v7864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:45.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:44 smithi114 ceph-mon[30820]: pgmap v7865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:44 smithi116 ceph-mon[35411]: pgmap v7865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:47.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:46 smithi116 ceph-mon[35411]: pgmap v7866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:46 smithi114 ceph-mon[30820]: pgmap v7866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:48 smithi114 ceph-mon[30820]: pgmap v7867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:48 smithi116 ceph-mon[35411]: pgmap v7867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:50 smithi114 ceph-mon[30820]: pgmap v7868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:50:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:50 smithi116 ceph-mon[35411]: pgmap v7868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:50:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:52 smithi114 ceph-mon[30820]: pgmap v7869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:52 smithi116 ceph-mon[35411]: pgmap v7869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:50:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:54 smithi114 ceph-mon[30820]: pgmap v7870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:54 smithi116 ceph-mon[35411]: pgmap v7870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:56 smithi114 ceph-mon[30820]: pgmap v7871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:56 smithi116 ceph-mon[35411]: pgmap v7871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:50:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:50:58 smithi114 ceph-mon[30820]: pgmap v7872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:50:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:50:58 smithi116 ceph-mon[35411]: pgmap v7872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:00 smithi114 ceph-mon[30820]: pgmap v7873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:00 smithi116 ceph-mon[35411]: pgmap v7873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:02 smithi114 ceph-mon[30820]: pgmap v7874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:02 smithi116 ceph-mon[35411]: pgmap v7874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:05.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:04 smithi114 ceph-mon[30820]: pgmap v7875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:04 smithi116 ceph-mon[35411]: pgmap v7875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:06 smithi114 ceph-mon[30820]: pgmap v7876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:06 smithi116 ceph-mon[35411]: pgmap v7876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:08 smithi114 ceph-mon[30820]: pgmap v7877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:08 smithi116 ceph-mon[35411]: pgmap v7877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:10 smithi114 ceph-mon[30820]: pgmap v7878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:10 smithi116 ceph-mon[35411]: pgmap v7878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:13 smithi114 ceph-mon[30820]: pgmap v7879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:13 smithi116 ceph-mon[35411]: pgmap v7879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:15 smithi114 ceph-mon[30820]: pgmap v7880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:15 smithi116 ceph-mon[35411]: pgmap v7880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:17 smithi114 ceph-mon[30820]: pgmap v7881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:17 smithi116 ceph-mon[35411]: pgmap v7881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:18 smithi114 ceph-mon[30820]: pgmap v7882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:18 smithi116 ceph-mon[35411]: pgmap v7882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:20 smithi114 ceph-mon[30820]: pgmap v7883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:20 smithi116 ceph-mon[35411]: pgmap v7883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:22 smithi114 ceph-mon[30820]: pgmap v7884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:22 smithi116 ceph-mon[35411]: pgmap v7884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:24 smithi114 ceph-mon[30820]: pgmap v7885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:51:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:24 smithi116 ceph-mon[35411]: pgmap v7885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:51:26.742 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-mon[30820]: pgmap v7886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:26 smithi116 ceph-mon[35411]: pgmap v7886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:27.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:51:26.740+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:51:27.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:51:27.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:51:27.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:51:27.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:51:27.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:51:27.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:51:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:51:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:51:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:51:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:51:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:51:27.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:51:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:51:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:51:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:51:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:51:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:51:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:51:27.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:27 smithi114 ceph-mon[30820]: pgmap v7887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:27 smithi114 ceph-mon[30820]: pgmap v7888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:51:27.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:51:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:51:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:51:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:51:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:51:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:51:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:51:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:51:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:51:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:51:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:51:27.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:51:27.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:51:27.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:27 smithi116 ceph-mon[35411]: pgmap v7887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:51:27.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:27 smithi116 ceph-mon[35411]: pgmap v7888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:51:27.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:51:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:29 smithi114 ceph-mon[30820]: pgmap v7889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:51:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:29 smithi116 ceph-mon[35411]: pgmap v7889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:51:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:31 smithi114 ceph-mon[30820]: pgmap v7890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:51:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:31 smithi116 ceph-mon[35411]: pgmap v7890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:51:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:33 smithi114 ceph-mon[30820]: pgmap v7891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:51:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:33 smithi116 ceph-mon[35411]: pgmap v7891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:51:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:35 smithi114 ceph-mon[30820]: pgmap v7892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:51:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:35 smithi116 ceph-mon[35411]: pgmap v7892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:51:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:37 smithi114 ceph-mon[30820]: pgmap v7893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:51:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:37 smithi116 ceph-mon[35411]: pgmap v7893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:51:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:39 smithi114 ceph-mon[30820]: pgmap v7894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:39 smithi116 ceph-mon[35411]: pgmap v7894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:41 smithi114 ceph-mon[30820]: pgmap v7895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:41 smithi116 ceph-mon[35411]: pgmap v7895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:43 smithi114 ceph-mon[30820]: pgmap v7896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:43 smithi116 ceph-mon[35411]: pgmap v7896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:45 smithi114 ceph-mon[30820]: pgmap v7897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:45 smithi116 ceph-mon[35411]: pgmap v7897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:47 smithi114 ceph-mon[30820]: pgmap v7898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:47 smithi116 ceph-mon[35411]: pgmap v7898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:49 smithi114 ceph-mon[30820]: pgmap v7899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:49 smithi116 ceph-mon[35411]: pgmap v7899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:51 smithi114 ceph-mon[30820]: pgmap v7900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:51 smithi116 ceph-mon[35411]: pgmap v7900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:51:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:53 smithi114 ceph-mon[30820]: pgmap v7901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:53 smithi116 ceph-mon[35411]: pgmap v7901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:51:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:55 smithi114 ceph-mon[30820]: pgmap v7902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:55 smithi116 ceph-mon[35411]: pgmap v7902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:51:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:57 smithi114 ceph-mon[30820]: pgmap v7903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:51:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:57 smithi116 ceph-mon[35411]: pgmap v7903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:51:59 smithi114 ceph-mon[30820]: pgmap v7904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:51:59 smithi116 ceph-mon[35411]: pgmap v7904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:01 smithi114 ceph-mon[30820]: pgmap v7905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:01 smithi116 ceph-mon[35411]: pgmap v7905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:03 smithi114 ceph-mon[30820]: pgmap v7906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:03 smithi116 ceph-mon[35411]: pgmap v7906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:05 smithi114 ceph-mon[30820]: pgmap v7907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:52:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:05 smithi116 ceph-mon[35411]: pgmap v7907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:52:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:07 smithi114 ceph-mon[30820]: pgmap v7908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:07 smithi116 ceph-mon[35411]: pgmap v7908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:09 smithi114 ceph-mon[30820]: pgmap v7909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:09 smithi116 ceph-mon[35411]: pgmap v7909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:11 smithi114 ceph-mon[30820]: pgmap v7910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:11 smithi116 ceph-mon[35411]: pgmap v7910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:13 smithi114 ceph-mon[30820]: pgmap v7911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:13 smithi116 ceph-mon[35411]: pgmap v7911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:15 smithi114 ceph-mon[30820]: pgmap v7912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:52:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:15 smithi116 ceph-mon[35411]: pgmap v7912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:52:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:17 smithi114 ceph-mon[30820]: pgmap v7913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:17 smithi116 ceph-mon[35411]: pgmap v7913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:19 smithi114 ceph-mon[30820]: pgmap v7914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:19 smithi116 ceph-mon[35411]: pgmap v7914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:21 smithi114 ceph-mon[30820]: pgmap v7915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:21 smithi116 ceph-mon[35411]: pgmap v7915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:23 smithi114 ceph-mon[30820]: pgmap v7916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:23 smithi116 ceph-mon[35411]: pgmap v7916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:25 smithi114 ceph-mon[30820]: pgmap v7917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:52:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:25 smithi116 ceph-mon[35411]: pgmap v7917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:52:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:52:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:52:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:27 smithi116 ceph-mon[35411]: pgmap v7918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:28.196 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:27 smithi114 ceph-mon[30820]: pgmap v7918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:29 smithi114 ceph-mon[30820]: pgmap v7919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:52:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:52:30.100+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:52:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:52:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:52:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:52:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:52:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:52:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:52:30.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:52:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:52:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:52:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:52:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:52:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:52:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:52:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:52:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:29 smithi116 ceph-mon[35411]: pgmap v7919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T10:52:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:52:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:52:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:52:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:52:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:52:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:52:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:52:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:52:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:52:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:52:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:52:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:52:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:52:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:52:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:52:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:52:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:52:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:52:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:52:31.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:52:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:52:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:52:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:52:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:31.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:31 smithi114 ceph-mon[30820]: pgmap v7920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:52:31.928 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:52:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:52:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:52:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:52:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:31 smithi116 ceph-mon[35411]: pgmap v7920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:52:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:52:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:32 smithi114 ceph-mon[30820]: pgmap v7921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:52:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:32 smithi114 ceph-mon[30820]: pgmap v7922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:52:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:52:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:52:33.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:32 smithi116 ceph-mon[35411]: pgmap v7921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:52:33.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:32 smithi116 ceph-mon[35411]: pgmap v7922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T10:52:33.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:52:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:34 smithi114 ceph-mon[30820]: pgmap v7923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T10:52:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:34 smithi116 ceph-mon[35411]: pgmap v7923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T10:52:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:36 smithi114 ceph-mon[30820]: pgmap v7924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T10:52:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:36 smithi116 ceph-mon[35411]: pgmap v7924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T10:52:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:38 smithi114 ceph-mon[30820]: pgmap v7925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T10:52:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:38 smithi116 ceph-mon[35411]: pgmap v7925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T10:52:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:40 smithi114 ceph-mon[30820]: pgmap v7926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:52:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:40 smithi116 ceph-mon[35411]: pgmap v7926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:52:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:42 smithi114 ceph-mon[30820]: pgmap v7927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:52:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:42 smithi116 ceph-mon[35411]: pgmap v7927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:52:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:44 smithi114 ceph-mon[30820]: pgmap v7928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:44 smithi116 ceph-mon[35411]: pgmap v7928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:46 smithi114 ceph-mon[30820]: pgmap v7929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:46 smithi116 ceph-mon[35411]: pgmap v7929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:48 smithi114 ceph-mon[30820]: pgmap v7930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:48 smithi116 ceph-mon[35411]: pgmap v7930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:50 smithi114 ceph-mon[30820]: pgmap v7931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:52:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:50 smithi116 ceph-mon[35411]: pgmap v7931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:52:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:52 smithi114 ceph-mon[30820]: pgmap v7932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:52 smithi116 ceph-mon[35411]: pgmap v7932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:52:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:54 smithi114 ceph-mon[30820]: pgmap v7933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:54 smithi116 ceph-mon[35411]: pgmap v7933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:56 smithi114 ceph-mon[30820]: pgmap v7934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:56 smithi116 ceph-mon[35411]: pgmap v7934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:52:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:52:58 smithi114 ceph-mon[30820]: pgmap v7935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:52:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:52:58 smithi116 ceph-mon[35411]: pgmap v7935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:00 smithi114 ceph-mon[30820]: pgmap v7936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:00 smithi116 ceph-mon[35411]: pgmap v7936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:02 smithi114 ceph-mon[30820]: pgmap v7937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:02 smithi116 ceph-mon[35411]: pgmap v7937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:04 smithi114 ceph-mon[30820]: pgmap v7938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:04 smithi116 ceph-mon[35411]: pgmap v7938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:06 smithi114 ceph-mon[30820]: pgmap v7939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:53:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:06 smithi116 ceph-mon[35411]: pgmap v7939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:53:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:08 smithi114 ceph-mon[30820]: pgmap v7940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:08 smithi116 ceph-mon[35411]: pgmap v7940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:10 smithi114 ceph-mon[30820]: pgmap v7941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:10 smithi116 ceph-mon[35411]: pgmap v7941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:12 smithi114 ceph-mon[30820]: pgmap v7942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:12 smithi116 ceph-mon[35411]: pgmap v7942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:14 smithi114 ceph-mon[30820]: pgmap v7943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:14 smithi116 ceph-mon[35411]: pgmap v7943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:16 smithi114 ceph-mon[30820]: pgmap v7944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:53:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:16 smithi116 ceph-mon[35411]: pgmap v7944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:53:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:18 smithi114 ceph-mon[30820]: pgmap v7945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:18 smithi116 ceph-mon[35411]: pgmap v7945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:20 smithi114 ceph-mon[30820]: pgmap v7946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:20 smithi116 ceph-mon[35411]: pgmap v7946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:22 smithi114 ceph-mon[30820]: pgmap v7947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:22 smithi116 ceph-mon[35411]: pgmap v7947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:24 smithi114 ceph-mon[30820]: pgmap v7948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:24 smithi116 ceph-mon[35411]: pgmap v7948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:26 smithi114 ceph-mon[30820]: pgmap v7949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:53:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:26 smithi116 ceph-mon[35411]: pgmap v7949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:53:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:28 smithi114 ceph-mon[30820]: pgmap v7950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:28 smithi116 ceph-mon[35411]: pgmap v7950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:30 smithi114 ceph-mon[30820]: pgmap v7951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:30 smithi116 ceph-mon[35411]: pgmap v7951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:32 smithi114 ceph-mon[30820]: pgmap v7952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:32 smithi116 ceph-mon[35411]: pgmap v7952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:33.252 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:53:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:53:34.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:34 smithi114 ceph-mon[30820]: pgmap v7953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:34 smithi116 ceph-mon[35411]: pgmap v7953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:53:35.316+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:53:35.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:53:35.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:53:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:53:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:53:35.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:53:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:53:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:53:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:53:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:36 smithi114 ceph-mon[30820]: pgmap v7954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T10:53:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:36 smithi114 ceph-mon[30820]: pgmap v7955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:53:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:53:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:36 smithi116 ceph-mon[35411]: pgmap v7954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T10:53:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:36 smithi116 ceph-mon[35411]: pgmap v7955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:53:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:38 smithi114 ceph-mon[30820]: pgmap v7956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:53:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:38 smithi116 ceph-mon[35411]: pgmap v7956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:53:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:40 smithi114 ceph-mon[30820]: pgmap v7957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:53:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:40 smithi116 ceph-mon[35411]: pgmap v7957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:53:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:42 smithi114 ceph-mon[30820]: pgmap v7958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:53:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:42 smithi116 ceph-mon[35411]: pgmap v7958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:53:44.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:44 smithi114 ceph-mon[30820]: pgmap v7959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:53:44.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:44 smithi116 ceph-mon[35411]: pgmap v7959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:53:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:46 smithi114 ceph-mon[30820]: pgmap v7960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:53:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:46 smithi116 ceph-mon[35411]: pgmap v7960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:53:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:48 smithi114 ceph-mon[30820]: pgmap v7961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:53:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:48 smithi116 ceph-mon[35411]: pgmap v7961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:53:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:50 smithi114 ceph-mon[30820]: pgmap v7962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:50 smithi116 ceph-mon[35411]: pgmap v7962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:53:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:52 smithi114 ceph-mon[30820]: pgmap v7963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:52 smithi116 ceph-mon[35411]: pgmap v7963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:53:54.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:54 smithi114 ceph-mon[30820]: pgmap v7964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:54 smithi116 ceph-mon[35411]: pgmap v7964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:53:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:56 smithi114 ceph-mon[30820]: pgmap v7965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:53:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:56 smithi116 ceph-mon[35411]: pgmap v7965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:53:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:53:58 smithi114 ceph-mon[30820]: pgmap v7966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:53:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:53:58 smithi116 ceph-mon[35411]: pgmap v7966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:00 smithi114 ceph-mon[30820]: pgmap v7967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:00 smithi116 ceph-mon[35411]: pgmap v7967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:02 smithi114 ceph-mon[30820]: pgmap v7968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:02 smithi116 ceph-mon[35411]: pgmap v7968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:04 smithi114 ceph-mon[30820]: pgmap v7969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:04 smithi116 ceph-mon[35411]: pgmap v7969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:06 smithi114 ceph-mon[30820]: pgmap v7970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:54:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:06 smithi116 ceph-mon[35411]: pgmap v7970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:54:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:08 smithi114 ceph-mon[30820]: pgmap v7971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:08.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:08 smithi116 ceph-mon[35411]: pgmap v7971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:10 smithi114 ceph-mon[30820]: pgmap v7972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:10 smithi116 ceph-mon[35411]: pgmap v7972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:12 smithi114 ceph-mon[30820]: pgmap v7973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:12 smithi116 ceph-mon[35411]: pgmap v7973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:14 smithi116 ceph-mon[35411]: pgmap v7974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:14 smithi114 ceph-mon[30820]: pgmap v7974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:16 smithi116 ceph-mon[35411]: pgmap v7975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:54:16.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:16 smithi114 ceph-mon[30820]: pgmap v7975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:54:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:18 smithi116 ceph-mon[35411]: pgmap v7976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:18 smithi114 ceph-mon[30820]: pgmap v7976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:20 smithi116 ceph-mon[35411]: pgmap v7977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:20 smithi114 ceph-mon[30820]: pgmap v7977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:23.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:22 smithi114 ceph-mon[30820]: pgmap v7978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:22 smithi116 ceph-mon[35411]: pgmap v7978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:24 smithi114 ceph-mon[30820]: pgmap v7979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:24 smithi116 ceph-mon[35411]: pgmap v7979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:26 smithi114 ceph-mon[30820]: pgmap v7980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:54:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:26 smithi116 ceph-mon[35411]: pgmap v7980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:54:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:28 smithi114 ceph-mon[30820]: pgmap v7981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:28 smithi116 ceph-mon[35411]: pgmap v7981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:30 smithi114 ceph-mon[30820]: pgmap v7982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:30 smithi116 ceph-mon[35411]: pgmap v7982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:32 smithi114 ceph-mon[30820]: pgmap v7983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:32 smithi116 ceph-mon[35411]: pgmap v7983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:34 smithi114 ceph-mon[30820]: pgmap v7984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:35.387 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:34 smithi116 ceph-mon[35411]: pgmap v7984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:54:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:54:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:36 smithi114 ceph-mon[30820]: pgmap v7985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:54:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:36 smithi116 ceph-mon[35411]: pgmap v7985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:54:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-mon[30820]: pgmap v7986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:54:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:54:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:38.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:54:38.613+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:54:38.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:54:38.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:54:38.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:54:38.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:54:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:54:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:54:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:54:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:54:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:54:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:54:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:54:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:38 smithi116 ceph-mon[35411]: pgmap v7986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:54:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:54:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:54:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:54:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:54:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:54:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:54:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:54:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:39 smithi114 ceph-mon[30820]: pgmap v7987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:54:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:39 smithi114 ceph-mon[30820]: pgmap v7988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:54:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:54:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:54:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:54:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:39 smithi116 ceph-mon[35411]: pgmap v7987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:54:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:39 smithi116 ceph-mon[35411]: pgmap v7988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:54:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:41 smithi114 ceph-mon[30820]: pgmap v7989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:54:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:41 smithi116 ceph-mon[35411]: pgmap v7989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:54:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:43 smithi114 ceph-mon[30820]: pgmap v7990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:54:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:43 smithi116 ceph-mon[35411]: pgmap v7990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:54:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:45 smithi114 ceph-mon[30820]: pgmap v7991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T10:54:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:45 smithi116 ceph-mon[35411]: pgmap v7991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T10:54:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:47 smithi114 ceph-mon[30820]: pgmap v7992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:54:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:47 smithi116 ceph-mon[35411]: pgmap v7992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:54:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:49 smithi114 ceph-mon[30820]: pgmap v7993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:54:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:49 smithi116 ceph-mon[35411]: pgmap v7993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:54:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:51 smithi114 ceph-mon[30820]: pgmap v7994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:54:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:51 smithi116 ceph-mon[35411]: pgmap v7994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:54:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:53 smithi114 ceph-mon[30820]: pgmap v7995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:54.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:53 smithi116 ceph-mon[35411]: pgmap v7995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:54:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:55 smithi114 ceph-mon[30820]: pgmap v7996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:55 smithi116 ceph-mon[35411]: pgmap v7996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:54:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:57 smithi114 ceph-mon[30820]: pgmap v7997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:54:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:57 smithi116 ceph-mon[35411]: pgmap v7997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:54:59 smithi114 ceph-mon[30820]: pgmap v7998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:54:59 smithi116 ceph-mon[35411]: pgmap v7998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:01 smithi114 ceph-mon[30820]: pgmap v7999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:01 smithi116 ceph-mon[35411]: pgmap v7999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:03 smithi114 ceph-mon[30820]: pgmap v8000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:03 smithi116 ceph-mon[35411]: pgmap v8000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:05 smithi114 ceph-mon[30820]: pgmap v8001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:55:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:05 smithi116 ceph-mon[35411]: pgmap v8001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:55:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:07 smithi114 ceph-mon[30820]: pgmap v8002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:07 smithi116 ceph-mon[35411]: pgmap v8002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:09 smithi114 ceph-mon[30820]: pgmap v8003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:09 smithi116 ceph-mon[35411]: pgmap v8003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:11 smithi114 ceph-mon[30820]: pgmap v8004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:11 smithi116 ceph-mon[35411]: pgmap v8004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:14.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:13 smithi114 ceph-mon[30820]: pgmap v8005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:13 smithi116 ceph-mon[35411]: pgmap v8005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:15 smithi114 ceph-mon[30820]: pgmap v8006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:55:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:15 smithi116 ceph-mon[35411]: pgmap v8006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:55:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:17 smithi114 ceph-mon[30820]: pgmap v8007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:17 smithi116 ceph-mon[35411]: pgmap v8007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:19 smithi114 ceph-mon[30820]: pgmap v8008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:19 smithi116 ceph-mon[35411]: pgmap v8008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:21 smithi114 ceph-mon[30820]: pgmap v8009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:21 smithi116 ceph-mon[35411]: pgmap v8009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:23 smithi114 ceph-mon[30820]: pgmap v8010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:23 smithi116 ceph-mon[35411]: pgmap v8010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:25 smithi114 ceph-mon[30820]: pgmap v8011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:55:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:25 smithi116 ceph-mon[35411]: pgmap v8011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:55:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:27 smithi114 ceph-mon[30820]: pgmap v8012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:27 smithi116 ceph-mon[35411]: pgmap v8012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:29 smithi114 ceph-mon[30820]: pgmap v8013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:29 smithi116 ceph-mon[35411]: pgmap v8013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:31 smithi114 ceph-mon[30820]: pgmap v8014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:31 smithi116 ceph-mon[35411]: pgmap v8014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:33 smithi114 ceph-mon[30820]: pgmap v8015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:33 smithi116 ceph-mon[35411]: pgmap v8015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:35 smithi114 ceph-mon[30820]: pgmap v8016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:55:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:35 smithi116 ceph-mon[35411]: pgmap v8016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:55:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:37 smithi114 ceph-mon[30820]: pgmap v8017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:37 smithi116 ceph-mon[35411]: pgmap v8017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:55:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:55:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:55:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:39 smithi114 ceph-mon[30820]: pgmap v8018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:39 smithi116 ceph-mon[35411]: pgmap v8018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-mon[30820]: pgmap v8019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:55:41.973+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:55:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:55:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:55:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:55:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:55:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:55:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:55:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:55:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:55:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:55:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:55:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:55:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:55:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:55:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:41 smithi116 ceph-mon[35411]: pgmap v8019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:42 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:55:43.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:55:43.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:42 smithi114 ceph-mon[30820]: pgmap v8020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:55:43.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:42 smithi114 ceph-mon[30820]: pgmap v8021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:55:43.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:55:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:42 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:42 smithi116 ceph-mon[35411]: pgmap v8020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:42 smithi116 ceph-mon[35411]: pgmap v8021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:55:43.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:55:45.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:45 smithi114 ceph-mon[30820]: pgmap v8022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:55:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:45 smithi116 ceph-mon[35411]: pgmap v8022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:55:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:47 smithi114 ceph-mon[30820]: pgmap v8023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:55:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:47 smithi116 ceph-mon[35411]: pgmap v8023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:55:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:49 smithi114 ceph-mon[30820]: pgmap v8024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:55:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:49 smithi116 ceph-mon[35411]: pgmap v8024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:55:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:51 smithi114 ceph-mon[30820]: pgmap v8025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:55:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:51 smithi116 ceph-mon[35411]: pgmap v8025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:55:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:53 smithi114 ceph-mon[30820]: pgmap v8026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:55:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:53 smithi116 ceph-mon[35411]: pgmap v8026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:55:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:55 smithi114 ceph-mon[30820]: pgmap v8027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:55 smithi116 ceph-mon[35411]: pgmap v8027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:57 smithi114 ceph-mon[30820]: pgmap v8028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:57 smithi116 ceph-mon[35411]: pgmap v8028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:55:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:55:59 smithi114 ceph-mon[30820]: pgmap v8029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:55:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:55:59 smithi116 ceph-mon[35411]: pgmap v8029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:01.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:01 smithi114 ceph-mon[30820]: pgmap v8030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:01 smithi116 ceph-mon[35411]: pgmap v8030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:02 smithi114 ceph-mon[30820]: pgmap v8031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:02 smithi116 ceph-mon[35411]: pgmap v8031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:05 smithi114 ceph-mon[30820]: pgmap v8032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:05 smithi116 ceph-mon[35411]: pgmap v8032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:07 smithi114 ceph-mon[30820]: pgmap v8033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:56:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:07 smithi116 ceph-mon[35411]: pgmap v8033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:56:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:09 smithi114 ceph-mon[30820]: pgmap v8034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:09 smithi116 ceph-mon[35411]: pgmap v8034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:11 smithi114 ceph-mon[30820]: pgmap v8035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:11 smithi116 ceph-mon[35411]: pgmap v8035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:13 smithi114 ceph-mon[30820]: pgmap v8036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:13 smithi116 ceph-mon[35411]: pgmap v8036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:15 smithi114 ceph-mon[30820]: pgmap v8037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:15 smithi116 ceph-mon[35411]: pgmap v8037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:17.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:17 smithi114 ceph-mon[30820]: pgmap v8038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:56:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:17 smithi116 ceph-mon[35411]: pgmap v8038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:56:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:18 smithi114 ceph-mon[30820]: pgmap v8039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:18 smithi116 ceph-mon[35411]: pgmap v8039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:21 smithi114 ceph-mon[30820]: pgmap v8040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:21 smithi116 ceph-mon[35411]: pgmap v8040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:23 smithi114 ceph-mon[30820]: pgmap v8041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:23 smithi116 ceph-mon[35411]: pgmap v8041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:25 smithi114 ceph-mon[30820]: pgmap v8042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:25.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:25 smithi116 ceph-mon[35411]: pgmap v8042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:27 smithi114 ceph-mon[30820]: pgmap v8043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:56:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:27 smithi116 ceph-mon[35411]: pgmap v8043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:56:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:29 smithi114 ceph-mon[30820]: pgmap v8044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:29 smithi116 ceph-mon[35411]: pgmap v8044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:31 smithi114 ceph-mon[30820]: pgmap v8045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:31 smithi116 ceph-mon[35411]: pgmap v8045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:32 smithi114 ceph-mon[30820]: pgmap v8046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:32 smithi116 ceph-mon[35411]: pgmap v8046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:35 smithi114 ceph-mon[30820]: pgmap v8047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:35 smithi116 ceph-mon[35411]: pgmap v8047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:37.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:37 smithi114 ceph-mon[30820]: pgmap v8048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:56:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:37 smithi116 ceph-mon[35411]: pgmap v8048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:56:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:39 smithi114 ceph-mon[30820]: pgmap v8049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:39 smithi116 ceph-mon[35411]: pgmap v8049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:41 smithi114 ceph-mon[30820]: pgmap v8050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:41 smithi116 ceph-mon[35411]: pgmap v8050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:56:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:43 smithi114 ceph-mon[30820]: pgmap v8051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:56:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:43 smithi116 ceph-mon[35411]: pgmap v8051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:56:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:56:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:44 smithi114 ceph-mon[30820]: pgmap v8052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:44 smithi116 ceph-mon[35411]: pgmap v8052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:56:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:56:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:56:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:56:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:56:45.332+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:56:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:56:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:56:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:56:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:56:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:56:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:56:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:56:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:56:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:56:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:56:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:56:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:56:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:56:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:56:45.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:56:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:46 smithi114 ceph-mon[30820]: pgmap v8053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T10:56:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:46 smithi114 ceph-mon[30820]: pgmap v8054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:56:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:56:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:56:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:56:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:56:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:56:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:56:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:56:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:56:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:56:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:46 smithi116 ceph-mon[35411]: pgmap v8053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T10:56:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:46 smithi116 ceph-mon[35411]: pgmap v8054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:56:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:48 smithi114 ceph-mon[30820]: pgmap v8055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:56:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:48 smithi116 ceph-mon[35411]: pgmap v8055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:56:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:50 smithi114 ceph-mon[30820]: pgmap v8056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:56:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:50 smithi116 ceph-mon[35411]: pgmap v8056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:56:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:52 smithi114 ceph-mon[30820]: pgmap v8057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:56:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:52 smithi116 ceph-mon[35411]: pgmap v8057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:56:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:54 smithi114 ceph-mon[30820]: pgmap v8058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:56:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:54 smithi116 ceph-mon[35411]: pgmap v8058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:56:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:56 smithi116 ceph-mon[35411]: pgmap v8059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:56:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:56 smithi114 ceph-mon[30820]: pgmap v8059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:56:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:56:58 smithi116 ceph-mon[35411]: pgmap v8060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:56:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:56:58 smithi114 ceph-mon[30820]: pgmap v8060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:00 smithi116 ceph-mon[35411]: pgmap v8061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:00 smithi114 ceph-mon[30820]: pgmap v8061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:02 smithi114 ceph-mon[30820]: pgmap v8062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:02 smithi116 ceph-mon[35411]: pgmap v8062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:04 smithi114 ceph-mon[30820]: pgmap v8063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:04 smithi116 ceph-mon[35411]: pgmap v8063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:06 smithi114 ceph-mon[30820]: pgmap v8064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:06 smithi116 ceph-mon[35411]: pgmap v8064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:08 smithi114 ceph-mon[30820]: pgmap v8065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:08 smithi116 ceph-mon[35411]: pgmap v8065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:10.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:10 smithi114 ceph-mon[30820]: pgmap v8066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:10 smithi116 ceph-mon[35411]: pgmap v8066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:12.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:12 smithi114 ceph-mon[30820]: pgmap v8067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:12 smithi116 ceph-mon[35411]: pgmap v8067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:14 smithi114 ceph-mon[30820]: pgmap v8068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:14 smithi116 ceph-mon[35411]: pgmap v8068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:16.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:16 smithi114 ceph-mon[30820]: pgmap v8069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:16.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:16 smithi116 ceph-mon[35411]: pgmap v8069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:18 smithi114 ceph-mon[30820]: pgmap v8070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:18 smithi116 ceph-mon[35411]: pgmap v8070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:20 smithi114 ceph-mon[30820]: pgmap v8071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:20 smithi116 ceph-mon[35411]: pgmap v8071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:22 smithi114 ceph-mon[30820]: pgmap v8072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:22.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:22 smithi116 ceph-mon[35411]: pgmap v8072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:24 smithi114 ceph-mon[30820]: pgmap v8073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:24 smithi116 ceph-mon[35411]: pgmap v8073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:26 smithi114 ceph-mon[30820]: pgmap v8074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:26.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:26 smithi116 ceph-mon[35411]: pgmap v8074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:28.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:28 smithi114 ceph-mon[30820]: pgmap v8075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:28 smithi116 ceph-mon[35411]: pgmap v8075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:30 smithi114 ceph-mon[30820]: pgmap v8076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:30 smithi116 ceph-mon[35411]: pgmap v8076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:32 smithi114 ceph-mon[30820]: pgmap v8077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:32.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:32 smithi116 ceph-mon[35411]: pgmap v8077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:34 smithi116 ceph-mon[35411]: pgmap v8078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:34 smithi114 ceph-mon[30820]: pgmap v8078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:36.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:36 smithi116 ceph-mon[35411]: pgmap v8079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:36 smithi114 ceph-mon[30820]: pgmap v8079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:38 smithi116 ceph-mon[35411]: pgmap v8080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:38 smithi114 ceph-mon[30820]: pgmap v8080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:40.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:40 smithi116 ceph-mon[35411]: pgmap v8081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:40 smithi114 ceph-mon[30820]: pgmap v8081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:57:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:42 smithi114 ceph-mon[30820]: pgmap v8082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:42 smithi116 ceph-mon[35411]: pgmap v8082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:57:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:44 smithi114 ceph-mon[30820]: pgmap v8083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:44 smithi116 ceph-mon[35411]: pgmap v8083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:57:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:57:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:57:47.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:46 smithi114 ceph-mon[30820]: pgmap v8084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:46 smithi116 ceph-mon[35411]: pgmap v8084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T10:57:48.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-mon[30820]: pgmap v8085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:48.970 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:57:48.692+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:57:48.970 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:57:48.970 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:57:48.970 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:57:48.970 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:57:48.970 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:57:48.970 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:57:48.971 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:57:48.972 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:57:48.972 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:57:48.972 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:57:48.972 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:57:48.972 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:57:48.972 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:57:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:57:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:48 smithi116 ceph-mon[35411]: pgmap v8085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:57:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:57:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:57:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:57:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:57:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:57:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:49 smithi114 ceph-mon[30820]: pgmap v8086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:57:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:49 smithi114 ceph-mon[30820]: pgmap v8087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:57:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:57:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:57:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:49 smithi116 ceph-mon[35411]: pgmap v8086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:57:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:49 smithi116 ceph-mon[35411]: pgmap v8087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:57:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:57:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:51 smithi114 ceph-mon[30820]: pgmap v8088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:57:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:51 smithi116 ceph-mon[35411]: pgmap v8088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:57:53.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:53 smithi116 ceph-mon[35411]: pgmap v8089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:57:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:53 smithi114 ceph-mon[30820]: pgmap v8089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:57:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:55 smithi114 ceph-mon[30820]: pgmap v8090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:57:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:55 smithi116 ceph-mon[35411]: pgmap v8090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T10:57:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:57 smithi114 ceph-mon[30820]: pgmap v8091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:57:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:57 smithi116 ceph-mon[35411]: pgmap v8091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:58:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:57:59 smithi114 ceph-mon[30820]: pgmap v8092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:58:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:57:59 smithi116 ceph-mon[35411]: pgmap v8092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:58:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:01 smithi114 ceph-mon[30820]: pgmap v8093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:01 smithi116 ceph-mon[35411]: pgmap v8093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:03 smithi114 ceph-mon[30820]: pgmap v8094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:03 smithi116 ceph-mon[35411]: pgmap v8094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:05 smithi114 ceph-mon[30820]: pgmap v8095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:05 smithi116 ceph-mon[35411]: pgmap v8095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:07 smithi114 ceph-mon[30820]: pgmap v8096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:07 smithi116 ceph-mon[35411]: pgmap v8096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:09 smithi114 ceph-mon[30820]: pgmap v8097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:09 smithi116 ceph-mon[35411]: pgmap v8097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:11 smithi114 ceph-mon[30820]: pgmap v8098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:11 smithi116 ceph-mon[35411]: pgmap v8098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:13 smithi114 ceph-mon[30820]: pgmap v8099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:13 smithi116 ceph-mon[35411]: pgmap v8099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:15 smithi114 ceph-mon[30820]: pgmap v8100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:15 smithi116 ceph-mon[35411]: pgmap v8100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:17 smithi114 ceph-mon[30820]: pgmap v8101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:17 smithi116 ceph-mon[35411]: pgmap v8101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:19 smithi114 ceph-mon[30820]: pgmap v8102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:19 smithi116 ceph-mon[35411]: pgmap v8102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:21 smithi114 ceph-mon[30820]: pgmap v8103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:21 smithi116 ceph-mon[35411]: pgmap v8103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:23 smithi114 ceph-mon[30820]: pgmap v8104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:23 smithi116 ceph-mon[35411]: pgmap v8104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:25 smithi114 ceph-mon[30820]: pgmap v8105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:25 smithi116 ceph-mon[35411]: pgmap v8105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:28.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:27 smithi114 ceph-mon[30820]: pgmap v8106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:27 smithi116 ceph-mon[35411]: pgmap v8106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:29 smithi114 ceph-mon[30820]: pgmap v8107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:29 smithi116 ceph-mon[35411]: pgmap v8107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:31 smithi114 ceph-mon[30820]: pgmap v8108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:31 smithi116 ceph-mon[35411]: pgmap v8108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:33 smithi114 ceph-mon[30820]: pgmap v8109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:33 smithi116 ceph-mon[35411]: pgmap v8109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:35 smithi114 ceph-mon[30820]: pgmap v8110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:35 smithi116 ceph-mon[35411]: pgmap v8110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:37 smithi114 ceph-mon[30820]: pgmap v8111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:37 smithi116 ceph-mon[35411]: pgmap v8111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:39 smithi114 ceph-mon[30820]: pgmap v8112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:39 smithi116 ceph-mon[35411]: pgmap v8112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:41 smithi114 ceph-mon[30820]: pgmap v8113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:41 smithi116 ceph-mon[35411]: pgmap v8113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:43 smithi114 ceph-mon[30820]: pgmap v8114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:43.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:43 smithi116 ceph-mon[35411]: pgmap v8114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:45 smithi114 ceph-mon[30820]: pgmap v8115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:45 smithi116 ceph-mon[35411]: pgmap v8115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:58:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:47 smithi114 ceph-mon[30820]: pgmap v8116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:48.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:47 smithi116 ceph-mon[35411]: pgmap v8116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:58:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:49 smithi114 ceph-mon[30820]: pgmap v8117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:58:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:49 smithi116 ceph-mon[35411]: pgmap v8117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:58:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:58:52.054 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:51 smithi114 ceph-mon[30820]: pgmap v8118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:51 smithi116 ceph-mon[35411]: pgmap v8118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:58:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:58:52.052+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:58:52.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:58:52.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:58:52.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:58:52.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:58:52.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:58:52.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:58:52.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:58:52.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:58:52.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:58:52.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:58:53.079 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:52 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:58:53.080 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:58:53.081 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:52 smithi116 ceph-mon[35411]: pgmap v8119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:58:53.082 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:52 smithi116 ceph-mon[35411]: pgmap v8120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:58:53.082 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:58:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:58:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:58:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:58:53.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:58:53.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:58:53.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:58:53.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:58:53.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-mon[30820]: pgmap v8119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T10:58:53.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-mon[30820]: pgmap v8120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:58:53.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:58:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:54 smithi114 ceph-mon[30820]: pgmap v8121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:58:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:54 smithi116 ceph-mon[35411]: pgmap v8121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T10:58:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:56 smithi114 ceph-mon[30820]: pgmap v8122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:58:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:56 smithi116 ceph-mon[35411]: pgmap v8122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:58:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:58:58 smithi114 ceph-mon[30820]: pgmap v8123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:58:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:58:58 smithi116 ceph-mon[35411]: pgmap v8123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:59:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:00 smithi114 ceph-mon[30820]: pgmap v8124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:59:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:00 smithi116 ceph-mon[35411]: pgmap v8124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T10:59:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:02 smithi114 ceph-mon[30820]: pgmap v8125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:59:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:02 smithi116 ceph-mon[35411]: pgmap v8125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T10:59:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:04 smithi114 ceph-mon[30820]: pgmap v8126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:04 smithi116 ceph-mon[35411]: pgmap v8126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:06 smithi114 ceph-mon[30820]: pgmap v8127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:06 smithi116 ceph-mon[35411]: pgmap v8127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:08 smithi114 ceph-mon[30820]: pgmap v8128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:08 smithi116 ceph-mon[35411]: pgmap v8128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:10 smithi114 ceph-mon[30820]: pgmap v8129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:10 smithi116 ceph-mon[35411]: pgmap v8129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:12 smithi114 ceph-mon[30820]: pgmap v8130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:12 smithi116 ceph-mon[35411]: pgmap v8130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:14 smithi114 ceph-mon[30820]: pgmap v8131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:14 smithi116 ceph-mon[35411]: pgmap v8131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:16 smithi114 ceph-mon[30820]: pgmap v8132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:16 smithi116 ceph-mon[35411]: pgmap v8132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:18 smithi114 ceph-mon[30820]: pgmap v8133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:18 smithi116 ceph-mon[35411]: pgmap v8133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:20 smithi114 ceph-mon[30820]: pgmap v8134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:20 smithi116 ceph-mon[35411]: pgmap v8134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:22 smithi114 ceph-mon[30820]: pgmap v8135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:22 smithi116 ceph-mon[35411]: pgmap v8135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:24 smithi114 ceph-mon[30820]: pgmap v8136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:24 smithi116 ceph-mon[35411]: pgmap v8136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:26 smithi114 ceph-mon[30820]: pgmap v8137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:26 smithi116 ceph-mon[35411]: pgmap v8137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:28 smithi114 ceph-mon[30820]: pgmap v8138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:28 smithi116 ceph-mon[35411]: pgmap v8138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:30 smithi114 ceph-mon[30820]: pgmap v8139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:30 smithi116 ceph-mon[35411]: pgmap v8139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:32.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:32 smithi114 ceph-mon[30820]: pgmap v8140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:32 smithi116 ceph-mon[35411]: pgmap v8140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:34 smithi114 ceph-mon[30820]: pgmap v8141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:34 smithi116 ceph-mon[35411]: pgmap v8141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:36 smithi114 ceph-mon[30820]: pgmap v8142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:36 smithi116 ceph-mon[35411]: pgmap v8142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:38 smithi114 ceph-mon[30820]: pgmap v8143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:38 smithi116 ceph-mon[35411]: pgmap v8143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:40 smithi114 ceph-mon[30820]: pgmap v8144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:40 smithi116 ceph-mon[35411]: pgmap v8144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:42 smithi114 ceph-mon[30820]: pgmap v8145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:42 smithi116 ceph-mon[35411]: pgmap v8145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:44 smithi114 ceph-mon[30820]: pgmap v8146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:44 smithi116 ceph-mon[35411]: pgmap v8146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:46 smithi114 ceph-mon[30820]: pgmap v8147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:46 smithi116 ceph-mon[35411]: pgmap v8147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T10:59:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:48 smithi114 ceph-mon[30820]: pgmap v8148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:48 smithi116 ceph-mon[35411]: pgmap v8148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:50 smithi114 ceph-mon[30820]: pgmap v8149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:50 smithi116 ceph-mon[35411]: pgmap v8149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T10:59:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:52 smithi114 ceph-mon[30820]: pgmap v8150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:52 smithi116 ceph-mon[35411]: pgmap v8150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T10:59:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:59:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T10:59:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:54 smithi114 ceph-mon[30820]: pgmap v8151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:54 smithi116 ceph-mon[35411]: pgmap v8151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T10:59:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:59:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:59:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:59:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T10:59:55.412+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:59:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T10:59:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:59:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T10:59:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:59:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:59:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T10:59:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:59:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T10:59:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:59:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:59:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 10:59:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:59:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T10:59:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T10:59:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T10:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:59:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:56 smithi114 ceph-mon[30820]: pgmap v8152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T10:59:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:56 smithi114 ceph-mon[30820]: pgmap v8153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T10:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T10:59:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:56 smithi116 ceph-mon[35411]: pgmap v8152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T10:59:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:56 smithi116 ceph-mon[35411]: pgmap v8153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T10:59:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 10:59:58 smithi114 ceph-mon[30820]: pgmap v8154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T10:59:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 10:59:58 smithi116 ceph-mon[35411]: pgmap v8154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:00:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:00 smithi114 ceph-mon[30820]: pgmap v8155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:00:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:00:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:00 smithi116 ceph-mon[35411]: pgmap v8155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:00:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:00:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:02 smithi114 ceph-mon[30820]: pgmap v8156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:00:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:02 smithi116 ceph-mon[35411]: pgmap v8156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:00:04.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:04 smithi114 ceph-mon[30820]: pgmap v8157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:00:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:04 smithi116 ceph-mon[35411]: pgmap v8157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:00:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:06 smithi114 ceph-mon[30820]: pgmap v8158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:00:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:06 smithi116 ceph-mon[35411]: pgmap v8158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:00:08.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:08 smithi114 ceph-mon[30820]: pgmap v8159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:08.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:08 smithi116 ceph-mon[35411]: pgmap v8159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:10.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:10 smithi116 ceph-mon[35411]: pgmap v8160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:10 smithi114 ceph-mon[30820]: pgmap v8160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:12.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:12 smithi116 ceph-mon[35411]: pgmap v8161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:12 smithi114 ceph-mon[30820]: pgmap v8161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:14.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:14 smithi116 ceph-mon[35411]: pgmap v8162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:14 smithi114 ceph-mon[30820]: pgmap v8162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:16 smithi114 ceph-mon[30820]: pgmap v8163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:00:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:16 smithi116 ceph-mon[35411]: pgmap v8163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:00:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:18 smithi114 ceph-mon[30820]: pgmap v8164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:18 smithi116 ceph-mon[35411]: pgmap v8164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:20.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:20 smithi114 ceph-mon[30820]: pgmap v8165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:20 smithi116 ceph-mon[35411]: pgmap v8165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:23.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:22 smithi114 ceph-mon[30820]: pgmap v8166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:22 smithi116 ceph-mon[35411]: pgmap v8166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:24.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:24 smithi116 ceph-mon[35411]: pgmap v8167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:24 smithi114 ceph-mon[30820]: pgmap v8167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:26 smithi114 ceph-mon[30820]: pgmap v8168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:00:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:26 smithi116 ceph-mon[35411]: pgmap v8168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:00:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:28 smithi114 ceph-mon[30820]: pgmap v8169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:28 smithi116 ceph-mon[35411]: pgmap v8169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:30 smithi114 ceph-mon[30820]: pgmap v8170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:31.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:30 smithi116 ceph-mon[35411]: pgmap v8170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:32.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:32 smithi114 ceph-mon[30820]: pgmap v8171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:33.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:32 smithi116 ceph-mon[35411]: pgmap v8171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:34 smithi114 ceph-mon[30820]: pgmap v8172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:35.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:34 smithi116 ceph-mon[35411]: pgmap v8172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:36 smithi114 ceph-mon[30820]: pgmap v8173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:00:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:36 smithi116 ceph-mon[35411]: pgmap v8173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:00:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:38 smithi114 ceph-mon[30820]: pgmap v8174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:38 smithi116 ceph-mon[35411]: pgmap v8174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:41.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:40 smithi114 ceph-mon[30820]: pgmap v8175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:41.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:40 smithi116 ceph-mon[35411]: pgmap v8175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:42 smithi114 ceph-mon[30820]: pgmap v8176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:42 smithi116 ceph-mon[35411]: pgmap v8176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:44 smithi114 ceph-mon[30820]: pgmap v8177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:44 smithi116 ceph-mon[35411]: pgmap v8177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:47.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:46 smithi114 ceph-mon[30820]: pgmap v8178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:00:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:46 smithi116 ceph-mon[35411]: pgmap v8178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:00:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:48 smithi114 ceph-mon[30820]: pgmap v8179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:48 smithi116 ceph-mon[35411]: pgmap v8179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:00:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:50 smithi114 ceph-mon[30820]: pgmap v8180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:50 smithi116 ceph-mon[35411]: pgmap v8180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:00:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:52 smithi114 ceph-mon[30820]: pgmap v8181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:52 smithi116 ceph-mon[35411]: pgmap v8181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:00:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:54 smithi114 ceph-mon[30820]: pgmap v8182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:54 smithi116 ceph-mon[35411]: pgmap v8182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:55.844 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:00:56.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:00:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: pgmap v8183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: pgmap v8183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:00:58.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:57 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:00:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:57 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:00:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:57 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:00:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:57 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:00:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-mon[30820]: pgmap v8184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:00:59.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:00:58.772+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:00:59.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:00:59.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:00:59.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:00:59.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:00:59.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:00:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:00:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:00:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:00:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:00:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:00:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:00:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:58 smithi116 ceph-mon[35411]: pgmap v8184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:59 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:01:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:01:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:01:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:01:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:01:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:59 smithi114 ceph-mon[30820]: pgmap v8185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:01:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:59 smithi114 ceph-mon[30820]: pgmap v8186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:01:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:00:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:59 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:01:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:01:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:01:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:01:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:59 smithi116 ceph-mon[35411]: pgmap v8185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:01:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:59 smithi116 ceph-mon[35411]: pgmap v8186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:01:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:00:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:01:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:01 smithi114 ceph-mon[30820]: pgmap v8187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:01:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:01 smithi116 ceph-mon[35411]: pgmap v8187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:01:03.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:03 smithi114 ceph-mon[30820]: pgmap v8188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:01:03.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:03 smithi116 ceph-mon[35411]: pgmap v8188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:01:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:05 smithi114 ceph-mon[30820]: pgmap v8189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:01:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:05 smithi116 ceph-mon[35411]: pgmap v8189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:01:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:07 smithi114 ceph-mon[30820]: pgmap v8190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:01:08.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:07 smithi116 ceph-mon[35411]: pgmap v8190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:01:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:09 smithi114 ceph-mon[30820]: pgmap v8191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:01:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:09 smithi116 ceph-mon[35411]: pgmap v8191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:01:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:11 smithi114 ceph-mon[30820]: pgmap v8192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:11 smithi116 ceph-mon[35411]: pgmap v8192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:13 smithi114 ceph-mon[30820]: pgmap v8193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:13 smithi116 ceph-mon[35411]: pgmap v8193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:15 smithi114 ceph-mon[30820]: pgmap v8194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:01:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:15 smithi116 ceph-mon[35411]: pgmap v8194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:01:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:17 smithi114 ceph-mon[30820]: pgmap v8195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:17 smithi116 ceph-mon[35411]: pgmap v8195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:20.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:19 smithi114 ceph-mon[30820]: pgmap v8196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:20.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:19 smithi116 ceph-mon[35411]: pgmap v8196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:21 smithi114 ceph-mon[30820]: pgmap v8197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:22.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:21 smithi116 ceph-mon[35411]: pgmap v8197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:23 smithi114 ceph-mon[30820]: pgmap v8198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:23 smithi116 ceph-mon[35411]: pgmap v8198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:25 smithi114 ceph-mon[30820]: pgmap v8199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:01:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:25 smithi116 ceph-mon[35411]: pgmap v8199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:01:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:27 smithi114 ceph-mon[30820]: pgmap v8200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:27 smithi116 ceph-mon[35411]: pgmap v8200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:30.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:29 smithi114 ceph-mon[30820]: pgmap v8201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:29 smithi116 ceph-mon[35411]: pgmap v8201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:32.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:31 smithi114 ceph-mon[30820]: pgmap v8202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:31 smithi116 ceph-mon[35411]: pgmap v8202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:33 smithi114 ceph-mon[30820]: pgmap v8203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:33 smithi116 ceph-mon[35411]: pgmap v8203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:36.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:35 smithi114 ceph-mon[30820]: pgmap v8204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:01:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:35 smithi116 ceph-mon[35411]: pgmap v8204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:01:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:37 smithi114 ceph-mon[30820]: pgmap v8205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:37 smithi116 ceph-mon[35411]: pgmap v8205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:40.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:39 smithi114 ceph-mon[30820]: pgmap v8206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:39 smithi116 ceph-mon[35411]: pgmap v8206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:41 smithi114 ceph-mon[30820]: pgmap v8207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:42.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:41 smithi116 ceph-mon[35411]: pgmap v8207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:43 smithi114 ceph-mon[30820]: pgmap v8208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:43 smithi116 ceph-mon[35411]: pgmap v8208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:01:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:45 smithi114 ceph-mon[30820]: pgmap v8209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:01:46.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:45 smithi116 ceph-mon[35411]: pgmap v8209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:01:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:47 smithi114 ceph-mon[30820]: pgmap v8210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:47 smithi116 ceph-mon[35411]: pgmap v8210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:50.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:49 smithi114 ceph-mon[30820]: pgmap v8211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:01:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:49 smithi116 ceph-mon[35411]: pgmap v8211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:01:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:51 smithi116 ceph-mon[35411]: pgmap v8212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:51 smithi114 ceph-mon[30820]: pgmap v8212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:01:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:53 smithi116 ceph-mon[35411]: pgmap v8213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:01:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:53 smithi114 ceph-mon[30820]: pgmap v8213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:01:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:55 smithi116 ceph-mon[35411]: pgmap v8214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:01:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:55 smithi114 ceph-mon[30820]: pgmap v8214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:01:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:57 smithi114 ceph-mon[30820]: pgmap v8215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:57 smithi116 ceph-mon[35411]: pgmap v8215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:01:59.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:01:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:02:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:01:59 smithi114 ceph-mon[30820]: pgmap v8216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:01:59 smithi116 ceph-mon[35411]: pgmap v8216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:01 smithi114 ceph-mon[30820]: pgmap v8217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:02:02.132+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:02:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:02:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:02:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:02:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:02:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:02:02.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:02:02.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:02:02.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:02:02.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:02:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:01 smithi116 ceph-mon[35411]: pgmap v8217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:02:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:02:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:02:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:02:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:02:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:02:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:02:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:02:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:02:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:02:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:02:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:02:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:02:04.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:02:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:02:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:02:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:03 smithi114 ceph-mon[30820]: pgmap v8218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:02:04.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:03 smithi114 ceph-mon[30820]: pgmap v8219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:02:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:02:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:02:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:02:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:02:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:02:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:02:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:02:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:02:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:02:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:02:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:02:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:02:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:02:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:02:04.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:02:04.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:03 smithi116 ceph-mon[35411]: pgmap v8218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:02:04.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:03 smithi116 ceph-mon[35411]: pgmap v8219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:02:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:05 smithi114 ceph-mon[30820]: pgmap v8220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:02:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:05 smithi116 ceph-mon[35411]: pgmap v8220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:02:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:07 smithi114 ceph-mon[30820]: pgmap v8221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:02:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:07 smithi116 ceph-mon[35411]: pgmap v8221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:02:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:09 smithi114 ceph-mon[30820]: pgmap v8222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:02:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:09 smithi116 ceph-mon[35411]: pgmap v8222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:02:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:11 smithi114 ceph-mon[30820]: pgmap v8223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:02:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:11 smithi116 ceph-mon[35411]: pgmap v8223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:02:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:13 smithi114 ceph-mon[30820]: pgmap v8224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:02:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:13 smithi116 ceph-mon[35411]: pgmap v8224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:02:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:15 smithi114 ceph-mon[30820]: pgmap v8225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:15 smithi116 ceph-mon[35411]: pgmap v8225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:17 smithi114 ceph-mon[30820]: pgmap v8226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:17 smithi116 ceph-mon[35411]: pgmap v8226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:19 smithi114 ceph-mon[30820]: pgmap v8227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:19 smithi116 ceph-mon[35411]: pgmap v8227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:21 smithi114 ceph-mon[30820]: pgmap v8228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:02:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:21 smithi116 ceph-mon[35411]: pgmap v8228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:02:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:23 smithi114 ceph-mon[30820]: pgmap v8229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:02:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:23 smithi116 ceph-mon[35411]: pgmap v8229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:02:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:25 smithi114 ceph-mon[30820]: pgmap v8230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:25 smithi116 ceph-mon[35411]: pgmap v8230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:27 smithi114 ceph-mon[30820]: pgmap v8231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:27 smithi116 ceph-mon[35411]: pgmap v8231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:29 smithi114 ceph-mon[30820]: pgmap v8232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:30 smithi116 ceph-mon[35411]: pgmap v8232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:32 smithi114 ceph-mon[30820]: pgmap v8233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:02:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:32 smithi116 ceph-mon[35411]: pgmap v8233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:02:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:34 smithi114 ceph-mon[30820]: pgmap v8234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:02:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:34 smithi116 ceph-mon[35411]: pgmap v8234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:02:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:36 smithi114 ceph-mon[30820]: pgmap v8235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:36 smithi116 ceph-mon[35411]: pgmap v8235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:38 smithi114 ceph-mon[30820]: pgmap v8236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:38.503 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:38 smithi116 ceph-mon[35411]: pgmap v8236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:40 smithi114 ceph-mon[30820]: pgmap v8237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:40 smithi116 ceph-mon[35411]: pgmap v8237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:42 smithi114 ceph-mon[30820]: pgmap v8238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:02:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:42 smithi116 ceph-mon[35411]: pgmap v8238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:02:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:44 smithi114 ceph-mon[30820]: pgmap v8239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:02:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:44 smithi116 ceph-mon[35411]: pgmap v8239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:02:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:46 smithi114 ceph-mon[30820]: pgmap v8240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:46 smithi116 ceph-mon[35411]: pgmap v8240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:48 smithi114 ceph-mon[30820]: pgmap v8241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:48 smithi116 ceph-mon[35411]: pgmap v8241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:50 smithi114 ceph-mon[30820]: pgmap v8242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:50 smithi116 ceph-mon[35411]: pgmap v8242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:02:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:52 smithi114 ceph-mon[30820]: pgmap v8243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:02:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:52 smithi116 ceph-mon[35411]: pgmap v8243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:02:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:54 smithi114 ceph-mon[30820]: pgmap v8244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:02:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:54 smithi116 ceph-mon[35411]: pgmap v8244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:02:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:56 smithi114 ceph-mon[30820]: pgmap v8245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:02:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:56 smithi116 ceph-mon[35411]: pgmap v8245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:02:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:02:58 smithi114 ceph-mon[30820]: pgmap v8246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:02:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:02:58 smithi116 ceph-mon[35411]: pgmap v8246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:00 smithi114 ceph-mon[30820]: pgmap v8247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:03:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:00 smithi116 ceph-mon[35411]: pgmap v8247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:03:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:02 smithi114 ceph-mon[30820]: pgmap v8248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:03:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:02 smithi116 ceph-mon[35411]: pgmap v8248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:03:03.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:03:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:03:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:04 smithi114 ceph-mon[30820]: pgmap v8249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:04 smithi116 ceph-mon[35411]: pgmap v8249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:05.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:03:05.492+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:03:05.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:03:05.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:03:05.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:03:05.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:03:05.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:03:05.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:03:05.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:03:05.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:03:05.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:03:05.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:03:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:03:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:06 smithi114 ceph-mon[30820]: pgmap v8250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:03:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:03:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:03:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:06 smithi116 ceph-mon[35411]: pgmap v8250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:03:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:03:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:03:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:03:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:03:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:03:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:03:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:03:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:03:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:03:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:03:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:03:07.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:03:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:03:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:03:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:03:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:03:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:03:07.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:03:07.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:03:07.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:03:07.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:03:07.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:03:07.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:03:07.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:03:07.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:03:07.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:07 smithi114 ceph-mon[30820]: pgmap v8251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T11:03:07.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:07 smithi114 ceph-mon[30820]: pgmap v8252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:03:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:03:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:03:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:03:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:03:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:03:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:03:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:03:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:03:07.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:03:07.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:03:07.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:03:07.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:03:07.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:03:07.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:03:07.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:03:07.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:03:07.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:03:07.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:03:07.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:03:07.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:03:07.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:03:07.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:03:07.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:07 smithi116 ceph-mon[35411]: pgmap v8251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T11:03:07.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:07 smithi116 ceph-mon[35411]: pgmap v8252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:09 smithi114 ceph-mon[30820]: pgmap v8253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:09 smithi116 ceph-mon[35411]: pgmap v8253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:11.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:11 smithi114 ceph-mon[30820]: pgmap v8254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:11 smithi116 ceph-mon[35411]: pgmap v8254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:13 smithi114 ceph-mon[30820]: pgmap v8255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:13 smithi116 ceph-mon[35411]: pgmap v8255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:15 smithi114 ceph-mon[30820]: pgmap v8256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:15 smithi116 ceph-mon[35411]: pgmap v8256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:03:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:17 smithi114 ceph-mon[30820]: pgmap v8257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:03:17.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:17 smithi116 ceph-mon[35411]: pgmap v8257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:03:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:19 smithi114 ceph-mon[30820]: pgmap v8258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:19 smithi116 ceph-mon[35411]: pgmap v8258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:21.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:21 smithi114 ceph-mon[30820]: pgmap v8259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:03:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:21 smithi116 ceph-mon[35411]: pgmap v8259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:03:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:23 smithi114 ceph-mon[30820]: pgmap v8260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:23 smithi116 ceph-mon[35411]: pgmap v8260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:25 smithi114 ceph-mon[30820]: pgmap v8261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:25 smithi116 ceph-mon[35411]: pgmap v8261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:27.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:27 smithi114 ceph-mon[30820]: pgmap v8262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:27 smithi116 ceph-mon[35411]: pgmap v8262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:29 smithi114 ceph-mon[30820]: pgmap v8263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:29 smithi116 ceph-mon[35411]: pgmap v8263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:31.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:31 smithi114 ceph-mon[30820]: pgmap v8264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:03:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:31 smithi116 ceph-mon[35411]: pgmap v8264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:03:33.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:33 smithi114 ceph-mon[30820]: pgmap v8265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:33 smithi116 ceph-mon[35411]: pgmap v8265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:35.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:35 smithi114 ceph-mon[30820]: pgmap v8266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:35 smithi116 ceph-mon[35411]: pgmap v8266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:37 smithi114 ceph-mon[30820]: pgmap v8267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:37 smithi116 ceph-mon[35411]: pgmap v8267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:39 smithi114 ceph-mon[30820]: pgmap v8268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:39 smithi116 ceph-mon[35411]: pgmap v8268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:41 smithi114 ceph-mon[30820]: pgmap v8269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:03:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:41 smithi116 ceph-mon[35411]: pgmap v8269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:03:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:43 smithi114 ceph-mon[30820]: pgmap v8270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:43 smithi116 ceph-mon[35411]: pgmap v8270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:44 smithi114 ceph-mon[30820]: pgmap v8271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:44 smithi116 ceph-mon[35411]: pgmap v8271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:46 smithi114 ceph-mon[30820]: pgmap v8272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:46 smithi116 ceph-mon[35411]: pgmap v8272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:48 smithi114 ceph-mon[30820]: pgmap v8273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:48 smithi116 ceph-mon[35411]: pgmap v8273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:50 smithi114 ceph-mon[30820]: pgmap v8274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:03:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:50 smithi116 ceph-mon[35411]: pgmap v8274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:03:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:52 smithi114 ceph-mon[30820]: pgmap v8275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:52 smithi116 ceph-mon[35411]: pgmap v8275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:03:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:54 smithi114 ceph-mon[30820]: pgmap v8276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:54 smithi116 ceph-mon[35411]: pgmap v8276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:56 smithi114 ceph-mon[30820]: pgmap v8277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:56.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:56 smithi116 ceph-mon[35411]: pgmap v8277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:03:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:03:58 smithi114 ceph-mon[30820]: pgmap v8278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:03:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:03:58 smithi116 ceph-mon[35411]: pgmap v8278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:00.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:00 smithi114 ceph-mon[30820]: pgmap v8279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:00.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:00 smithi116 ceph-mon[35411]: pgmap v8279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:02.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:02 smithi114 ceph-mon[30820]: pgmap v8280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:02.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:02 smithi116 ceph-mon[35411]: pgmap v8280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:04.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:04 smithi114 ceph-mon[30820]: pgmap v8281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:04.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:04 smithi116 ceph-mon[35411]: pgmap v8281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:06.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:06 smithi114 ceph-mon[30820]: pgmap v8282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:04:06.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:06 smithi116 ceph-mon[35411]: pgmap v8282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:04:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:09.136 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:04:08.852+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:04:09.136 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:04:09.136 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:04:09.137 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:04:09.138 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:04:09.138 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:04:09.138 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:04:09.138 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:04:09.138 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:04:09.138 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:04:09.138 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:04:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:04:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:09 smithi114 ceph-mon[30820]: pgmap v8283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:04:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:04:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:09 smithi116 ceph-mon[35411]: pgmap v8283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:04:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:04:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:04:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:10 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:10 smithi114 ceph-mon[30820]: pgmap v8284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:04:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:10 smithi114 ceph-mon[30820]: pgmap v8285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:10 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:10 smithi116 ceph-mon[35411]: pgmap v8284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:04:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:10 smithi116 ceph-mon[35411]: pgmap v8285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:04:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:12 smithi114 ceph-mon[30820]: pgmap v8286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:04:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:12 smithi116 ceph-mon[35411]: pgmap v8286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:04:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:14 smithi114 ceph-mon[30820]: pgmap v8287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:04:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:14 smithi116 ceph-mon[35411]: pgmap v8287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:04:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:16 smithi114 ceph-mon[30820]: pgmap v8288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:04:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:16 smithi116 ceph-mon[35411]: pgmap v8288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:04:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:18 smithi114 ceph-mon[30820]: pgmap v8289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:04:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:18 smithi116 ceph-mon[35411]: pgmap v8289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:04:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:19 smithi114 ceph-mon[30820]: pgmap v8290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:04:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:19 smithi116 ceph-mon[35411]: pgmap v8290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:04:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:21 smithi116 ceph-mon[35411]: pgmap v8291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:21 smithi114 ceph-mon[30820]: pgmap v8291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:23 smithi116 ceph-mon[35411]: pgmap v8292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:24.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:23 smithi114 ceph-mon[30820]: pgmap v8292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:25 smithi114 ceph-mon[30820]: pgmap v8293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:25 smithi116 ceph-mon[35411]: pgmap v8293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:27 smithi116 ceph-mon[35411]: pgmap v8294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:27 smithi114 ceph-mon[30820]: pgmap v8294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:29 smithi114 ceph-mon[30820]: pgmap v8295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:29 smithi116 ceph-mon[35411]: pgmap v8295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:31 smithi114 ceph-mon[30820]: pgmap v8296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:31 smithi116 ceph-mon[35411]: pgmap v8296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:33 smithi114 ceph-mon[30820]: pgmap v8297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:33 smithi116 ceph-mon[35411]: pgmap v8297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:35 smithi114 ceph-mon[30820]: pgmap v8298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:35 smithi116 ceph-mon[35411]: pgmap v8298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:37 smithi114 ceph-mon[30820]: pgmap v8299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:37 smithi116 ceph-mon[35411]: pgmap v8299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:39 smithi114 ceph-mon[30820]: pgmap v8300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:39 smithi116 ceph-mon[35411]: pgmap v8300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:41 smithi114 ceph-mon[30820]: pgmap v8301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:41 smithi116 ceph-mon[35411]: pgmap v8301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:43 smithi114 ceph-mon[30820]: pgmap v8302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:43 smithi116 ceph-mon[35411]: pgmap v8302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:45 smithi114 ceph-mon[30820]: pgmap v8303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:45 smithi116 ceph-mon[35411]: pgmap v8303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:47 smithi114 ceph-mon[30820]: pgmap v8304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:47 smithi116 ceph-mon[35411]: pgmap v8304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:49 smithi114 ceph-mon[30820]: pgmap v8305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:49 smithi116 ceph-mon[35411]: pgmap v8305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:51 smithi114 ceph-mon[30820]: pgmap v8306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:51 smithi116 ceph-mon[35411]: pgmap v8306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:04:54.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:53 smithi114 ceph-mon[30820]: pgmap v8307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:53 smithi116 ceph-mon[35411]: pgmap v8307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:04:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:55 smithi114 ceph-mon[30820]: pgmap v8308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:55 smithi116 ceph-mon[35411]: pgmap v8308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:04:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:57 smithi114 ceph-mon[30820]: pgmap v8309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:04:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:57 smithi116 ceph-mon[35411]: pgmap v8309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:04:59 smithi114 ceph-mon[30820]: pgmap v8310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:04:59 smithi116 ceph-mon[35411]: pgmap v8310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:01 smithi114 ceph-mon[30820]: pgmap v8311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:02 smithi116 ceph-mon[35411]: pgmap v8311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:03 smithi114 ceph-mon[30820]: pgmap v8312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:03 smithi116 ceph-mon[35411]: pgmap v8312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:05 smithi114 ceph-mon[30820]: pgmap v8313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:05:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:05 smithi116 ceph-mon[35411]: pgmap v8313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:05:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:07 smithi114 ceph-mon[30820]: pgmap v8314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:07 smithi116 ceph-mon[35411]: pgmap v8314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:09 smithi114 ceph-mon[30820]: pgmap v8315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:05:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:05:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:09 smithi116 ceph-mon[35411]: pgmap v8315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:05:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:05:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:11 smithi114 ceph-mon[30820]: pgmap v8316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:05:12.148+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:05:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:05:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:05:12.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:05:12.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:05:12.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:05:12.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:05:12.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:05:12.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:05:12.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:05:12.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:05:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:11 smithi116 ceph-mon[35411]: pgmap v8316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:05:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:05:13.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:05:13.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:05:13.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:05:13.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:05:13.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-mon[30820]: pgmap v8317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:05:13.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-mon[30820]: pgmap v8318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:05:13.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:12 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:05:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:12 smithi116 ceph-mon[35411]: pgmap v8317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:12 smithi116 ceph-mon[35411]: pgmap v8318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:05:13.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:05:14.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:14 smithi114 ceph-mon[30820]: pgmap v8319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T11:05:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:14 smithi116 ceph-mon[35411]: pgmap v8319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T11:05:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:16 smithi114 ceph-mon[30820]: pgmap v8320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:05:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:16 smithi116 ceph-mon[35411]: pgmap v8320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:05:18.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:18 smithi114 ceph-mon[30820]: pgmap v8321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:05:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:18 smithi116 ceph-mon[35411]: pgmap v8321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:05:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:20 smithi114 ceph-mon[30820]: pgmap v8322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:05:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:20 smithi116 ceph-mon[35411]: pgmap v8322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:05:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:22 smithi114 ceph-mon[30820]: pgmap v8323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:05:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:22 smithi116 ceph-mon[35411]: pgmap v8323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:05:24.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:24 smithi114 ceph-mon[30820]: pgmap v8324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:24 smithi116 ceph-mon[35411]: pgmap v8324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:26 smithi114 ceph-mon[30820]: pgmap v8325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:26 smithi116 ceph-mon[35411]: pgmap v8325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:28.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:28 smithi114 ceph-mon[30820]: pgmap v8326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:28 smithi116 ceph-mon[35411]: pgmap v8326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:30.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:30 smithi116 ceph-mon[35411]: pgmap v8327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:05:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:30 smithi114 ceph-mon[30820]: pgmap v8327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:05:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:32 smithi114 ceph-mon[30820]: pgmap v8328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:32 smithi116 ceph-mon[35411]: pgmap v8328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:34 smithi114 ceph-mon[30820]: pgmap v8329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:34 smithi116 ceph-mon[35411]: pgmap v8329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:36 smithi114 ceph-mon[30820]: pgmap v8330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:36 smithi116 ceph-mon[35411]: pgmap v8330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:38 smithi114 ceph-mon[30820]: pgmap v8331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:38 smithi116 ceph-mon[35411]: pgmap v8331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:41.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:41 smithi114 ceph-mon[30820]: pgmap v8332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:05:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:41 smithi116 ceph-mon[35411]: pgmap v8332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:05:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:43 smithi114 ceph-mon[30820]: pgmap v8333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:43 smithi116 ceph-mon[35411]: pgmap v8333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:45 smithi114 ceph-mon[30820]: pgmap v8334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:45 smithi116 ceph-mon[35411]: pgmap v8334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:47 smithi114 ceph-mon[30820]: pgmap v8335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:47.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:47 smithi116 ceph-mon[35411]: pgmap v8335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:49 smithi114 ceph-mon[30820]: pgmap v8336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:49.698 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:49 smithi116 ceph-mon[35411]: pgmap v8336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:51.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:51 smithi114 ceph-mon[30820]: pgmap v8337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:05:51.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:51 smithi116 ceph-mon[35411]: pgmap v8337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:05:53.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:53 smithi114 ceph-mon[30820]: pgmap v8338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:53 smithi116 ceph-mon[35411]: pgmap v8338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:05:55.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:55 smithi114 ceph-mon[30820]: pgmap v8339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:55.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:55 smithi116 ceph-mon[35411]: pgmap v8339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:57.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:57 smithi114 ceph-mon[30820]: pgmap v8340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:57 smithi116 ceph-mon[35411]: pgmap v8340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:05:59.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:05:59 smithi114 ceph-mon[30820]: pgmap v8341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:05:59.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:05:59 smithi116 ceph-mon[35411]: pgmap v8341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:01.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:01 smithi114 ceph-mon[30820]: pgmap v8342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:01.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:01 smithi116 ceph-mon[35411]: pgmap v8342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:03 smithi114 ceph-mon[30820]: pgmap v8343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:03.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:03 smithi116 ceph-mon[35411]: pgmap v8343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:05.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:05 smithi114 ceph-mon[30820]: pgmap v8344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:05.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:05 smithi116 ceph-mon[35411]: pgmap v8344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:07.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:07 smithi114 ceph-mon[30820]: pgmap v8345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:06:07.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:07 smithi116 ceph-mon[35411]: pgmap v8345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:06:09.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:09 smithi114 ceph-mon[30820]: pgmap v8346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:09.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:09 smithi116 ceph-mon[35411]: pgmap v8346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:11 smithi114 ceph-mon[30820]: pgmap v8347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:11 smithi116 ceph-mon[35411]: pgmap v8347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:13.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:13 smithi114 ceph-mon[30820]: pgmap v8348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:06:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:13 smithi116 ceph-mon[35411]: pgmap v8348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:06:15.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-mon[30820]: pgmap v8349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:06:15.508+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:06:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:06:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:06:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:06:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:06:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:06:15.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:06:15.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:06:15.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:06:15.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:06:15.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:06:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:06:15.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:15 smithi116 ceph-mon[35411]: pgmap v8349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:16 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:06:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:06:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:06:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:06:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:06:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:06:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:16 smithi114 ceph-mon[30820]: pgmap v8350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T11:06:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:16 smithi114 ceph-mon[30820]: pgmap v8351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:16 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:06:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:06:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:06:16.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:16 smithi116 ceph-mon[35411]: pgmap v8350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T11:06:16.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:16 smithi116 ceph-mon[35411]: pgmap v8351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:16.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:06:18.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:18 smithi114 ceph-mon[30820]: pgmap v8352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:18.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:18 smithi116 ceph-mon[35411]: pgmap v8352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:20.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:20 smithi114 ceph-mon[30820]: pgmap v8353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:20.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:20 smithi116 ceph-mon[35411]: pgmap v8353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:22.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:22 smithi114 ceph-mon[30820]: pgmap v8354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:22 smithi116 ceph-mon[35411]: pgmap v8354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:24.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:24 smithi114 ceph-mon[30820]: pgmap v8355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:24.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:24 smithi116 ceph-mon[35411]: pgmap v8355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:06:26.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:26 smithi114 ceph-mon[30820]: pgmap v8356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:06:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:26 smithi116 ceph-mon[35411]: pgmap v8356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:06:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:28 smithi114 ceph-mon[30820]: pgmap v8357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:28.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:28 smithi116 ceph-mon[35411]: pgmap v8357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:30.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:30 smithi114 ceph-mon[30820]: pgmap v8358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:30.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:30 smithi116 ceph-mon[35411]: pgmap v8358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:32 smithi114 ceph-mon[30820]: pgmap v8359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:32 smithi116 ceph-mon[35411]: pgmap v8359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:34 smithi114 ceph-mon[30820]: pgmap v8360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:34.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:34 smithi116 ceph-mon[35411]: pgmap v8360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:36.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:36 smithi114 ceph-mon[30820]: pgmap v8361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:06:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:36 smithi116 ceph-mon[35411]: pgmap v8361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:06:38.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:38 smithi114 ceph-mon[30820]: pgmap v8362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:38.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:38 smithi116 ceph-mon[35411]: pgmap v8362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:40.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:40 smithi114 ceph-mon[30820]: pgmap v8363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:40 smithi116 ceph-mon[35411]: pgmap v8363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:42 smithi114 ceph-mon[30820]: pgmap v8364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:42 smithi116 ceph-mon[35411]: pgmap v8364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:44 smithi114 ceph-mon[30820]: pgmap v8365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:44 smithi116 ceph-mon[35411]: pgmap v8365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:46 smithi114 ceph-mon[30820]: pgmap v8366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:06:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:46 smithi116 ceph-mon[35411]: pgmap v8366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:06:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:48 smithi114 ceph-mon[30820]: pgmap v8367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:48 smithi116 ceph-mon[35411]: pgmap v8367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:50.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:50 smithi114 ceph-mon[30820]: pgmap v8368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:50 smithi116 ceph-mon[35411]: pgmap v8368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:06:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:52 smithi114 ceph-mon[30820]: pgmap v8369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:52 smithi116 ceph-mon[35411]: pgmap v8369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:06:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:54 smithi114 ceph-mon[30820]: pgmap v8370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:54.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:54 smithi116 ceph-mon[35411]: pgmap v8370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:56.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:56 smithi114 ceph-mon[30820]: pgmap v8371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:06:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:56 smithi116 ceph-mon[35411]: pgmap v8371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:06:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:06:58 smithi114 ceph-mon[30820]: pgmap v8372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:06:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:06:58 smithi116 ceph-mon[35411]: pgmap v8372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:00 smithi114 ceph-mon[30820]: pgmap v8373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:00 smithi116 ceph-mon[35411]: pgmap v8373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:02 smithi114 ceph-mon[30820]: pgmap v8374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:02 smithi116 ceph-mon[35411]: pgmap v8374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:04 smithi114 ceph-mon[30820]: pgmap v8375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:04 smithi116 ceph-mon[35411]: pgmap v8375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:07.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:06 smithi114 ceph-mon[30820]: pgmap v8376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:06 smithi116 ceph-mon[35411]: pgmap v8376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:08 smithi114 ceph-mon[30820]: pgmap v8377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:08 smithi116 ceph-mon[35411]: pgmap v8377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:10 smithi114 ceph-mon[30820]: pgmap v8378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:10 smithi116 ceph-mon[35411]: pgmap v8378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:12 smithi114 ceph-mon[30820]: pgmap v8379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:12 smithi116 ceph-mon[35411]: pgmap v8379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:14 smithi114 ceph-mon[30820]: pgmap v8380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:15.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:14 smithi116 ceph-mon[35411]: pgmap v8380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:07:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:07:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:16 smithi114 ceph-mon[30820]: pgmap v8381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:16 smithi116 ceph-mon[35411]: pgmap v8381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-mon[30820]: pgmap v8382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:19.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:07:18.868+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:07:19.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:07:19.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:07:19.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:07:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:07:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:07:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:07:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:07:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:07:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:07:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:07:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:07:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:07:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:07:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:18 smithi116 ceph-mon[35411]: pgmap v8382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:07:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:07:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:07:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:19 smithi114 ceph-mon[30820]: pgmap v8383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:07:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:19 smithi114 ceph-mon[30820]: pgmap v8384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:07:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:07:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:07:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:07:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:19 smithi116 ceph-mon[35411]: pgmap v8383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:07:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:19 smithi116 ceph-mon[35411]: pgmap v8384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:07:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:07:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:21 smithi114 ceph-mon[30820]: pgmap v8385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:07:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:21 smithi116 ceph-mon[35411]: pgmap v8385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:07:24.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:23 smithi114 ceph-mon[30820]: pgmap v8386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:07:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:23 smithi116 ceph-mon[35411]: pgmap v8386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:07:26.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:25 smithi114 ceph-mon[30820]: pgmap v8387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:07:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:25 smithi116 ceph-mon[35411]: pgmap v8387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:07:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:27 smithi114 ceph-mon[30820]: pgmap v8388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:07:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:27 smithi116 ceph-mon[35411]: pgmap v8388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:07:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:29 smithi114 ceph-mon[30820]: pgmap v8389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:07:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:29 smithi116 ceph-mon[35411]: pgmap v8389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:07:32.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:31 smithi116 ceph-mon[35411]: pgmap v8390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:31 smithi114 ceph-mon[30820]: pgmap v8390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:33 smithi114 ceph-mon[30820]: pgmap v8391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:33 smithi116 ceph-mon[35411]: pgmap v8391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:35 smithi114 ceph-mon[30820]: pgmap v8392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:35 smithi116 ceph-mon[35411]: pgmap v8392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:37 smithi114 ceph-mon[30820]: pgmap v8393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:37 smithi116 ceph-mon[35411]: pgmap v8393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:39 smithi114 ceph-mon[30820]: pgmap v8394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:39 smithi116 ceph-mon[35411]: pgmap v8394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:41 smithi114 ceph-mon[30820]: pgmap v8395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:41 smithi116 ceph-mon[35411]: pgmap v8395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:43 smithi114 ceph-mon[30820]: pgmap v8396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:43 smithi116 ceph-mon[35411]: pgmap v8396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:45 smithi114 ceph-mon[30820]: pgmap v8397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:45 smithi116 ceph-mon[35411]: pgmap v8397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:47 smithi114 ceph-mon[30820]: pgmap v8398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:47 smithi116 ceph-mon[35411]: pgmap v8398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:49 smithi114 ceph-mon[30820]: pgmap v8399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:49 smithi116 ceph-mon[35411]: pgmap v8399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:51 smithi114 ceph-mon[30820]: pgmap v8400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:51 smithi116 ceph-mon[35411]: pgmap v8400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:07:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:53 smithi114 ceph-mon[30820]: pgmap v8401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:53 smithi116 ceph-mon[35411]: pgmap v8401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:07:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:55 smithi114 ceph-mon[30820]: pgmap v8402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:55 smithi116 ceph-mon[35411]: pgmap v8402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:07:58.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:07:57 smithi114 ceph-mon[30820]: pgmap v8403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:07:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:07:57 smithi116 ceph-mon[35411]: pgmap v8403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:00 smithi114 ceph-mon[30820]: pgmap v8404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:00 smithi116 ceph-mon[35411]: pgmap v8404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:02 smithi114 ceph-mon[30820]: pgmap v8405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:08:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:02 smithi116 ceph-mon[35411]: pgmap v8405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:08:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:04 smithi114 ceph-mon[30820]: pgmap v8406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:04 smithi116 ceph-mon[35411]: pgmap v8406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:06 smithi114 ceph-mon[30820]: pgmap v8407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:06 smithi116 ceph-mon[35411]: pgmap v8407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:08 smithi114 ceph-mon[30820]: pgmap v8408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:08.496 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:08 smithi116 ceph-mon[35411]: pgmap v8408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:10 smithi114 ceph-mon[30820]: pgmap v8409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:10 smithi116 ceph-mon[35411]: pgmap v8409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:12 smithi114 ceph-mon[30820]: pgmap v8410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:08:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:12 smithi116 ceph-mon[35411]: pgmap v8410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:08:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:14 smithi114 ceph-mon[30820]: pgmap v8411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:14 smithi116 ceph-mon[35411]: pgmap v8411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:16 smithi114 ceph-mon[30820]: pgmap v8412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:16 smithi116 ceph-mon[35411]: pgmap v8412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:18 smithi114 ceph-mon[30820]: pgmap v8413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:18 smithi116 ceph-mon[35411]: pgmap v8413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:08:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:08:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:20 smithi114 ceph-mon[30820]: pgmap v8414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:20 smithi116 ceph-mon[35411]: pgmap v8414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-mon[30820]: pgmap v8415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:08:22.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:08:22.228+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:08:22.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:08:22.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:08:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:08:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:08:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:08:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:08:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:08:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:08:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:08:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:08:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:22 smithi116 ceph-mon[35411]: pgmap v8415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:08:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:08:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:08:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:08:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:08:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:08:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:08:24.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:08:24.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:08:24.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:24 smithi114 ceph-mon[30820]: pgmap v8416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:08:24.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:24 smithi114 ceph-mon[30820]: pgmap v8417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:08:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:24 smithi116 ceph-mon[35411]: pgmap v8416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:08:24.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:24 smithi116 ceph-mon[35411]: pgmap v8417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:08:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:26 smithi114 ceph-mon[30820]: pgmap v8418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:08:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:26 smithi116 ceph-mon[35411]: pgmap v8418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:08:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:28 smithi114 ceph-mon[30820]: pgmap v8419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:08:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:28 smithi116 ceph-mon[35411]: pgmap v8419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:08:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:30 smithi114 ceph-mon[30820]: pgmap v8420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:08:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:30 smithi116 ceph-mon[35411]: pgmap v8420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:08:32.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:32 smithi114 ceph-mon[30820]: pgmap v8421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:08:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:32 smithi116 ceph-mon[35411]: pgmap v8421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:08:34.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:34 smithi114 ceph-mon[30820]: pgmap v8422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:08:34.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:34 smithi116 ceph-mon[35411]: pgmap v8422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:08:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:36 smithi114 ceph-mon[30820]: pgmap v8423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:36 smithi116 ceph-mon[35411]: pgmap v8423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:38 smithi114 ceph-mon[30820]: pgmap v8424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:38 smithi116 ceph-mon[35411]: pgmap v8424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:40 smithi114 ceph-mon[30820]: pgmap v8425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:40 smithi116 ceph-mon[35411]: pgmap v8425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:42 smithi114 ceph-mon[30820]: pgmap v8426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:42 smithi116 ceph-mon[35411]: pgmap v8426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:44 smithi114 ceph-mon[30820]: pgmap v8427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:44 smithi116 ceph-mon[35411]: pgmap v8427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:46 smithi114 ceph-mon[30820]: pgmap v8428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:46 smithi116 ceph-mon[35411]: pgmap v8428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:48 smithi114 ceph-mon[30820]: pgmap v8429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:48 smithi116 ceph-mon[35411]: pgmap v8429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:50 smithi114 ceph-mon[30820]: pgmap v8430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:50 smithi116 ceph-mon[35411]: pgmap v8430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:08:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:52 smithi114 ceph-mon[30820]: pgmap v8431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:52 smithi116 ceph-mon[35411]: pgmap v8431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:54 smithi114 ceph-mon[30820]: pgmap v8432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:54 smithi116 ceph-mon[35411]: pgmap v8432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:56 smithi114 ceph-mon[30820]: pgmap v8433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:56 smithi116 ceph-mon[35411]: pgmap v8433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:08:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:08:58 smithi114 ceph-mon[30820]: pgmap v8434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:08:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:08:58 smithi116 ceph-mon[35411]: pgmap v8434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:00 smithi114 ceph-mon[30820]: pgmap v8435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:00 smithi116 ceph-mon[35411]: pgmap v8435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:02 smithi114 ceph-mon[30820]: pgmap v8436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:02 smithi116 ceph-mon[35411]: pgmap v8436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:04 smithi114 ceph-mon[30820]: pgmap v8437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:04 smithi116 ceph-mon[35411]: pgmap v8437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:06 smithi114 ceph-mon[30820]: pgmap v8438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:06 smithi116 ceph-mon[35411]: pgmap v8438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:08 smithi114 ceph-mon[30820]: pgmap v8439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:08 smithi116 ceph-mon[35411]: pgmap v8439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:10 smithi114 ceph-mon[30820]: pgmap v8440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:10 smithi116 ceph-mon[35411]: pgmap v8440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:12 smithi114 ceph-mon[30820]: pgmap v8441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:12 smithi116 ceph-mon[35411]: pgmap v8441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:14.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:14 smithi114 ceph-mon[30820]: pgmap v8442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:14 smithi116 ceph-mon[35411]: pgmap v8442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:16 smithi114 ceph-mon[30820]: pgmap v8443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:16 smithi116 ceph-mon[35411]: pgmap v8443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:18 smithi114 ceph-mon[30820]: pgmap v8444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:18 smithi116 ceph-mon[35411]: pgmap v8444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:20 smithi114 ceph-mon[30820]: pgmap v8445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:20 smithi116 ceph-mon[35411]: pgmap v8445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:22 smithi114 ceph-mon[30820]: pgmap v8446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:22 smithi116 ceph-mon[35411]: pgmap v8446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:23.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:09:23.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:09:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:24 smithi116 ceph-mon[35411]: pgmap v8447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:24 smithi114 ceph-mon[30820]: pgmap v8447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:09:25.588+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:09:25.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:09:25.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:09:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:09:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:26 smithi116 ceph-mon[35411]: pgmap v8448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:09:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:09:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:09:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:26 smithi114 ceph-mon[30820]: pgmap v8448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:09:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:09:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:09:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:09:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:27 smithi114 ceph-mon[30820]: pgmap v8449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:09:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:27 smithi114 ceph-mon[30820]: pgmap v8450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:09:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:27 smithi116 ceph-mon[35411]: pgmap v8449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:09:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:27 smithi116 ceph-mon[35411]: pgmap v8450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:09:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:29 smithi114 ceph-mon[30820]: pgmap v8451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:09:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:29 smithi116 ceph-mon[35411]: pgmap v8451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:09:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:31 smithi114 ceph-mon[30820]: pgmap v8452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:09:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:31 smithi116 ceph-mon[35411]: pgmap v8452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:09:33.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:33 smithi116 ceph-mon[35411]: pgmap v8453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:09:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:33 smithi114 ceph-mon[30820]: pgmap v8453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:09:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:35 smithi114 ceph-mon[30820]: pgmap v8454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:09:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:35 smithi116 ceph-mon[35411]: pgmap v8454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:09:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:37 smithi114 ceph-mon[30820]: pgmap v8455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:09:37.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:37 smithi116 ceph-mon[35411]: pgmap v8455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:09:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:39 smithi114 ceph-mon[30820]: pgmap v8456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:39 smithi116 ceph-mon[35411]: pgmap v8456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:41 smithi114 ceph-mon[30820]: pgmap v8457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:41 smithi116 ceph-mon[35411]: pgmap v8457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:43.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:43 smithi114 ceph-mon[30820]: pgmap v8458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:43 smithi116 ceph-mon[35411]: pgmap v8458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:44.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:44 smithi114 ceph-mon[30820]: pgmap v8459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:44 smithi116 ceph-mon[35411]: pgmap v8459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:46.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:46 smithi114 ceph-mon[30820]: pgmap v8460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:09:46.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:46 smithi116 ceph-mon[35411]: pgmap v8460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:09:48.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:48 smithi114 ceph-mon[30820]: pgmap v8461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:48.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:48 smithi116 ceph-mon[35411]: pgmap v8461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:50 smithi114 ceph-mon[30820]: pgmap v8462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:50.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:50 smithi116 ceph-mon[35411]: pgmap v8462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:09:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:52 smithi116 ceph-mon[35411]: pgmap v8463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:52 smithi114 ceph-mon[30820]: pgmap v8463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:09:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:54 smithi116 ceph-mon[35411]: pgmap v8464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:55.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:54 smithi114 ceph-mon[30820]: pgmap v8464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:56 smithi116 ceph-mon[35411]: pgmap v8465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:09:56.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:56 smithi114 ceph-mon[30820]: pgmap v8465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:09:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:09:58 smithi114 ceph-mon[30820]: pgmap v8466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:09:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:09:58 smithi116 ceph-mon[35411]: pgmap v8466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:00 smithi114 ceph-mon[30820]: pgmap v8467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:10:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:00 smithi116 ceph-mon[35411]: pgmap v8467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:10:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:02 smithi114 ceph-mon[30820]: pgmap v8468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:02 smithi116 ceph-mon[35411]: pgmap v8468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:04.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:04 smithi116 ceph-mon[35411]: pgmap v8469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:05.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:04 smithi114 ceph-mon[30820]: pgmap v8469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:06 smithi114 ceph-mon[30820]: pgmap v8470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:06 smithi116 ceph-mon[35411]: pgmap v8470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:09.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:08 smithi114 ceph-mon[30820]: pgmap v8471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:08 smithi116 ceph-mon[35411]: pgmap v8471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:10 smithi114 ceph-mon[30820]: pgmap v8472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:10 smithi116 ceph-mon[35411]: pgmap v8472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:12 smithi114 ceph-mon[30820]: pgmap v8473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:12 smithi116 ceph-mon[35411]: pgmap v8473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:14 smithi114 ceph-mon[30820]: pgmap v8474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:14 smithi116 ceph-mon[35411]: pgmap v8474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:16 smithi114 ceph-mon[30820]: pgmap v8475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:16 smithi116 ceph-mon[35411]: pgmap v8475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:18 smithi114 ceph-mon[30820]: pgmap v8476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:18 smithi116 ceph-mon[35411]: pgmap v8476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:21.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:20 smithi114 ceph-mon[30820]: pgmap v8477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:20 smithi116 ceph-mon[35411]: pgmap v8477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:22 smithi114 ceph-mon[30820]: pgmap v8478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:23.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:22 smithi116 ceph-mon[35411]: pgmap v8478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:24 smithi114 ceph-mon[30820]: pgmap v8479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:24 smithi116 ceph-mon[35411]: pgmap v8479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:26.067 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:10:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:10:27.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:26 smithi114 ceph-mon[30820]: pgmap v8480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:26 smithi116 ceph-mon[35411]: pgmap v8480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-mon[30820]: pgmap v8481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:29.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:10:28.949+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:10:29.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:10:29.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:10:29.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:10:29.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:10:29.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:10:29.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:10:29.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:10:29.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:10:29.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:10:29.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:10:29.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:10:29.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:10:29.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:10:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:10:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:28 smithi116 ceph-mon[35411]: pgmap v8481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:29 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:10:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:10:30.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:29 smithi114 ceph-mon[30820]: pgmap v8482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:10:30.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:29 smithi114 ceph-mon[30820]: pgmap v8483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:10:30.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:29 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:10:30.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:10:30.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:10:30.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:10:30.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:10:30.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:10:30.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:10:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:10:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:10:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:10:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:10:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:10:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:10:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:10:30.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:10:30.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:10:30.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:10:30.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:10:30.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:29 smithi116 ceph-mon[35411]: pgmap v8482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:10:30.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:29 smithi116 ceph-mon[35411]: pgmap v8483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:10:30.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:10:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:32 smithi114 ceph-mon[30820]: pgmap v8484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:10:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:32 smithi116 ceph-mon[35411]: pgmap v8484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:10:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:34 smithi114 ceph-mon[30820]: pgmap v8485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:10:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:34 smithi116 ceph-mon[35411]: pgmap v8485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:10:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:36 smithi114 ceph-mon[30820]: pgmap v8486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:10:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:36 smithi116 ceph-mon[35411]: pgmap v8486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:10:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:38 smithi114 ceph-mon[30820]: pgmap v8487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:10:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:38 smithi116 ceph-mon[35411]: pgmap v8487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:10:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:40 smithi114 ceph-mon[30820]: pgmap v8488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:10:40.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:40 smithi116 ceph-mon[35411]: pgmap v8488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:10:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:42 smithi114 ceph-mon[30820]: pgmap v8489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:42 smithi116 ceph-mon[35411]: pgmap v8489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:44 smithi114 ceph-mon[30820]: pgmap v8490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:44.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:44 smithi116 ceph-mon[35411]: pgmap v8490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:46 smithi114 ceph-mon[30820]: pgmap v8491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:46 smithi116 ceph-mon[35411]: pgmap v8491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:48 smithi114 ceph-mon[30820]: pgmap v8492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:48 smithi116 ceph-mon[35411]: pgmap v8492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:50.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:50 smithi114 ceph-mon[30820]: pgmap v8493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:50 smithi116 ceph-mon[35411]: pgmap v8493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:52 smithi114 ceph-mon[30820]: pgmap v8494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:52 smithi116 ceph-mon[35411]: pgmap v8494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:54 smithi114 ceph-mon[30820]: pgmap v8495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:54 smithi116 ceph-mon[35411]: pgmap v8495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:10:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:56 smithi114 ceph-mon[30820]: pgmap v8496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:56 smithi116 ceph-mon[35411]: pgmap v8496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:10:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:10:58 smithi114 ceph-mon[30820]: pgmap v8497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:10:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:10:58 smithi116 ceph-mon[35411]: pgmap v8497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:00 smithi114 ceph-mon[30820]: pgmap v8498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:00 smithi116 ceph-mon[35411]: pgmap v8498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:02 smithi114 ceph-mon[30820]: pgmap v8499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:11:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:02 smithi116 ceph-mon[35411]: pgmap v8499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:11:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:04 smithi114 ceph-mon[30820]: pgmap v8500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:04.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:04 smithi116 ceph-mon[35411]: pgmap v8500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:06 smithi114 ceph-mon[30820]: pgmap v8501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:06 smithi116 ceph-mon[35411]: pgmap v8501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:08.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:08 smithi114 ceph-mon[30820]: pgmap v8502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:08.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:08 smithi116 ceph-mon[35411]: pgmap v8502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:10 smithi114 ceph-mon[30820]: pgmap v8503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:10 smithi116 ceph-mon[35411]: pgmap v8503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:12.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:12 smithi114 ceph-mon[30820]: pgmap v8504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:11:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:12 smithi116 ceph-mon[35411]: pgmap v8504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:11:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:14 smithi114 ceph-mon[30820]: pgmap v8505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:14 smithi116 ceph-mon[35411]: pgmap v8505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:16 smithi114 ceph-mon[30820]: pgmap v8506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:16 smithi116 ceph-mon[35411]: pgmap v8506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:18 smithi114 ceph-mon[30820]: pgmap v8507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:18.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:18 smithi116 ceph-mon[35411]: pgmap v8507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:20 smithi114 ceph-mon[30820]: pgmap v8508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:20 smithi116 ceph-mon[35411]: pgmap v8508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:22.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:22 smithi114 ceph-mon[30820]: pgmap v8509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:11:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:22 smithi116 ceph-mon[35411]: pgmap v8509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:11:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:24 smithi114 ceph-mon[30820]: pgmap v8510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:24.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:24 smithi116 ceph-mon[35411]: pgmap v8510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:26.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:26 smithi114 ceph-mon[30820]: pgmap v8511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:26.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:26 smithi116 ceph-mon[35411]: pgmap v8511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:28.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:28 smithi114 ceph-mon[30820]: pgmap v8512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:28 smithi116 ceph-mon[35411]: pgmap v8512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:29.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:11:29.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:11:30.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: pgmap v8513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: pgmap v8513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:11:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:31 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:11:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:31 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:11:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:31 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:11:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:31 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:11:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:31 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:11:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:31 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:11:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:31 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:11:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:31 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:11:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-mon[30820]: pgmap v8514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:11:32.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:11:32.309+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:11:32.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:11:32.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:11:32.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:11:32.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:11:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:11:32.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:32 smithi116 ceph-mon[35411]: pgmap v8514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:11:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:11:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:11:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:11:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:11:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:11:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:11:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:11:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:11:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:11:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:11:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:11:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:11:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:11:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:11:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:11:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:11:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:34 smithi114 ceph-mon[30820]: pgmap v8515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:11:34.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:34 smithi114 ceph-mon[30820]: pgmap v8516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:11:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:34 smithi116 ceph-mon[35411]: pgmap v8515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:11:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:34 smithi116 ceph-mon[35411]: pgmap v8516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:11:36.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:36 smithi114 ceph-mon[30820]: pgmap v8517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:11:36.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:36 smithi116 ceph-mon[35411]: pgmap v8517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:11:38.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:38 smithi114 ceph-mon[30820]: pgmap v8518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:11:38.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:38 smithi116 ceph-mon[35411]: pgmap v8518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:11:40.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:40 smithi114 ceph-mon[30820]: pgmap v8519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:11:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:40 smithi116 ceph-mon[35411]: pgmap v8519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:11:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:42 smithi114 ceph-mon[30820]: pgmap v8520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:11:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:42 smithi116 ceph-mon[35411]: pgmap v8520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:11:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:44 smithi114 ceph-mon[30820]: pgmap v8521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:11:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:44 smithi116 ceph-mon[35411]: pgmap v8521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:11:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:46 smithi114 ceph-mon[30820]: pgmap v8522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:46 smithi116 ceph-mon[35411]: pgmap v8522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:48 smithi116 ceph-mon[35411]: pgmap v8523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:48.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:48 smithi114 ceph-mon[30820]: pgmap v8523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:50 smithi116 ceph-mon[35411]: pgmap v8524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:50.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:50 smithi114 ceph-mon[30820]: pgmap v8524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:11:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:52 smithi116 ceph-mon[35411]: pgmap v8525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:11:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:52 smithi114 ceph-mon[30820]: pgmap v8525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:11:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:54 smithi116 ceph-mon[35411]: pgmap v8526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:11:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:54 smithi114 ceph-mon[30820]: pgmap v8526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:11:56.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:56 smithi114 ceph-mon[30820]: pgmap v8527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:56 smithi116 ceph-mon[35411]: pgmap v8527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:11:58.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:11:58 smithi114 ceph-mon[30820]: pgmap v8528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:11:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:11:58 smithi116 ceph-mon[35411]: pgmap v8528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:00 smithi114 ceph-mon[30820]: pgmap v8529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:00 smithi116 ceph-mon[35411]: pgmap v8529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:02 smithi114 ceph-mon[30820]: pgmap v8530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:12:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:02 smithi116 ceph-mon[35411]: pgmap v8530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:12:04.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:04 smithi116 ceph-mon[35411]: pgmap v8531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:12:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:04 smithi114 ceph-mon[30820]: pgmap v8531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:12:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:06 smithi114 ceph-mon[30820]: pgmap v8532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:06 smithi116 ceph-mon[35411]: pgmap v8532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:08 smithi114 ceph-mon[30820]: pgmap v8533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:08 smithi116 ceph-mon[35411]: pgmap v8533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:10 smithi114 ceph-mon[30820]: pgmap v8534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:10 smithi116 ceph-mon[35411]: pgmap v8534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:12 smithi114 ceph-mon[30820]: pgmap v8535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:12:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:12 smithi116 ceph-mon[35411]: pgmap v8535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:12:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:14 smithi114 ceph-mon[30820]: pgmap v8536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:12:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:14 smithi116 ceph-mon[35411]: pgmap v8536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:12:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:16 smithi114 ceph-mon[30820]: pgmap v8537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:16 smithi116 ceph-mon[35411]: pgmap v8537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:18 smithi114 ceph-mon[30820]: pgmap v8538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:18 smithi116 ceph-mon[35411]: pgmap v8538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:20.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:20 smithi114 ceph-mon[30820]: pgmap v8539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:20 smithi116 ceph-mon[35411]: pgmap v8539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:22 smithi114 ceph-mon[30820]: pgmap v8540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:12:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:22 smithi116 ceph-mon[35411]: pgmap v8540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:12:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:24 smithi114 ceph-mon[30820]: pgmap v8541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:12:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:24 smithi116 ceph-mon[35411]: pgmap v8541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:12:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:26 smithi114 ceph-mon[30820]: pgmap v8542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:26 smithi116 ceph-mon[35411]: pgmap v8542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:27 smithi114 ceph-mon[30820]: pgmap v8543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:27 smithi116 ceph-mon[35411]: pgmap v8543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:29 smithi114 ceph-mon[30820]: pgmap v8544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:29.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:29 smithi116 ceph-mon[35411]: pgmap v8544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:31.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:31 smithi114 ceph-mon[30820]: pgmap v8545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T11:12:31.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:31 smithi116 ceph-mon[35411]: pgmap v8545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T11:12:33.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:33 smithi114 ceph-mon[30820]: pgmap v8546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:12:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:12:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:33 smithi116 ceph-mon[35411]: pgmap v8546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:12:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:12:35.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-mon[30820]: pgmap v8547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:35.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:35 smithi116 ceph-mon[35411]: pgmap v8547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:12:35.668+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:12:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:12:36.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:12:36.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:12:36.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:12:36.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:12:36.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:12:36.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:12:36.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:12:36.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:12:36.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:12:36.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:12:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:12:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:12:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:12:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:12:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:36 smithi114 ceph-mon[30820]: pgmap v8548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:12:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:36 smithi114 ceph-mon[30820]: pgmap v8549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:12:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:12:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:36 smithi116 ceph-mon[35411]: pgmap v8548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:36 smithi116 ceph-mon[35411]: pgmap v8549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:12:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:12:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:38 smithi114 ceph-mon[30820]: pgmap v8550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:12:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:38 smithi116 ceph-mon[35411]: pgmap v8550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:12:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:40 smithi114 ceph-mon[30820]: pgmap v8551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:12:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:40 smithi116 ceph-mon[35411]: pgmap v8551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:12:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:42 smithi114 ceph-mon[30820]: pgmap v8552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:12:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:42 smithi116 ceph-mon[35411]: pgmap v8552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:12:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:44 smithi114 ceph-mon[30820]: pgmap v8553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:12:45.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:44 smithi116 ceph-mon[35411]: pgmap v8553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:12:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:46 smithi114 ceph-mon[30820]: pgmap v8554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:12:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:46 smithi116 ceph-mon[35411]: pgmap v8554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:12:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:48 smithi114 ceph-mon[30820]: pgmap v8555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:49.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:48 smithi116 ceph-mon[35411]: pgmap v8555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:51.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:50 smithi114 ceph-mon[30820]: pgmap v8556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:51.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:50 smithi116 ceph-mon[35411]: pgmap v8556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:12:53.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:52 smithi114 ceph-mon[30820]: pgmap v8557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:53.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:52 smithi116 ceph-mon[35411]: pgmap v8557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:12:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:54 smithi114 ceph-mon[30820]: pgmap v8558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:55.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:54 smithi116 ceph-mon[35411]: pgmap v8558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:57.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:56 smithi114 ceph-mon[30820]: pgmap v8559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:12:57.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:56 smithi116 ceph-mon[35411]: pgmap v8559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:12:59.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:12:58 smithi114 ceph-mon[30820]: pgmap v8560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:12:59.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:12:58 smithi116 ceph-mon[35411]: pgmap v8560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:00 smithi114 ceph-mon[30820]: pgmap v8561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:13:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:00 smithi116 ceph-mon[35411]: pgmap v8561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:13:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:02 smithi114 ceph-mon[30820]: pgmap v8562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:02 smithi116 ceph-mon[35411]: pgmap v8562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:04 smithi114 ceph-mon[30820]: pgmap v8563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:04 smithi116 ceph-mon[35411]: pgmap v8563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:06 smithi114 ceph-mon[30820]: pgmap v8564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:07.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:06 smithi116 ceph-mon[35411]: pgmap v8564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:08 smithi114 ceph-mon[30820]: pgmap v8565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:08 smithi116 ceph-mon[35411]: pgmap v8565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:10 smithi114 ceph-mon[30820]: pgmap v8566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:13:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:10 smithi116 ceph-mon[35411]: pgmap v8566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:13:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:12 smithi114 ceph-mon[30820]: pgmap v8567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:12 smithi116 ceph-mon[35411]: pgmap v8567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:15.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:14 smithi114 ceph-mon[30820]: pgmap v8568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:13:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:14 smithi116 ceph-mon[35411]: pgmap v8568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:13:17.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:16 smithi114 ceph-mon[30820]: pgmap v8569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:17.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:16 smithi116 ceph-mon[35411]: pgmap v8569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:18 smithi114 ceph-mon[30820]: pgmap v8570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:13:19.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:18 smithi116 ceph-mon[35411]: pgmap v8570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:13:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:20 smithi114 ceph-mon[30820]: pgmap v8571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:13:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:20 smithi116 ceph-mon[35411]: pgmap v8571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:13:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:22 smithi114 ceph-mon[30820]: pgmap v8572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:22 smithi116 ceph-mon[35411]: pgmap v8572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:25.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:24 smithi114 ceph-mon[30820]: pgmap v8573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:25.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:24 smithi116 ceph-mon[35411]: pgmap v8573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:26 smithi114 ceph-mon[30820]: pgmap v8574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:27.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:26 smithi116 ceph-mon[35411]: pgmap v8574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:28 smithi114 ceph-mon[30820]: pgmap v8575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:28 smithi116 ceph-mon[35411]: pgmap v8575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:31.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:30 smithi114 ceph-mon[30820]: pgmap v8576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:13:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:30 smithi116 ceph-mon[35411]: pgmap v8576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:13:33.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:32 smithi114 ceph-mon[30820]: pgmap v8577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:32 smithi116 ceph-mon[35411]: pgmap v8577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:35.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:34 smithi114 ceph-mon[30820]: pgmap v8578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:34 smithi116 ceph-mon[35411]: pgmap v8578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:36.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:13:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:13:37.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:36 smithi114 ceph-mon[30820]: pgmap v8579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:37.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:36 smithi116 ceph-mon[35411]: pgmap v8579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:38 smithi114 ceph-mon[30820]: pgmap v8580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:13:39.028+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:13:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:13:39.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:13:39.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:13:39.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:13:39.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:13:39.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:13:39.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:13:39.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:13:39.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:13:39.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:13:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:38 smithi116 ceph-mon[35411]: pgmap v8580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:13:40.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:13:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:13:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:13:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:13:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:13:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-mon[30820]: pgmap v8581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:13:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-mon[30820]: pgmap v8582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:13:40.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:13:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:13:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:13:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:13:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:13:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:13:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:13:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:13:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:13:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:13:40.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:13:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:13:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:13:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:13:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:13:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:39 smithi116 ceph-mon[35411]: pgmap v8581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:13:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:39 smithi116 ceph-mon[35411]: pgmap v8582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:13:40.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:13:42.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:42 smithi114 ceph-mon[30820]: pgmap v8583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:13:42.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:42 smithi116 ceph-mon[35411]: pgmap v8583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:13:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:44 smithi114 ceph-mon[30820]: pgmap v8584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:13:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:44 smithi116 ceph-mon[35411]: pgmap v8584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:13:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:46 smithi114 ceph-mon[30820]: pgmap v8585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:13:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:46 smithi116 ceph-mon[35411]: pgmap v8585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:13:48.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:48 smithi114 ceph-mon[30820]: pgmap v8586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:13:48.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:48 smithi116 ceph-mon[35411]: pgmap v8586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:13:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:50 smithi114 ceph-mon[30820]: pgmap v8587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:13:50.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:50 smithi116 ceph-mon[35411]: pgmap v8587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:13:52.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:52 smithi114 ceph-mon[30820]: pgmap v8588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:52 smithi116 ceph-mon[35411]: pgmap v8588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:13:54.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:54 smithi114 ceph-mon[30820]: pgmap v8589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:54.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:54 smithi116 ceph-mon[35411]: pgmap v8589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:13:56.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:56 smithi114 ceph-mon[30820]: pgmap v8590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:13:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:56 smithi116 ceph-mon[35411]: pgmap v8590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:13:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:13:58 smithi114 ceph-mon[30820]: pgmap v8591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:13:58.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:13:58 smithi116 ceph-mon[35411]: pgmap v8591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:00.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:00 smithi114 ceph-mon[30820]: pgmap v8592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:00.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:00 smithi116 ceph-mon[35411]: pgmap v8592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:02.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:02 smithi114 ceph-mon[30820]: pgmap v8593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:02.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:02 smithi116 ceph-mon[35411]: pgmap v8593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:04.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:04 smithi114 ceph-mon[30820]: pgmap v8594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:04 smithi116 ceph-mon[35411]: pgmap v8594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:06.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:06 smithi114 ceph-mon[30820]: pgmap v8595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:14:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:06 smithi116 ceph-mon[35411]: pgmap v8595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:14:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:08 smithi114 ceph-mon[30820]: pgmap v8596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:08 smithi116 ceph-mon[35411]: pgmap v8596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:10.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:10 smithi114 ceph-mon[30820]: pgmap v8597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:10.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:10 smithi116 ceph-mon[35411]: pgmap v8597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:12 smithi114 ceph-mon[30820]: pgmap v8598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:12.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:12 smithi116 ceph-mon[35411]: pgmap v8598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:14 smithi114 ceph-mon[30820]: pgmap v8599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:14.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:14 smithi116 ceph-mon[35411]: pgmap v8599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:16.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:16 smithi114 ceph-mon[30820]: pgmap v8600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:14:16.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:16 smithi116 ceph-mon[35411]: pgmap v8600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:14:18.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:18 smithi114 ceph-mon[30820]: pgmap v8601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:18 smithi116 ceph-mon[35411]: pgmap v8601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:20.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:20 smithi114 ceph-mon[30820]: pgmap v8602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:20.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:20 smithi116 ceph-mon[35411]: pgmap v8602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:22.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:22 smithi116 ceph-mon[35411]: pgmap v8603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:22 smithi114 ceph-mon[30820]: pgmap v8603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:24 smithi116 ceph-mon[35411]: pgmap v8604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:14:24.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:24 smithi114 ceph-mon[30820]: pgmap v8604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:14:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:26 smithi116 ceph-mon[35411]: pgmap v8605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:14:26.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:26 smithi114 ceph-mon[30820]: pgmap v8605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:14:28.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:28 smithi116 ceph-mon[35411]: pgmap v8606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:28 smithi114 ceph-mon[30820]: pgmap v8606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:30.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:30 smithi114 ceph-mon[30820]: pgmap v8607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:14:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:30 smithi116 ceph-mon[35411]: pgmap v8607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:14:32.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:32 smithi114 ceph-mon[30820]: pgmap v8608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:32 smithi116 ceph-mon[35411]: pgmap v8608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:34.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:34 smithi114 ceph-mon[30820]: pgmap v8609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:34 smithi116 ceph-mon[35411]: pgmap v8609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:36.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:36 smithi116 ceph-mon[35411]: pgmap v8610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:14:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:36 smithi114 ceph-mon[30820]: pgmap v8610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:14:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:38 smithi114 ceph-mon[30820]: pgmap v8611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:38 smithi116 ceph-mon[35411]: pgmap v8611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:39.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:14:39.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:14:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:40 smithi114 ceph-mon[30820]: pgmap v8612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:40 smithi116 ceph-mon[35411]: pgmap v8612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:14:41.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:41 smithi114 ceph-mon[30820]: pgmap v8613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:41 smithi116 ceph-mon[35411]: pgmap v8613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:14:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:14:42.388+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:14:42.661 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:14:42.662 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:14:42.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:14:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:14:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:14:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:14:42.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:14:43.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:43 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:14:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:43 smithi114 ceph-mon[30820]: pgmap v8614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:14:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:43 smithi114 ceph-mon[30820]: pgmap v8615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:43 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:14:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:14:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:14:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:43 smithi116 ceph-mon[35411]: pgmap v8614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:14:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:43 smithi116 ceph-mon[35411]: pgmap v8615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:14:45.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:45 smithi114 ceph-mon[30820]: pgmap v8616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:14:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:45 smithi116 ceph-mon[35411]: pgmap v8616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:14:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:47 smithi116 ceph-mon[35411]: pgmap v8617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:14:48.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:47 smithi114 ceph-mon[30820]: pgmap v8617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:14:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:49 smithi116 ceph-mon[35411]: pgmap v8618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:14:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:49 smithi114 ceph-mon[30820]: pgmap v8618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:14:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:51 smithi116 ceph-mon[35411]: pgmap v8619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:14:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:51 smithi114 ceph-mon[30820]: pgmap v8619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:14:54.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:53 smithi114 ceph-mon[30820]: pgmap v8620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:14:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:53 smithi116 ceph-mon[35411]: pgmap v8620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:14:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:55 smithi114 ceph-mon[30820]: pgmap v8621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:14:56.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:55 smithi116 ceph-mon[35411]: pgmap v8621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:14:57.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:57 smithi114 ceph-mon[30820]: pgmap v8622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:14:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:57 smithi116 ceph-mon[35411]: pgmap v8622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:14:59 smithi114 ceph-mon[30820]: pgmap v8623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:14:59 smithi116 ceph-mon[35411]: pgmap v8623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:02.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:01 smithi114 ceph-mon[30820]: pgmap v8624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:02.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:01 smithi116 ceph-mon[35411]: pgmap v8624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:03 smithi114 ceph-mon[30820]: pgmap v8625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:03 smithi116 ceph-mon[35411]: pgmap v8625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:05 smithi114 ceph-mon[30820]: pgmap v8626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:05 smithi116 ceph-mon[35411]: pgmap v8626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:07.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:07 smithi116 ceph-mon[35411]: pgmap v8627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:07 smithi114 ceph-mon[30820]: pgmap v8627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:09 smithi114 ceph-mon[30820]: pgmap v8628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:09 smithi116 ceph-mon[35411]: pgmap v8628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:11 smithi114 ceph-mon[30820]: pgmap v8629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:11 smithi116 ceph-mon[35411]: pgmap v8629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:14.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:13 smithi114 ceph-mon[30820]: pgmap v8630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:14.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:13 smithi116 ceph-mon[35411]: pgmap v8630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:16.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:15 smithi114 ceph-mon[30820]: pgmap v8631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:16.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:15 smithi116 ceph-mon[35411]: pgmap v8631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:17 smithi114 ceph-mon[30820]: pgmap v8632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:18.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:17 smithi116 ceph-mon[35411]: pgmap v8632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:19 smithi114 ceph-mon[30820]: pgmap v8633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:19 smithi116 ceph-mon[35411]: pgmap v8633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:22.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:21 smithi114 ceph-mon[30820]: pgmap v8634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:21 smithi116 ceph-mon[35411]: pgmap v8634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:24.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:23 smithi114 ceph-mon[30820]: pgmap v8635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:24.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:23 smithi116 ceph-mon[35411]: pgmap v8635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:26.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:25 smithi114 ceph-mon[30820]: pgmap v8636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:26.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:25 smithi116 ceph-mon[35411]: pgmap v8636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:27 smithi114 ceph-mon[30820]: pgmap v8637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:27 smithi116 ceph-mon[35411]: pgmap v8637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:29 smithi114 ceph-mon[30820]: pgmap v8638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:29 smithi116 ceph-mon[35411]: pgmap v8638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:31 smithi114 ceph-mon[30820]: pgmap v8639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:31 smithi116 ceph-mon[35411]: pgmap v8639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:34.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:33 smithi114 ceph-mon[30820]: pgmap v8640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:33 smithi116 ceph-mon[35411]: pgmap v8640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:35 smithi114 ceph-mon[30820]: pgmap v8641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:35 smithi116 ceph-mon[35411]: pgmap v8641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:38.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:37 smithi114 ceph-mon[30820]: pgmap v8642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:38.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:37 smithi116 ceph-mon[35411]: pgmap v8642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:15:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:39 smithi114 ceph-mon[30820]: pgmap v8643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:39 smithi116 ceph-mon[35411]: pgmap v8643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:42.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:41 smithi114 ceph-mon[30820]: pgmap v8644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:41 smithi116 ceph-mon[35411]: pgmap v8644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:15:43.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:15:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:15:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:43.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:44.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:43 smithi114 ceph-mon[30820]: pgmap v8645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:43 smithi116 ceph-mon[35411]: pgmap v8645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:15:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-mon[30820]: pgmap v8646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:15:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:15:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:46.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:15:45.748+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:15:46.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:15:46.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:15:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:15:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:45 smithi116 ceph-mon[35411]: pgmap v8646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:15:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:15:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:15:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:15:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:15:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:15:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:15:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:15:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:15:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:15:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:15:47.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:15:47.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:15:47.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:15:47.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:15:47.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:46 smithi114 ceph-mon[30820]: pgmap v8647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:15:47.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:46 smithi114 ceph-mon[30820]: pgmap v8648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:15:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:46 smithi116 ceph-mon[35411]: pgmap v8647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:15:47.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:46 smithi116 ceph-mon[35411]: pgmap v8648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:15:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:48 smithi114 ceph-mon[30820]: pgmap v8649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:15:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:48 smithi116 ceph-mon[35411]: pgmap v8649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:15:51.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:50 smithi114 ceph-mon[30820]: pgmap v8650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:15:51.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:50 smithi116 ceph-mon[35411]: pgmap v8650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:15:53.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:52 smithi114 ceph-mon[30820]: pgmap v8651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:15:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:52 smithi116 ceph-mon[35411]: pgmap v8651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:15:55.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:54 smithi114 ceph-mon[30820]: pgmap v8652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:15:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:54 smithi116 ceph-mon[35411]: pgmap v8652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:15:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:56 smithi114 ceph-mon[30820]: pgmap v8653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:15:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:56 smithi116 ceph-mon[35411]: pgmap v8653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:15:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:15:58 smithi114 ceph-mon[30820]: pgmap v8654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:15:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:15:58 smithi116 ceph-mon[35411]: pgmap v8654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:00 smithi114 ceph-mon[30820]: pgmap v8655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:01.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:00 smithi116 ceph-mon[35411]: pgmap v8655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:03 smithi114 ceph-mon[30820]: pgmap v8656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:03 smithi116 ceph-mon[35411]: pgmap v8656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:05 smithi114 ceph-mon[30820]: pgmap v8657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:05.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:05 smithi116 ceph-mon[35411]: pgmap v8657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:07.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:07 smithi114 ceph-mon[30820]: pgmap v8658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:07.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:07 smithi116 ceph-mon[35411]: pgmap v8658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:09.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:09 smithi114 ceph-mon[30820]: pgmap v8659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:09 smithi116 ceph-mon[35411]: pgmap v8659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:11 smithi114 ceph-mon[30820]: pgmap v8660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:11 smithi116 ceph-mon[35411]: pgmap v8660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:13.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:13 smithi114 ceph-mon[30820]: pgmap v8661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:13 smithi116 ceph-mon[35411]: pgmap v8661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:15.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:15 smithi114 ceph-mon[30820]: pgmap v8662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:15.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:15 smithi116 ceph-mon[35411]: pgmap v8662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:17 smithi114 ceph-mon[30820]: pgmap v8663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:17 smithi116 ceph-mon[35411]: pgmap v8663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:19 smithi114 ceph-mon[30820]: pgmap v8664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:19 smithi116 ceph-mon[35411]: pgmap v8664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:21 smithi114 ceph-mon[30820]: pgmap v8665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:21 smithi116 ceph-mon[35411]: pgmap v8665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:23.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:23 smithi114 ceph-mon[30820]: pgmap v8666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:23 smithi116 ceph-mon[35411]: pgmap v8666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:25.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:25 smithi114 ceph-mon[30820]: pgmap v8667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:25 smithi116 ceph-mon[35411]: pgmap v8667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:27 smithi114 ceph-mon[30820]: pgmap v8668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:27 smithi116 ceph-mon[35411]: pgmap v8668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:29 smithi114 ceph-mon[30820]: pgmap v8669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:29 smithi116 ceph-mon[35411]: pgmap v8669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:31 smithi114 ceph-mon[30820]: pgmap v8670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:31 smithi116 ceph-mon[35411]: pgmap v8670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:33 smithi114 ceph-mon[30820]: pgmap v8671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:33 smithi116 ceph-mon[35411]: pgmap v8671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:35 smithi114 ceph-mon[30820]: pgmap v8672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:35.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:35 smithi116 ceph-mon[35411]: pgmap v8672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:37 smithi114 ceph-mon[30820]: pgmap v8673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:37.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:37 smithi116 ceph-mon[35411]: pgmap v8673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:39 smithi114 ceph-mon[30820]: pgmap v8674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:39 smithi116 ceph-mon[35411]: pgmap v8674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:41 smithi114 ceph-mon[30820]: pgmap v8675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:41 smithi116 ceph-mon[35411]: pgmap v8675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:16:43.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:43 smithi114 ceph-mon[30820]: pgmap v8676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:43 smithi116 ceph-mon[35411]: pgmap v8676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:16:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:45 smithi114 ceph-mon[30820]: pgmap v8677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:45.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:45 smithi116 ceph-mon[35411]: pgmap v8677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:46.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:16:46.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:16:47.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:47 smithi114 ceph-mon[30820]: pgmap v8678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:16:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:16:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:47 smithi116 ceph-mon[35411]: pgmap v8678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:16:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:16:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:16:49.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-mon[30820]: pgmap v8679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:49.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:16:49.108+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:16:49.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:16:49.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:16:49.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:16:49.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:16:49.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:16:49.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:16:49.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:16:49.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:16:49.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:16:49.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:16:49.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:16:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:16:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:49 smithi116 ceph-mon[35411]: pgmap v8679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:50 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:16:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:16:50.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:16:50.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:16:50.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:50 smithi114 ceph-mon[30820]: pgmap v8680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:16:50.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:50 smithi114 ceph-mon[30820]: pgmap v8681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:16:50.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:50 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:16:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:16:50.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:16:50.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:16:50.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:50 smithi116 ceph-mon[35411]: pgmap v8680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:16:50.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:50 smithi116 ceph-mon[35411]: pgmap v8681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:16:50.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:16:52.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:52 smithi116 ceph-mon[35411]: pgmap v8682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:16:52.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:52 smithi114 ceph-mon[30820]: pgmap v8682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:16:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:54 smithi116 ceph-mon[35411]: pgmap v8683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:16:54.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:54 smithi114 ceph-mon[30820]: pgmap v8683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:16:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:56 smithi116 ceph-mon[35411]: pgmap v8684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:16:56.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:56 smithi114 ceph-mon[30820]: pgmap v8684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:16:58.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:16:58 smithi114 ceph-mon[30820]: pgmap v8685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:16:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:16:58 smithi116 ceph-mon[35411]: pgmap v8685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:17:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:00 smithi114 ceph-mon[30820]: pgmap v8686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:17:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:00 smithi116 ceph-mon[35411]: pgmap v8686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:17:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:02 smithi114 ceph-mon[30820]: pgmap v8687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:02 smithi116 ceph-mon[35411]: pgmap v8687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:04.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:03 smithi114 ceph-mon[30820]: pgmap v8688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:03 smithi116 ceph-mon[35411]: pgmap v8688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:06 smithi116 ceph-mon[35411]: pgmap v8689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:06 smithi114 ceph-mon[30820]: pgmap v8689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:08 smithi116 ceph-mon[35411]: pgmap v8690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:08 smithi114 ceph-mon[30820]: pgmap v8690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:10 smithi114 ceph-mon[30820]: pgmap v8691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:10 smithi116 ceph-mon[35411]: pgmap v8691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:12.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:12 smithi114 ceph-mon[30820]: pgmap v8692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:12 smithi116 ceph-mon[35411]: pgmap v8692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:14 smithi114 ceph-mon[30820]: pgmap v8693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:14 smithi116 ceph-mon[35411]: pgmap v8693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:16 smithi114 ceph-mon[30820]: pgmap v8694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:16 smithi116 ceph-mon[35411]: pgmap v8694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:18 smithi114 ceph-mon[30820]: pgmap v8695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:18 smithi116 ceph-mon[35411]: pgmap v8695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:20 smithi114 ceph-mon[30820]: pgmap v8696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:20 smithi116 ceph-mon[35411]: pgmap v8696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:22.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:22 smithi114 ceph-mon[30820]: pgmap v8697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:22 smithi116 ceph-mon[35411]: pgmap v8697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:24 smithi114 ceph-mon[30820]: pgmap v8698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:24 smithi116 ceph-mon[35411]: pgmap v8698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:26 smithi114 ceph-mon[30820]: pgmap v8699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:26.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:26 smithi116 ceph-mon[35411]: pgmap v8699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:28.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:28 smithi114 ceph-mon[30820]: pgmap v8700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:28 smithi116 ceph-mon[35411]: pgmap v8700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:30 smithi114 ceph-mon[30820]: pgmap v8701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:30 smithi116 ceph-mon[35411]: pgmap v8701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:32 smithi114 ceph-mon[30820]: pgmap v8702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:32 smithi116 ceph-mon[35411]: pgmap v8702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:34 smithi114 ceph-mon[30820]: pgmap v8703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:34 smithi116 ceph-mon[35411]: pgmap v8703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:36.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:36 smithi114 ceph-mon[30820]: pgmap v8704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:36 smithi116 ceph-mon[35411]: pgmap v8704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:38 smithi114 ceph-mon[30820]: pgmap v8705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:38.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:38 smithi116 ceph-mon[35411]: pgmap v8705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:40.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:40 smithi114 ceph-mon[30820]: pgmap v8706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:40 smithi116 ceph-mon[35411]: pgmap v8706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:42 smithi114 ceph-mon[30820]: pgmap v8707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:42.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:42 smithi116 ceph-mon[35411]: pgmap v8707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:44 smithi114 ceph-mon[30820]: pgmap v8708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:44 smithi116 ceph-mon[35411]: pgmap v8708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:46 smithi114 ceph-mon[30820]: pgmap v8709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:46.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:46 smithi116 ceph-mon[35411]: pgmap v8709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:17:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:48 smithi114 ceph-mon[30820]: pgmap v8710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:48.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:48 smithi116 ceph-mon[35411]: pgmap v8710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:17:49.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:17:49.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:17:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:50 smithi114 ceph-mon[30820]: pgmap v8711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:17:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:17:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:50 smithi116 ceph-mon[35411]: pgmap v8711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:17:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:17:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:17:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-mon[30820]: pgmap v8712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:52.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:17:52.468+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:17:52.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:17:52.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:17:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:17:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:17:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:17:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:17:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:17:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:17:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:17:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:17:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:17:52.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:52 smithi116 ceph-mon[35411]: pgmap v8712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:17:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:17:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:17:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:17:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:17:53.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:17:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:17:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:17:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:17:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:17:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:54 smithi114 ceph-mon[30820]: pgmap v8713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:17:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:54 smithi114 ceph-mon[30820]: pgmap v8714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:17:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:54 smithi116 ceph-mon[35411]: pgmap v8713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:17:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:54 smithi116 ceph-mon[35411]: pgmap v8714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:17:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:56 smithi114 ceph-mon[30820]: pgmap v8715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:17:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:56 smithi116 ceph-mon[35411]: pgmap v8715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:17:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:17:58 smithi114 ceph-mon[30820]: pgmap v8716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:17:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:17:58 smithi116 ceph-mon[35411]: pgmap v8716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:18:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:00 smithi114 ceph-mon[30820]: pgmap v8717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:18:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:00 smithi116 ceph-mon[35411]: pgmap v8717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:18:02.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:02 smithi114 ceph-mon[30820]: pgmap v8718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:18:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:02 smithi116 ceph-mon[35411]: pgmap v8718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:18:04.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:04 smithi114 ceph-mon[30820]: pgmap v8719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:18:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:04 smithi116 ceph-mon[35411]: pgmap v8719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:18:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:06 smithi114 ceph-mon[30820]: pgmap v8720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:06 smithi116 ceph-mon[35411]: pgmap v8720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:08 smithi114 ceph-mon[30820]: pgmap v8721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:08 smithi116 ceph-mon[35411]: pgmap v8721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:10 smithi114 ceph-mon[30820]: pgmap v8722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:10 smithi116 ceph-mon[35411]: pgmap v8722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:11 smithi114 ceph-mon[30820]: pgmap v8723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:11 smithi116 ceph-mon[35411]: pgmap v8723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:13 smithi114 ceph-mon[30820]: pgmap v8724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:13.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:13 smithi116 ceph-mon[35411]: pgmap v8724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:15 smithi114 ceph-mon[30820]: pgmap v8725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:15 smithi116 ceph-mon[35411]: pgmap v8725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:17 smithi114 ceph-mon[30820]: pgmap v8726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:17.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:17 smithi116 ceph-mon[35411]: pgmap v8726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:19.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:19 smithi114 ceph-mon[30820]: pgmap v8727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:19.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:19 smithi116 ceph-mon[35411]: pgmap v8727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:21 smithi114 ceph-mon[30820]: pgmap v8728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:21.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:21 smithi116 ceph-mon[35411]: pgmap v8728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:23 smithi114 ceph-mon[30820]: pgmap v8729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:23 smithi116 ceph-mon[35411]: pgmap v8729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:25 smithi114 ceph-mon[30820]: pgmap v8730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:25 smithi116 ceph-mon[35411]: pgmap v8730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:27 smithi114 ceph-mon[30820]: pgmap v8731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:27 smithi116 ceph-mon[35411]: pgmap v8731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:29 smithi114 ceph-mon[30820]: pgmap v8732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:29 smithi116 ceph-mon[35411]: pgmap v8732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:31 smithi114 ceph-mon[30820]: pgmap v8733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:31 smithi116 ceph-mon[35411]: pgmap v8733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:33 smithi114 ceph-mon[30820]: pgmap v8734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:33 smithi116 ceph-mon[35411]: pgmap v8734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:35 smithi114 ceph-mon[30820]: pgmap v8735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:35 smithi116 ceph-mon[35411]: pgmap v8735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:37 smithi114 ceph-mon[30820]: pgmap v8736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:37.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:37 smithi116 ceph-mon[35411]: pgmap v8736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:39 smithi114 ceph-mon[30820]: pgmap v8737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:39 smithi116 ceph-mon[35411]: pgmap v8737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:41 smithi114 ceph-mon[30820]: pgmap v8738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:41 smithi116 ceph-mon[35411]: pgmap v8738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:43 smithi114 ceph-mon[30820]: pgmap v8739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:43 smithi116 ceph-mon[35411]: pgmap v8739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:45 smithi114 ceph-mon[30820]: pgmap v8740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:45.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:45 smithi116 ceph-mon[35411]: pgmap v8740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:47 smithi114 ceph-mon[30820]: pgmap v8741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:47 smithi116 ceph-mon[35411]: pgmap v8741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:18:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:49 smithi114 ceph-mon[30820]: pgmap v8742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:49 smithi116 ceph-mon[35411]: pgmap v8742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:51 smithi114 ceph-mon[30820]: pgmap v8743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:51.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:51 smithi116 ceph-mon[35411]: pgmap v8743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:18:52.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:18:52.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:18:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:53 smithi114 ceph-mon[30820]: pgmap v8744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:53 smithi116 ceph-mon[35411]: pgmap v8744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:18:55.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-mon[30820]: pgmap v8745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:55.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:18:55.828+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:18:55.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:18:55.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:18:55.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:18:55.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:18:55.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:18:55.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:18:55.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:18:55.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:18:55.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:18:55.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:18:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:18:55.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:55 smithi116 ceph-mon[35411]: pgmap v8745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:18:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:18:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:18:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:18:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:18:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:18:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:18:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:18:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:18:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:18:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:18:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:18:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:18:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:18:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:18:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:18:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:18:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:18:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:56 smithi114 ceph-mon[30820]: pgmap v8746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:18:56.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:56 smithi114 ceph-mon[30820]: pgmap v8747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:18:56.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:18:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:18:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:18:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:18:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:18:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:18:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:56 smithi116 ceph-mon[35411]: pgmap v8746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:56 smithi116 ceph-mon[35411]: pgmap v8747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:18:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:18:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:18:58 smithi114 ceph-mon[30820]: pgmap v8748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:18:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:18:58 smithi116 ceph-mon[35411]: pgmap v8748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:19:01.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:00 smithi114 ceph-mon[30820]: pgmap v8749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:19:01.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:00 smithi116 ceph-mon[35411]: pgmap v8749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:19:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:02 smithi114 ceph-mon[30820]: pgmap v8750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:19:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:02 smithi116 ceph-mon[35411]: pgmap v8750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:19:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:04 smithi114 ceph-mon[30820]: pgmap v8751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:19:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:04 smithi116 ceph-mon[35411]: pgmap v8751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:19:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:06 smithi114 ceph-mon[30820]: pgmap v8752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:19:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:06 smithi116 ceph-mon[35411]: pgmap v8752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:19:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:08 smithi114 ceph-mon[30820]: pgmap v8753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:09.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:08 smithi116 ceph-mon[35411]: pgmap v8753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:11.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:10 smithi114 ceph-mon[30820]: pgmap v8754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:11.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:10 smithi116 ceph-mon[35411]: pgmap v8754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:12 smithi116 ceph-mon[35411]: pgmap v8755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:12 smithi114 ceph-mon[30820]: pgmap v8755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:14 smithi116 ceph-mon[35411]: pgmap v8756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:14 smithi114 ceph-mon[30820]: pgmap v8756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:16 smithi116 ceph-mon[35411]: pgmap v8757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:16 smithi114 ceph-mon[30820]: pgmap v8757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:19.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:18 smithi114 ceph-mon[30820]: pgmap v8758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:19.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:18 smithi116 ceph-mon[35411]: pgmap v8758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:20 smithi114 ceph-mon[30820]: pgmap v8759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:21.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:20 smithi116 ceph-mon[35411]: pgmap v8759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:22 smithi114 ceph-mon[30820]: pgmap v8760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:22 smithi116 ceph-mon[35411]: pgmap v8760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:24 smithi114 ceph-mon[30820]: pgmap v8761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:24 smithi116 ceph-mon[35411]: pgmap v8761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:26 smithi114 ceph-mon[30820]: pgmap v8762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:26 smithi116 ceph-mon[35411]: pgmap v8762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:29.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:28 smithi114 ceph-mon[30820]: pgmap v8763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:28 smithi116 ceph-mon[35411]: pgmap v8763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:30 smithi114 ceph-mon[30820]: pgmap v8764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:31.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:30 smithi116 ceph-mon[35411]: pgmap v8764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:32 smithi114 ceph-mon[30820]: pgmap v8765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:33.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:32 smithi116 ceph-mon[35411]: pgmap v8765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:34 smithi114 ceph-mon[30820]: pgmap v8766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:34 smithi116 ceph-mon[35411]: pgmap v8766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:36 smithi114 ceph-mon[30820]: pgmap v8767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:36 smithi116 ceph-mon[35411]: pgmap v8767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:39.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:38 smithi114 ceph-mon[30820]: pgmap v8768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:38 smithi116 ceph-mon[35411]: pgmap v8768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:41.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:40 smithi116 ceph-mon[35411]: pgmap v8769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:40 smithi114 ceph-mon[30820]: pgmap v8769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:42 smithi114 ceph-mon[30820]: pgmap v8770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:43.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:42 smithi116 ceph-mon[35411]: pgmap v8770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:44 smithi114 ceph-mon[30820]: pgmap v8771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:44 smithi116 ceph-mon[35411]: pgmap v8771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:46 smithi114 ceph-mon[30820]: pgmap v8772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:46 smithi116 ceph-mon[35411]: pgmap v8772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:48 smithi114 ceph-mon[30820]: pgmap v8773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:49.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:48 smithi116 ceph-mon[35411]: pgmap v8773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:50 smithi114 ceph-mon[30820]: pgmap v8774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:50 smithi116 ceph-mon[35411]: pgmap v8774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:19:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:52 smithi114 ceph-mon[30820]: pgmap v8775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:53.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:52 smithi116 ceph-mon[35411]: pgmap v8775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:19:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:55 smithi114 ceph-mon[30820]: pgmap v8776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:55 smithi116 ceph-mon[35411]: pgmap v8776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:19:56.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:19:57.416 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:57 smithi114 ceph-mon[30820]: pgmap v8777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:57 smithi116 ceph-mon[35411]: pgmap v8777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:19:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-mon[30820]: pgmap v8778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:19:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:19:59.189+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:19:59.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:19:59.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:19:59.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:19:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:19:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:19:59 smithi116 ceph-mon[35411]: pgmap v8778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:00 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:20:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:20:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:20:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:20:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:20:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:20:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:20:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:20:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:00 smithi114 ceph-mon[30820]: pgmap v8779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:00 smithi114 ceph-mon[30820]: pgmap v8780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:20:00.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:00 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:20:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:20:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:00 smithi116 ceph-mon[35411]: pgmap v8779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:20:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:00 smithi116 ceph-mon[35411]: pgmap v8780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:20:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:20:00.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:20:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:02 smithi114 ceph-mon[30820]: pgmap v8781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:20:02.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:02 smithi116 ceph-mon[35411]: pgmap v8781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:20:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:04 smithi114 ceph-mon[30820]: pgmap v8782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:20:04.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:04 smithi116 ceph-mon[35411]: pgmap v8782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:20:06.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:06 smithi114 ceph-mon[30820]: pgmap v8783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:20:06.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:06 smithi116 ceph-mon[35411]: pgmap v8783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:20:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:08 smithi114 ceph-mon[30820]: pgmap v8784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:20:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:08 smithi116 ceph-mon[35411]: pgmap v8784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:20:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:10 smithi114 ceph-mon[30820]: pgmap v8785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:20:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:10 smithi116 ceph-mon[35411]: pgmap v8785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:20:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:12 smithi114 ceph-mon[30820]: pgmap v8786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:12 smithi116 ceph-mon[35411]: pgmap v8786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:14 smithi114 ceph-mon[30820]: pgmap v8787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:14 smithi116 ceph-mon[35411]: pgmap v8787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:16 smithi114 ceph-mon[30820]: pgmap v8788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:16 smithi116 ceph-mon[35411]: pgmap v8788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:18 smithi114 ceph-mon[30820]: pgmap v8789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:18 smithi116 ceph-mon[35411]: pgmap v8789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:20 smithi114 ceph-mon[30820]: pgmap v8790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:20 smithi116 ceph-mon[35411]: pgmap v8790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:22 smithi114 ceph-mon[30820]: pgmap v8791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:22 smithi116 ceph-mon[35411]: pgmap v8791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:24 smithi114 ceph-mon[30820]: pgmap v8792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:24 smithi116 ceph-mon[35411]: pgmap v8792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:26.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:26 smithi114 ceph-mon[30820]: pgmap v8793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:26 smithi116 ceph-mon[35411]: pgmap v8793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:28 smithi114 ceph-mon[30820]: pgmap v8794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:28.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:28 smithi116 ceph-mon[35411]: pgmap v8794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:30 smithi114 ceph-mon[30820]: pgmap v8795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:30.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:30 smithi116 ceph-mon[35411]: pgmap v8795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:32.673 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:32 smithi114 ceph-mon[30820]: pgmap v8796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:32.719 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:32 smithi116 ceph-mon[35411]: pgmap v8796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:34 smithi114 ceph-mon[30820]: pgmap v8797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:34.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:34 smithi116 ceph-mon[35411]: pgmap v8797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:36 smithi114 ceph-mon[30820]: pgmap v8798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:36.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:36 smithi116 ceph-mon[35411]: pgmap v8798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:38 smithi114 ceph-mon[30820]: pgmap v8799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:38 smithi116 ceph-mon[35411]: pgmap v8799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:40 smithi114 ceph-mon[30820]: pgmap v8800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:40.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:40 smithi116 ceph-mon[35411]: pgmap v8800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:41 smithi114 ceph-mon[30820]: pgmap v8801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:41 smithi116 ceph-mon[35411]: pgmap v8801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:44 smithi114 ceph-mon[30820]: pgmap v8802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:44.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:44 smithi116 ceph-mon[35411]: pgmap v8802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:46 smithi114 ceph-mon[30820]: pgmap v8803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:46 smithi116 ceph-mon[35411]: pgmap v8803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:48 smithi114 ceph-mon[30820]: pgmap v8804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:48 smithi116 ceph-mon[35411]: pgmap v8804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:50.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:50 smithi114 ceph-mon[30820]: pgmap v8805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:50.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:50 smithi116 ceph-mon[35411]: pgmap v8805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:52 smithi114 ceph-mon[30820]: pgmap v8806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:52 smithi116 ceph-mon[35411]: pgmap v8806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:20:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:54 smithi114 ceph-mon[30820]: pgmap v8807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:54 smithi116 ceph-mon[35411]: pgmap v8807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:20:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:56 smithi114 ceph-mon[30820]: pgmap v8808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:56.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:56 smithi116 ceph-mon[35411]: pgmap v8808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:20:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:20:58 smithi114 ceph-mon[30820]: pgmap v8809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:20:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:20:58 smithi116 ceph-mon[35411]: pgmap v8809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:21:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:00 smithi114 ceph-mon[30820]: pgmap v8810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:21:00.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:00 smithi116 ceph-mon[35411]: pgmap v8810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:21:02.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-mon[30820]: pgmap v8811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:02 smithi116 ceph-mon[35411]: pgmap v8811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:02.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:21:02.548+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:21:02.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:21:02.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:21:02.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:21:02.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:21:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:21:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:21:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:21:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:21:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:21:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:21:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:21:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:21:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:21:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:21:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:21:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:21:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:21:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:21:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:21:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:21:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:21:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:21:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:21:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:21:04.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:21:04.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:21:04.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:21:04.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:21:04.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:21:04.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:21:04.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:21:04.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:21:04.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:21:04.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:21:04.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:04 smithi114 ceph-mon[30820]: pgmap v8812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:21:04.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:04 smithi114 ceph-mon[30820]: pgmap v8813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:21:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:21:04.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:21:04.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:04 smithi116 ceph-mon[35411]: pgmap v8812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:21:04.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:04 smithi116 ceph-mon[35411]: pgmap v8813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:21:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:06 smithi114 ceph-mon[30820]: pgmap v8814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:21:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:06 smithi116 ceph-mon[35411]: pgmap v8814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:21:08.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:08 smithi114 ceph-mon[30820]: pgmap v8815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:21:08.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:08 smithi116 ceph-mon[35411]: pgmap v8815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:21:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:10 smithi114 ceph-mon[30820]: pgmap v8816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:21:10.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:10 smithi116 ceph-mon[35411]: pgmap v8816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:21:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:12 smithi114 ceph-mon[30820]: pgmap v8817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:21:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:12 smithi116 ceph-mon[35411]: pgmap v8817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:21:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:14 smithi114 ceph-mon[30820]: pgmap v8818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:21:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:14 smithi116 ceph-mon[35411]: pgmap v8818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:21:16.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:16 smithi114 ceph-mon[30820]: pgmap v8819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:16.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:16 smithi116 ceph-mon[35411]: pgmap v8819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:18 smithi114 ceph-mon[30820]: pgmap v8820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:18 smithi116 ceph-mon[35411]: pgmap v8820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:20 smithi114 ceph-mon[30820]: pgmap v8821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:20.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:20 smithi116 ceph-mon[35411]: pgmap v8821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:21 smithi114 ceph-mon[30820]: pgmap v8822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:21 smithi116 ceph-mon[35411]: pgmap v8822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:23 smithi114 ceph-mon[30820]: pgmap v8823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:23 smithi116 ceph-mon[35411]: pgmap v8823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:25 smithi114 ceph-mon[30820]: pgmap v8824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:25 smithi116 ceph-mon[35411]: pgmap v8824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:27 smithi114 ceph-mon[30820]: pgmap v8825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:27.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:27 smithi116 ceph-mon[35411]: pgmap v8825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:29 smithi114 ceph-mon[30820]: pgmap v8826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:29 smithi116 ceph-mon[35411]: pgmap v8826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:31 smithi114 ceph-mon[30820]: pgmap v8827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:31 smithi116 ceph-mon[35411]: pgmap v8827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:33 smithi114 ceph-mon[30820]: pgmap v8828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:33.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:33 smithi116 ceph-mon[35411]: pgmap v8828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:35 smithi114 ceph-mon[30820]: pgmap v8829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:35 smithi116 ceph-mon[35411]: pgmap v8829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:37 smithi114 ceph-mon[30820]: pgmap v8830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:37 smithi116 ceph-mon[35411]: pgmap v8830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:39 smithi114 ceph-mon[30820]: pgmap v8831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:39 smithi116 ceph-mon[35411]: pgmap v8831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:41 smithi114 ceph-mon[30820]: pgmap v8832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:41 smithi116 ceph-mon[35411]: pgmap v8832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:43 smithi114 ceph-mon[30820]: pgmap v8833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:43 smithi116 ceph-mon[35411]: pgmap v8833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:45 smithi114 ceph-mon[30820]: pgmap v8834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:45 smithi116 ceph-mon[35411]: pgmap v8834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:47 smithi114 ceph-mon[30820]: pgmap v8835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:47 smithi116 ceph-mon[35411]: pgmap v8835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:49 smithi114 ceph-mon[30820]: pgmap v8836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:49 smithi116 ceph-mon[35411]: pgmap v8836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:51 smithi116 ceph-mon[35411]: pgmap v8837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:51 smithi114 ceph-mon[30820]: pgmap v8837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:21:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:53 smithi116 ceph-mon[35411]: pgmap v8838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:53 smithi114 ceph-mon[30820]: pgmap v8838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:21:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:55 smithi116 ceph-mon[35411]: pgmap v8839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:55 smithi114 ceph-mon[30820]: pgmap v8839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:21:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:57 smithi114 ceph-mon[30820]: pgmap v8840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:21:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:57 smithi116 ceph-mon[35411]: pgmap v8840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:21:59 smithi114 ceph-mon[30820]: pgmap v8841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:21:59 smithi116 ceph-mon[35411]: pgmap v8841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:01.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:01 smithi114 ceph-mon[30820]: pgmap v8842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:01 smithi116 ceph-mon[35411]: pgmap v8842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:03.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:22:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:22:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: pgmap v8843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:22:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:22:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: pgmap v8843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:22:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:22:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:22:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:04 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:22:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:04 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:22:05.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:04 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:22:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:04 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:22:06.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-mon[30820]: pgmap v8844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:22:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:22:05.908+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:22:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:22:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:22:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:22:06.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:22:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:22:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:22:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:22:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:22:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:22:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:22:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:22:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:22:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:05 smithi116 ceph-mon[35411]: pgmap v8844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:22:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:06 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:22:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:22:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:22:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:22:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:06 smithi114 ceph-mon[30820]: pgmap v8845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:06 smithi114 ceph-mon[30820]: pgmap v8846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:22:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:06 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:22:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:22:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:06 smithi116 ceph-mon[35411]: pgmap v8845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:22:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:06 smithi116 ceph-mon[35411]: pgmap v8846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:22:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:22:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:08 smithi114 ceph-mon[30820]: pgmap v8847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:22:09.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:08 smithi116 ceph-mon[35411]: pgmap v8847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:22:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:10 smithi114 ceph-mon[30820]: pgmap v8848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:22:11.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:10 smithi116 ceph-mon[35411]: pgmap v8848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:22:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:12 smithi114 ceph-mon[30820]: pgmap v8849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:22:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:12 smithi116 ceph-mon[35411]: pgmap v8849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:22:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:14 smithi114 ceph-mon[30820]: pgmap v8850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T11:22:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:14 smithi116 ceph-mon[35411]: pgmap v8850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T11:22:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:16 smithi114 ceph-mon[30820]: pgmap v8851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:22:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:16 smithi116 ceph-mon[35411]: pgmap v8851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:22:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:18 smithi114 ceph-mon[30820]: pgmap v8852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:22:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:18 smithi116 ceph-mon[35411]: pgmap v8852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:22:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:20 smithi114 ceph-mon[30820]: pgmap v8853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:22:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:20 smithi116 ceph-mon[35411]: pgmap v8853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:22:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:22 smithi114 ceph-mon[30820]: pgmap v8854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:22 smithi116 ceph-mon[35411]: pgmap v8854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:24 smithi114 ceph-mon[30820]: pgmap v8855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:24 smithi116 ceph-mon[35411]: pgmap v8855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:27 smithi114 ceph-mon[30820]: pgmap v8856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:27.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:27 smithi116 ceph-mon[35411]: pgmap v8856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:29 smithi114 ceph-mon[30820]: pgmap v8857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:29 smithi116 ceph-mon[35411]: pgmap v8857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:31 smithi114 ceph-mon[30820]: pgmap v8858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:22:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:31 smithi116 ceph-mon[35411]: pgmap v8858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:22:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:33 smithi114 ceph-mon[30820]: pgmap v8859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:33 smithi116 ceph-mon[35411]: pgmap v8859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:35 smithi114 ceph-mon[30820]: pgmap v8860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:35 smithi116 ceph-mon[35411]: pgmap v8860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:37 smithi114 ceph-mon[30820]: pgmap v8861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:37 smithi116 ceph-mon[35411]: pgmap v8861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:39 smithi114 ceph-mon[30820]: pgmap v8862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:39.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:39 smithi116 ceph-mon[35411]: pgmap v8862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:41 smithi114 ceph-mon[30820]: pgmap v8863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:22:41.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:41 smithi116 ceph-mon[35411]: pgmap v8863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:22:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:43 smithi114 ceph-mon[30820]: pgmap v8864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:43 smithi116 ceph-mon[35411]: pgmap v8864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:45 smithi114 ceph-mon[30820]: pgmap v8865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:45 smithi116 ceph-mon[35411]: pgmap v8865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:47 smithi114 ceph-mon[30820]: pgmap v8866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:47.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:47 smithi116 ceph-mon[35411]: pgmap v8866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:49 smithi114 ceph-mon[30820]: pgmap v8867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:49 smithi116 ceph-mon[35411]: pgmap v8867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:51 smithi114 ceph-mon[30820]: pgmap v8868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:22:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:51 smithi116 ceph-mon[35411]: pgmap v8868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:22:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:53 smithi114 ceph-mon[30820]: pgmap v8869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:53 smithi116 ceph-mon[35411]: pgmap v8869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:22:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:55 smithi114 ceph-mon[30820]: pgmap v8870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:55.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:55 smithi116 ceph-mon[35411]: pgmap v8870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:57 smithi114 ceph-mon[30820]: pgmap v8871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:57 smithi116 ceph-mon[35411]: pgmap v8871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:22:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:22:59 smithi114 ceph-mon[30820]: pgmap v8872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:22:59.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:22:59 smithi116 ceph-mon[35411]: pgmap v8872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:01 smithi114 ceph-mon[30820]: pgmap v8873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:23:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:01 smithi116 ceph-mon[35411]: pgmap v8873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:23:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:03 smithi114 ceph-mon[30820]: pgmap v8874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:23:03.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:03 smithi116 ceph-mon[35411]: pgmap v8874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:23:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:05 smithi114 ceph-mon[30820]: pgmap v8875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:05 smithi116 ceph-mon[35411]: pgmap v8875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:23:06.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:23:07.081 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:07 smithi116 ceph-mon[35411]: pgmap v8876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:23:07.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:07 smithi114 ceph-mon[30820]: pgmap v8876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:23:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-mon[30820]: pgmap v8877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:09.377 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:09 smithi116 ceph-mon[35411]: pgmap v8877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:09.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:23:09.269+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:23:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:23:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:23:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:23:10.341 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:10 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:23:10.341 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:23:10.341 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:23:10.341 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:23:10.341 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:23:10.342 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:23:10.343 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:23:10.343 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:23:10.343 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:23:10.343 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:23:10.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:10.343 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:10.370 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:10 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:23:10.370 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:23:10.370 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:23:10.371 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:10.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:11 smithi114 ceph-mon[30820]: pgmap v8878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:23:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:11 smithi116 ceph-mon[35411]: pgmap v8878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:23:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:23:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:23:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:12 smithi114 ceph-mon[30820]: pgmap v8879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:23:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:12 smithi114 ceph-mon[30820]: pgmap v8880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:23:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:23:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:23:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:12 smithi116 ceph-mon[35411]: pgmap v8879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:23:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:12 smithi116 ceph-mon[35411]: pgmap v8880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:23:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:23:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:14 smithi114 ceph-mon[30820]: pgmap v8881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:23:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:14 smithi116 ceph-mon[35411]: pgmap v8881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:23:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:16 smithi114 ceph-mon[30820]: pgmap v8882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:23:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:16 smithi116 ceph-mon[35411]: pgmap v8882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:23:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:18 smithi114 ceph-mon[30820]: pgmap v8883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:23:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:18 smithi116 ceph-mon[35411]: pgmap v8883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:23:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:20 smithi114 ceph-mon[30820]: pgmap v8884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:23:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:20 smithi116 ceph-mon[35411]: pgmap v8884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:23:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:22 smithi114 ceph-mon[30820]: pgmap v8885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:23:22.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:22 smithi116 ceph-mon[35411]: pgmap v8885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:23:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:24 smithi114 ceph-mon[30820]: pgmap v8886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:24 smithi116 ceph-mon[35411]: pgmap v8886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:26 smithi114 ceph-mon[30820]: pgmap v8887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:23:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:26 smithi116 ceph-mon[35411]: pgmap v8887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:23:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:28 smithi114 ceph-mon[30820]: pgmap v8888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:23:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:28 smithi116 ceph-mon[35411]: pgmap v8888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:23:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:30 smithi114 ceph-mon[30820]: pgmap v8889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:30 smithi116 ceph-mon[35411]: pgmap v8889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:32 smithi114 ceph-mon[30820]: pgmap v8890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:23:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:32 smithi116 ceph-mon[35411]: pgmap v8890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:23:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:34 smithi114 ceph-mon[30820]: pgmap v8891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:34 smithi116 ceph-mon[35411]: pgmap v8891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:36 smithi114 ceph-mon[30820]: pgmap v8892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:23:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:36 smithi116 ceph-mon[35411]: pgmap v8892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:23:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:38 smithi114 ceph-mon[30820]: pgmap v8893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:23:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:38 smithi116 ceph-mon[35411]: pgmap v8893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:23:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:40 smithi114 ceph-mon[30820]: pgmap v8894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:40 smithi116 ceph-mon[35411]: pgmap v8894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:42 smithi114 ceph-mon[30820]: pgmap v8895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:23:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:42 smithi116 ceph-mon[35411]: pgmap v8895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:23:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:44 smithi114 ceph-mon[30820]: pgmap v8896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:44 smithi116 ceph-mon[35411]: pgmap v8896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:46 smithi114 ceph-mon[30820]: pgmap v8897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T11:23:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:46 smithi116 ceph-mon[35411]: pgmap v8897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T11:23:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:48 smithi114 ceph-mon[30820]: pgmap v8898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:23:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:48 smithi116 ceph-mon[35411]: pgmap v8898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:23:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:49 smithi114 ceph-mon[30820]: pgmap v8899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:49 smithi116 ceph-mon[35411]: pgmap v8899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:52 smithi114 ceph-mon[30820]: pgmap v8900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:23:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:52 smithi116 ceph-mon[35411]: pgmap v8900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:23:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:54 smithi114 ceph-mon[30820]: pgmap v8901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:54 smithi116 ceph-mon[35411]: pgmap v8901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:23:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:56 smithi114 ceph-mon[30820]: pgmap v8902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T11:23:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:56 smithi116 ceph-mon[35411]: pgmap v8902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T11:23:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:23:58 smithi114 ceph-mon[30820]: pgmap v8903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:23:58.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:23:58 smithi116 ceph-mon[35411]: pgmap v8903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:24:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:00 smithi114 ceph-mon[30820]: pgmap v8904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:00 smithi116 ceph-mon[35411]: pgmap v8904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:02 smithi114 ceph-mon[30820]: pgmap v8905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:02 smithi116 ceph-mon[35411]: pgmap v8905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:04 smithi114 ceph-mon[30820]: pgmap v8906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:04 smithi116 ceph-mon[35411]: pgmap v8906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:06 smithi114 ceph-mon[30820]: pgmap v8907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:24:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:06 smithi116 ceph-mon[35411]: pgmap v8907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:24:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:08 smithi114 ceph-mon[30820]: pgmap v8908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:24:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:08 smithi116 ceph-mon[35411]: pgmap v8908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:24:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:10 smithi114 ceph-mon[30820]: pgmap v8909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:10 smithi116 ceph-mon[35411]: pgmap v8909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:12 smithi114 ceph-mon[30820]: pgmap v8910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:24:12.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:12 smithi116 ceph-mon[35411]: pgmap v8910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:24:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-mon[30820]: pgmap v8911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:24:14.548+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:24:14.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:24:14.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:24:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:24:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:14 smithi116 ceph-mon[35411]: pgmap v8911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:24:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:24:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:24:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:24:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:24:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:24:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:16 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:24:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:24:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:24:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:24:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:24:16.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:24:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:24:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:24:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:24:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:24:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:16 smithi114 ceph-mon[30820]: pgmap v8912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T11:24:16.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:16 smithi114 ceph-mon[30820]: pgmap v8913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:16 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:24:16.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:16 smithi116 ceph-mon[35411]: pgmap v8912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T11:24:16.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:16 smithi116 ceph-mon[35411]: pgmap v8913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T11:24:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:18 smithi114 ceph-mon[30820]: pgmap v8914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:24:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:18 smithi116 ceph-mon[35411]: pgmap v8914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:24:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:19 smithi116 ceph-mon[35411]: pgmap v8915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:24:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:19 smithi114 ceph-mon[30820]: pgmap v8915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:24:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:21 smithi114 ceph-mon[30820]: pgmap v8916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:24:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:21 smithi116 ceph-mon[35411]: pgmap v8916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:24:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:23 smithi114 ceph-mon[30820]: pgmap v8917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:24:23.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:23 smithi116 ceph-mon[35411]: pgmap v8917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:24:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:25 smithi114 ceph-mon[30820]: pgmap v8918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:24:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:25 smithi116 ceph-mon[35411]: pgmap v8918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:24:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:27 smithi114 ceph-mon[30820]: pgmap v8919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:24:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:27 smithi116 ceph-mon[35411]: pgmap v8919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:24:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:29 smithi114 ceph-mon[30820]: pgmap v8920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:24:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:29 smithi116 ceph-mon[35411]: pgmap v8920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:24:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:31 smithi114 ceph-mon[30820]: pgmap v8921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:31 smithi116 ceph-mon[35411]: pgmap v8921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:33 smithi114 ceph-mon[30820]: pgmap v8922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:24:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:33 smithi116 ceph-mon[35411]: pgmap v8922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:24:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:35 smithi114 ceph-mon[30820]: pgmap v8923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:24:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:35 smithi116 ceph-mon[35411]: pgmap v8923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:24:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:37 smithi114 ceph-mon[30820]: pgmap v8924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:24:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:37 smithi116 ceph-mon[35411]: pgmap v8924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:24:39.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:39 smithi114 ceph-mon[30820]: pgmap v8925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:39 smithi116 ceph-mon[35411]: pgmap v8925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:41 smithi114 ceph-mon[30820]: pgmap v8926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:41 smithi116 ceph-mon[35411]: pgmap v8926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:43 smithi114 ceph-mon[30820]: pgmap v8927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:43 smithi116 ceph-mon[35411]: pgmap v8927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:45 smithi114 ceph-mon[30820]: pgmap v8928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:24:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:45 smithi116 ceph-mon[35411]: pgmap v8928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:24:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:47 smithi114 ceph-mon[30820]: pgmap v8929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:24:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:47 smithi116 ceph-mon[35411]: pgmap v8929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:24:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:49 smithi114 ceph-mon[30820]: pgmap v8930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:49 smithi116 ceph-mon[35411]: pgmap v8930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:51 smithi116 ceph-mon[35411]: pgmap v8931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:51 smithi114 ceph-mon[30820]: pgmap v8931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:24:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:53 smithi116 ceph-mon[35411]: pgmap v8932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:53 smithi114 ceph-mon[30820]: pgmap v8932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:24:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:55 smithi116 ceph-mon[35411]: pgmap v8933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:24:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:55 smithi114 ceph-mon[30820]: pgmap v8933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:24:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:57 smithi114 ceph-mon[30820]: pgmap v8934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:24:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:57 smithi116 ceph-mon[35411]: pgmap v8934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:24:59 smithi114 ceph-mon[30820]: pgmap v8935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:00.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:24:59 smithi116 ceph-mon[35411]: pgmap v8935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:01 smithi114 ceph-mon[30820]: pgmap v8936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:01 smithi116 ceph-mon[35411]: pgmap v8936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:03 smithi114 ceph-mon[30820]: pgmap v8937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:03 smithi116 ceph-mon[35411]: pgmap v8937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:05 smithi114 ceph-mon[30820]: pgmap v8938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:05 smithi116 ceph-mon[35411]: pgmap v8938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:07 smithi114 ceph-mon[30820]: pgmap v8939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:07 smithi116 ceph-mon[35411]: pgmap v8939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:09 smithi114 ceph-mon[30820]: pgmap v8940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:09 smithi116 ceph-mon[35411]: pgmap v8940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:11 smithi114 ceph-mon[30820]: pgmap v8941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:11 smithi116 ceph-mon[35411]: pgmap v8941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:13 smithi114 ceph-mon[30820]: pgmap v8942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:13 smithi116 ceph-mon[35411]: pgmap v8942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:15.277 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:25:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:25:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:15 smithi114 ceph-mon[30820]: pgmap v8943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:15 smithi116 ceph-mon[35411]: pgmap v8943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:18 smithi114 ceph-mon[30820]: pgmap v8944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:25:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:25:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:25:17.908+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:25:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:25:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:25:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:25:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:18 smithi116 ceph-mon[35411]: pgmap v8944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:25:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:25:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:25:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:25:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:25:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:25:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:25:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:19 smithi114 ceph-mon[30820]: pgmap v8945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:25:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:19 smithi114 ceph-mon[30820]: pgmap v8946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:25:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:25:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:25:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:25:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:25:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:25:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:25:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:25:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:25:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:25:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:19 smithi116 ceph-mon[35411]: pgmap v8945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:25:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:19 smithi116 ceph-mon[35411]: pgmap v8946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:25:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:21 smithi114 ceph-mon[30820]: pgmap v8947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:25:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:21 smithi116 ceph-mon[35411]: pgmap v8947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:25:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:23 smithi114 ceph-mon[30820]: pgmap v8948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:25:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:23 smithi116 ceph-mon[35411]: pgmap v8948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:25:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:25 smithi114 ceph-mon[30820]: pgmap v8949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:25:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:25 smithi116 ceph-mon[35411]: pgmap v8949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:25:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:27 smithi114 ceph-mon[30820]: pgmap v8950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:25:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:27 smithi116 ceph-mon[35411]: pgmap v8950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:29 smithi114 ceph-mon[30820]: pgmap v8951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:29 smithi116 ceph-mon[35411]: pgmap v8951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:25:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:31 smithi114 ceph-mon[30820]: pgmap v8952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:31 smithi116 ceph-mon[35411]: pgmap v8952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:33 smithi114 ceph-mon[30820]: pgmap v8953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:33 smithi116 ceph-mon[35411]: pgmap v8953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:35 smithi114 ceph-mon[30820]: pgmap v8954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:35 smithi116 ceph-mon[35411]: pgmap v8954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:37 smithi114 ceph-mon[30820]: pgmap v8955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:37 smithi116 ceph-mon[35411]: pgmap v8955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:39 smithi114 ceph-mon[30820]: pgmap v8956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:39 smithi116 ceph-mon[35411]: pgmap v8956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:41 smithi114 ceph-mon[30820]: pgmap v8957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:41 smithi116 ceph-mon[35411]: pgmap v8957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:43 smithi114 ceph-mon[30820]: pgmap v8958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:43 smithi116 ceph-mon[35411]: pgmap v8958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:45 smithi114 ceph-mon[30820]: pgmap v8959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:45 smithi116 ceph-mon[35411]: pgmap v8959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:47 smithi114 ceph-mon[30820]: pgmap v8960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:47 smithi116 ceph-mon[35411]: pgmap v8960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:49 smithi114 ceph-mon[30820]: pgmap v8961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:49 smithi116 ceph-mon[35411]: pgmap v8961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:51 smithi114 ceph-mon[30820]: pgmap v8962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:51 smithi116 ceph-mon[35411]: pgmap v8962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:25:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:53 smithi116 ceph-mon[35411]: pgmap v8963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:53 smithi114 ceph-mon[30820]: pgmap v8963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:25:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:55 smithi116 ceph-mon[35411]: pgmap v8964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:55 smithi114 ceph-mon[30820]: pgmap v8964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:57 smithi116 ceph-mon[35411]: pgmap v8965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:57 smithi114 ceph-mon[30820]: pgmap v8965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:25:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:25:59 smithi116 ceph-mon[35411]: pgmap v8966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:25:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:25:59 smithi114 ceph-mon[30820]: pgmap v8966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:01 smithi114 ceph-mon[30820]: pgmap v8967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:26:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:01 smithi116 ceph-mon[35411]: pgmap v8967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:26:03.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:03 smithi114 ceph-mon[30820]: pgmap v8968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:26:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:03 smithi116 ceph-mon[35411]: pgmap v8968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:26:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:05 smithi114 ceph-mon[30820]: pgmap v8969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:05 smithi116 ceph-mon[35411]: pgmap v8969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:07 smithi114 ceph-mon[30820]: pgmap v8970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:26:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:07 smithi116 ceph-mon[35411]: pgmap v8970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:26:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:09 smithi114 ceph-mon[30820]: pgmap v8971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:09 smithi116 ceph-mon[35411]: pgmap v8971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:11 smithi114 ceph-mon[30820]: pgmap v8972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:26:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:11 smithi116 ceph-mon[35411]: pgmap v8972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:26:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:13 smithi114 ceph-mon[30820]: pgmap v8973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:26:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:13 smithi116 ceph-mon[35411]: pgmap v8973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:26:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:15 smithi114 ceph-mon[30820]: pgmap v8974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:15 smithi116 ceph-mon[35411]: pgmap v8974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:17.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:17 smithi116 ceph-mon[35411]: pgmap v8975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:26:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:17 smithi114 ceph-mon[30820]: pgmap v8975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:26:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:26:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:26:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:19 smithi114 ceph-mon[30820]: pgmap v8976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:19 smithi116 ceph-mon[35411]: pgmap v8976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-mon[30820]: pgmap v8977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:26:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:26:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:26:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:21.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:26:21.205+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:26:21.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:26:21.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:26:21.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:26:21.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:26:21.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:26:21.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:26:21.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:26:21.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:26:21.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:26:21.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:26:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:26:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:21 smithi116 ceph-mon[35411]: pgmap v8977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:26:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:26:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:26:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:26:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:22 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:26:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:26:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:26:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:26:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:26:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:26:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:26:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:26:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:26:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:26:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:26:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:22 smithi114 ceph-mon[30820]: pgmap v8978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:26:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:22 smithi114 ceph-mon[30820]: pgmap v8979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:22 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:26:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:26:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:26:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:22 smithi116 ceph-mon[35411]: pgmap v8978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:26:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:22 smithi116 ceph-mon[35411]: pgmap v8979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:26:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:24 smithi114 ceph-mon[30820]: pgmap v8980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:26:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:24 smithi116 ceph-mon[35411]: pgmap v8980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:26:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:26 smithi114 ceph-mon[30820]: pgmap v8981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:26:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:26 smithi116 ceph-mon[35411]: pgmap v8981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:26:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:28 smithi114 ceph-mon[30820]: pgmap v8982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:26:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:28 smithi116 ceph-mon[35411]: pgmap v8982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:26:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:30 smithi114 ceph-mon[30820]: pgmap v8983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:26:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:30 smithi116 ceph-mon[35411]: pgmap v8983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:32 smithi114 ceph-mon[30820]: pgmap v8984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:26:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:32 smithi116 ceph-mon[35411]: pgmap v8984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:26:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:34 smithi114 ceph-mon[30820]: pgmap v8985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:34 smithi116 ceph-mon[35411]: pgmap v8985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:36 smithi114 ceph-mon[30820]: pgmap v8986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:26:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:36 smithi116 ceph-mon[35411]: pgmap v8986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:26:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:38 smithi114 ceph-mon[30820]: pgmap v8987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:26:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:38 smithi116 ceph-mon[35411]: pgmap v8987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:26:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:40 smithi114 ceph-mon[30820]: pgmap v8988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:40 smithi116 ceph-mon[35411]: pgmap v8988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:42 smithi114 ceph-mon[30820]: pgmap v8989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:26:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:42 smithi116 ceph-mon[35411]: pgmap v8989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:26:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:44 smithi114 ceph-mon[30820]: pgmap v8990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:44 smithi116 ceph-mon[35411]: pgmap v8990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:45 smithi114 ceph-mon[30820]: pgmap v8991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:26:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:45 smithi116 ceph-mon[35411]: pgmap v8991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:26:48.632 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:48 smithi116 ceph-mon[35411]: pgmap v8992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:26:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:48 smithi114 ceph-mon[30820]: pgmap v8992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:26:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:50 smithi114 ceph-mon[30820]: pgmap v8993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:50 smithi116 ceph-mon[35411]: pgmap v8993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:52 smithi114 ceph-mon[30820]: pgmap v8994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:26:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:52 smithi116 ceph-mon[35411]: pgmap v8994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:26:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:54 smithi114 ceph-mon[30820]: pgmap v8995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:54.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:54 smithi116 ceph-mon[35411]: pgmap v8995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:26:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:56 smithi114 ceph-mon[30820]: pgmap v8996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:26:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:56 smithi116 ceph-mon[35411]: pgmap v8996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:26:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:26:58 smithi114 ceph-mon[30820]: pgmap v8997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:26:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:26:58 smithi116 ceph-mon[35411]: pgmap v8997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:27:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:00 smithi114 ceph-mon[30820]: pgmap v8998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:00 smithi116 ceph-mon[35411]: pgmap v8998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:02 smithi114 ceph-mon[30820]: pgmap v8999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:02 smithi116 ceph-mon[35411]: pgmap v8999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:03 smithi114 ceph-mon[30820]: pgmap v9000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:03 smithi116 ceph-mon[35411]: pgmap v9000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:06 smithi114 ceph-mon[30820]: pgmap v9001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:27:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:06 smithi116 ceph-mon[35411]: pgmap v9001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:27:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:08 smithi114 ceph-mon[30820]: pgmap v9002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:27:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:08 smithi116 ceph-mon[35411]: pgmap v9002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:10 smithi114 ceph-mon[30820]: pgmap v9003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:10 smithi116 ceph-mon[35411]: pgmap v9003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:12 smithi114 ceph-mon[30820]: pgmap v9004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:12 smithi116 ceph-mon[35411]: pgmap v9004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:14 smithi114 ceph-mon[30820]: pgmap v9005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:14 smithi116 ceph-mon[35411]: pgmap v9005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:16 smithi114 ceph-mon[30820]: pgmap v9006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:27:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:16 smithi116 ceph-mon[35411]: pgmap v9006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:27:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:18 smithi114 ceph-mon[30820]: pgmap v9007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:27:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:18 smithi116 ceph-mon[35411]: pgmap v9007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:27:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:20 smithi114 ceph-mon[30820]: pgmap v9008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:20 smithi116 ceph-mon[35411]: pgmap v9008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:22 smithi114 ceph-mon[30820]: pgmap v9009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:27:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:22 smithi116 ceph-mon[35411]: pgmap v9009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:27:24.567 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-mon[30820]: pgmap v9010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:24 smithi116 ceph-mon[35411]: pgmap v9010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:27:24.564+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:27:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:27:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:27:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:27:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:27:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:27:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:27:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:27:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:27:24.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:27:24.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:27:24.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:27:24.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:27:24.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:27:24.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:27:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:27:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:27:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:27:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:27:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:27:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:27:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:27:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:27:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:27:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:27:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:27:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:27:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:27:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:27:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:27:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:27:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:27:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:27:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:27:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:26 smithi114 ceph-mon[30820]: pgmap v9011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:27:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:26 smithi114 ceph-mon[30820]: pgmap v9012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:27:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:27:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:27:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:27:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:27:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:27:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:27:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:27:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:27:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:27:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:27:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:26 smithi116 ceph-mon[35411]: pgmap v9011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:27:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:26 smithi116 ceph-mon[35411]: pgmap v9012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:27:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:28 smithi114 ceph-mon[30820]: pgmap v9013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:27:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:28 smithi116 ceph-mon[35411]: pgmap v9013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:27:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:30 smithi114 ceph-mon[30820]: pgmap v9014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:27:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:30 smithi116 ceph-mon[35411]: pgmap v9014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:27:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:32 smithi114 ceph-mon[30820]: pgmap v9015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:27:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:32 smithi116 ceph-mon[35411]: pgmap v9015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:27:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:34 smithi114 ceph-mon[30820]: pgmap v9016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:27:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:34 smithi116 ceph-mon[35411]: pgmap v9016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:27:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:36 smithi114 ceph-mon[30820]: pgmap v9017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:27:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:36 smithi116 ceph-mon[35411]: pgmap v9017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:27:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:38 smithi114 ceph-mon[30820]: pgmap v9018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:27:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:38 smithi116 ceph-mon[35411]: pgmap v9018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:27:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:40 smithi114 ceph-mon[30820]: pgmap v9019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:40 smithi116 ceph-mon[35411]: pgmap v9019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:42 smithi114 ceph-mon[30820]: pgmap v9020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:42 smithi116 ceph-mon[35411]: pgmap v9020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:44 smithi114 ceph-mon[30820]: pgmap v9021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:44 smithi116 ceph-mon[35411]: pgmap v9021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:46 smithi114 ceph-mon[30820]: pgmap v9022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:27:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:46 smithi116 ceph-mon[35411]: pgmap v9022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:27:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:48 smithi114 ceph-mon[30820]: pgmap v9023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:27:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:48 smithi116 ceph-mon[35411]: pgmap v9023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:27:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:50 smithi114 ceph-mon[30820]: pgmap v9024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:50 smithi116 ceph-mon[35411]: pgmap v9024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:52 smithi114 ceph-mon[30820]: pgmap v9025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:52 smithi116 ceph-mon[35411]: pgmap v9025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:27:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:54 smithi114 ceph-mon[30820]: pgmap v9026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:54 smithi116 ceph-mon[35411]: pgmap v9026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:27:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:56 smithi114 ceph-mon[30820]: pgmap v9027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:27:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:56 smithi116 ceph-mon[35411]: pgmap v9027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:27:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:27:58 smithi114 ceph-mon[30820]: pgmap v9028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:27:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:27:58 smithi116 ceph-mon[35411]: pgmap v9028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:00 smithi114 ceph-mon[30820]: pgmap v9029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:00 smithi116 ceph-mon[35411]: pgmap v9029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:02 smithi114 ceph-mon[30820]: pgmap v9030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:02 smithi116 ceph-mon[35411]: pgmap v9030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:04 smithi114 ceph-mon[30820]: pgmap v9031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:04 smithi116 ceph-mon[35411]: pgmap v9031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:06 smithi114 ceph-mon[30820]: pgmap v9032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:06 smithi116 ceph-mon[35411]: pgmap v9032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:08 smithi114 ceph-mon[30820]: pgmap v9033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:08 smithi116 ceph-mon[35411]: pgmap v9033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:10 smithi114 ceph-mon[30820]: pgmap v9034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:10 smithi116 ceph-mon[35411]: pgmap v9034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:12 smithi114 ceph-mon[30820]: pgmap v9035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:12 smithi116 ceph-mon[35411]: pgmap v9035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:13 smithi114 ceph-mon[30820]: pgmap v9036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:13 smithi116 ceph-mon[35411]: pgmap v9036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:15 smithi114 ceph-mon[30820]: pgmap v9037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:15 smithi116 ceph-mon[35411]: pgmap v9037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:17 smithi114 ceph-mon[30820]: pgmap v9038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:17 smithi116 ceph-mon[35411]: pgmap v9038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:19.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:19 smithi114 ceph-mon[30820]: pgmap v9039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:19 smithi116 ceph-mon[35411]: pgmap v9039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:21 smithi114 ceph-mon[30820]: pgmap v9040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:21 smithi116 ceph-mon[35411]: pgmap v9040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:23 smithi114 ceph-mon[30820]: pgmap v9041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:23 smithi116 ceph-mon[35411]: pgmap v9041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:25 smithi114 ceph-mon[30820]: pgmap v9042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:28:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:25 smithi116 ceph-mon[35411]: pgmap v9042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:28:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-mon[30820]: pgmap v9043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:27 smithi116 ceph-mon[35411]: pgmap v9043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:28:27.925+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:28:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:28:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:28:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:28:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:28:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:28:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:28:28.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:28:28.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:28:28.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:28:28.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:28:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:28:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:28:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:28:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:28:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:28:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:28 smithi116 ceph-mon[35411]: pgmap v9044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:28 smithi116 ceph-mon[35411]: pgmap v9045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:28:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:28:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:28:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:28:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:28:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:28:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:28:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:28:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:28 smithi114 ceph-mon[30820]: pgmap v9044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:28 smithi114 ceph-mon[30820]: pgmap v9045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:28:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:28:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:30 smithi114 ceph-mon[30820]: pgmap v9046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:28:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:30 smithi116 ceph-mon[35411]: pgmap v9046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:28:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:32 smithi114 ceph-mon[30820]: pgmap v9047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:28:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:32 smithi116 ceph-mon[35411]: pgmap v9047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:28:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:34 smithi114 ceph-mon[30820]: pgmap v9048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:28:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:34 smithi116 ceph-mon[35411]: pgmap v9048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:28:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:36 smithi114 ceph-mon[30820]: pgmap v9049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:28:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:36 smithi116 ceph-mon[35411]: pgmap v9049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:28:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:38 smithi114 ceph-mon[30820]: pgmap v9050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:28:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:39 smithi116 ceph-mon[35411]: pgmap v9050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:28:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:41 smithi114 ceph-mon[30820]: pgmap v9051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:41 smithi116 ceph-mon[35411]: pgmap v9051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:43 smithi114 ceph-mon[30820]: pgmap v9052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:43 smithi116 ceph-mon[35411]: pgmap v9052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:45 smithi114 ceph-mon[30820]: pgmap v9053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:45 smithi116 ceph-mon[35411]: pgmap v9053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:47 smithi114 ceph-mon[30820]: pgmap v9054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:47 smithi116 ceph-mon[35411]: pgmap v9054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:49 smithi114 ceph-mon[30820]: pgmap v9055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:49.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:49 smithi116 ceph-mon[35411]: pgmap v9055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:51 smithi114 ceph-mon[30820]: pgmap v9056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:51 smithi116 ceph-mon[35411]: pgmap v9056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:28:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:53 smithi114 ceph-mon[30820]: pgmap v9057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:53 smithi116 ceph-mon[35411]: pgmap v9057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:28:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:55 smithi114 ceph-mon[30820]: pgmap v9058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:55 smithi116 ceph-mon[35411]: pgmap v9058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:57 smithi114 ceph-mon[30820]: pgmap v9059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:57 smithi116 ceph-mon[35411]: pgmap v9059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:28:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:28:59 smithi114 ceph-mon[30820]: pgmap v9060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:28:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:28:59 smithi116 ceph-mon[35411]: pgmap v9060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:01 smithi114 ceph-mon[30820]: pgmap v9061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:29:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:01 smithi116 ceph-mon[35411]: pgmap v9061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:29:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:03 smithi114 ceph-mon[30820]: pgmap v9062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:29:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:03 smithi116 ceph-mon[35411]: pgmap v9062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:29:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:05 smithi114 ceph-mon[30820]: pgmap v9063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:05 smithi116 ceph-mon[35411]: pgmap v9063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:07 smithi114 ceph-mon[30820]: pgmap v9064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:29:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:07 smithi116 ceph-mon[35411]: pgmap v9064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:29:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:09 smithi114 ceph-mon[30820]: pgmap v9065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:09 smithi116 ceph-mon[35411]: pgmap v9065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:11 smithi114 ceph-mon[30820]: pgmap v9066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:29:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:11 smithi116 ceph-mon[35411]: pgmap v9066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:29:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:13 smithi114 ceph-mon[30820]: pgmap v9067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:29:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:13 smithi116 ceph-mon[35411]: pgmap v9067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:29:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:15 smithi114 ceph-mon[30820]: pgmap v9068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:15 smithi116 ceph-mon[35411]: pgmap v9068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:17 smithi114 ceph-mon[30820]: pgmap v9069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:29:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:17 smithi116 ceph-mon[35411]: pgmap v9069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:29:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:19 smithi114 ceph-mon[30820]: pgmap v9070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:19 smithi116 ceph-mon[35411]: pgmap v9070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:21.089 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:21 smithi114 ceph-mon[30820]: pgmap v9071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:29:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:21 smithi116 ceph-mon[35411]: pgmap v9071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:29:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:23 smithi114 ceph-mon[30820]: pgmap v9072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:29:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:23 smithi116 ceph-mon[35411]: pgmap v9072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:29:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:25 smithi114 ceph-mon[30820]: pgmap v9073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:25 smithi116 ceph-mon[35411]: pgmap v9073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:27 smithi114 ceph-mon[30820]: pgmap v9074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:29:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:27 smithi116 ceph-mon[35411]: pgmap v9074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:29:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:29:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:29:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:29 smithi114 ceph-mon[30820]: pgmap v9075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:29 smithi116 ceph-mon[35411]: pgmap v9075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-mon[30820]: pgmap v9076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:29:31.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:29:31.284+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:29:31.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:29:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:29:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:29:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:29:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:29:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:29:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:29:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:29:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:29:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:29:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:31 smithi116 ceph-mon[35411]: pgmap v9076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:29:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:32 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:29:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:29:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:29:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:29:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:32 smithi114 ceph-mon[30820]: pgmap v9077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:29:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:32 smithi114 ceph-mon[30820]: pgmap v9078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:29:32.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:29:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:32 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:29:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:29:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:29:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:29:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:29:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:29:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:29:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:29:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:29:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:29:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:29:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:29:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:32 smithi116 ceph-mon[35411]: pgmap v9077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:29:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:32 smithi116 ceph-mon[35411]: pgmap v9078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:29:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:29:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:34 smithi114 ceph-mon[30820]: pgmap v9079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:29:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:34 smithi116 ceph-mon[35411]: pgmap v9079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:29:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:36 smithi114 ceph-mon[30820]: pgmap v9080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:29:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:36 smithi116 ceph-mon[35411]: pgmap v9080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:29:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:38 smithi114 ceph-mon[30820]: pgmap v9081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:29:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:38 smithi116 ceph-mon[35411]: pgmap v9081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:29:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:40 smithi114 ceph-mon[30820]: pgmap v9082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:29:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:40 smithi116 ceph-mon[35411]: pgmap v9082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:29:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:42 smithi114 ceph-mon[30820]: pgmap v9083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:29:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:42 smithi116 ceph-mon[35411]: pgmap v9083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:29:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:44 smithi114 ceph-mon[30820]: pgmap v9084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:44 smithi116 ceph-mon[35411]: pgmap v9084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:46 smithi114 ceph-mon[30820]: pgmap v9085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:29:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:46 smithi116 ceph-mon[35411]: pgmap v9085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:29:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:48 smithi114 ceph-mon[30820]: pgmap v9086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:29:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:48 smithi116 ceph-mon[35411]: pgmap v9086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:29:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:50 smithi114 ceph-mon[30820]: pgmap v9087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:50 smithi116 ceph-mon[35411]: pgmap v9087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:52 smithi114 ceph-mon[30820]: pgmap v9088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:29:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:52 smithi116 ceph-mon[35411]: pgmap v9088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:29:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:54 smithi114 ceph-mon[30820]: pgmap v9089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:54 smithi116 ceph-mon[35411]: pgmap v9089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:29:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:56 smithi114 ceph-mon[30820]: pgmap v9090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:29:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:56 smithi116 ceph-mon[35411]: pgmap v9090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:29:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:29:58 smithi114 ceph-mon[30820]: pgmap v9091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:29:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:29:58 smithi116 ceph-mon[35411]: pgmap v9091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:30:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:00 smithi114 ceph-mon[30820]: pgmap v9092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:30:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:00 smithi116 ceph-mon[35411]: pgmap v9092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:30:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:02 smithi114 ceph-mon[30820]: pgmap v9093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:02 smithi116 ceph-mon[35411]: pgmap v9093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:04 smithi114 ceph-mon[30820]: pgmap v9094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:04 smithi116 ceph-mon[35411]: pgmap v9094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:06 smithi114 ceph-mon[30820]: pgmap v9095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:30:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:06 smithi116 ceph-mon[35411]: pgmap v9095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:30:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:08 smithi114 ceph-mon[30820]: pgmap v9096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:30:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:08 smithi116 ceph-mon[35411]: pgmap v9096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:30:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:10 smithi114 ceph-mon[30820]: pgmap v9097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:10 smithi116 ceph-mon[35411]: pgmap v9097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:12 smithi114 ceph-mon[30820]: pgmap v9098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:12 smithi116 ceph-mon[35411]: pgmap v9098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:14 smithi116 ceph-mon[35411]: pgmap v9099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:14 smithi114 ceph-mon[30820]: pgmap v9099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:16 smithi116 ceph-mon[35411]: pgmap v9100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:30:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:16 smithi114 ceph-mon[30820]: pgmap v9100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:30:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:18 smithi116 ceph-mon[35411]: pgmap v9101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:30:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:18 smithi114 ceph-mon[30820]: pgmap v9101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:20 smithi114 ceph-mon[30820]: pgmap v9102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:20 smithi116 ceph-mon[35411]: pgmap v9102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:22 smithi114 ceph-mon[30820]: pgmap v9103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:22.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:22 smithi116 ceph-mon[35411]: pgmap v9103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:23 smithi114 ceph-mon[30820]: pgmap v9104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:23 smithi116 ceph-mon[35411]: pgmap v9104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:26 smithi114 ceph-mon[30820]: pgmap v9105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:30:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:26 smithi116 ceph-mon[35411]: pgmap v9105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:30:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:28 smithi114 ceph-mon[30820]: pgmap v9106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:30:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:28 smithi116 ceph-mon[35411]: pgmap v9106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:30:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:30 smithi114 ceph-mon[30820]: pgmap v9107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:30 smithi116 ceph-mon[35411]: pgmap v9107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:32 smithi114 ceph-mon[30820]: pgmap v9108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:30:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:32 smithi116 ceph-mon[35411]: pgmap v9108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:30:34.648 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-mon[30820]: pgmap v9109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:34 smithi116 ceph-mon[35411]: pgmap v9109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:30:34.645+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:30:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:30:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:30:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:30:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:30:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:30:35.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:30:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:30:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:30:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:30:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:30:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:30:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:30:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:36 smithi114 ceph-mon[30820]: pgmap v9110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:30:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:36 smithi114 ceph-mon[30820]: pgmap v9111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:30:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:30:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:30:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:30:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:36 smithi116 ceph-mon[35411]: pgmap v9110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:30:36.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:36 smithi116 ceph-mon[35411]: pgmap v9111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:30:38.627 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:38 smithi114 ceph-mon[30820]: pgmap v9112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:30:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:38 smithi116 ceph-mon[35411]: pgmap v9112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:30:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:40 smithi114 ceph-mon[30820]: pgmap v9113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:30:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:40 smithi116 ceph-mon[35411]: pgmap v9113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:30:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:42 smithi114 ceph-mon[30820]: pgmap v9114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:30:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:42 smithi116 ceph-mon[35411]: pgmap v9114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:30:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:44 smithi114 ceph-mon[30820]: pgmap v9115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:30:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:44 smithi116 ceph-mon[35411]: pgmap v9115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:30:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:46 smithi114 ceph-mon[30820]: pgmap v9116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:46 smithi116 ceph-mon[35411]: pgmap v9116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:30:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:48 smithi114 ceph-mon[30820]: pgmap v9117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:30:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:48 smithi116 ceph-mon[35411]: pgmap v9117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:30:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:50 smithi114 ceph-mon[30820]: pgmap v9118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:50 smithi116 ceph-mon[35411]: pgmap v9118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:52 smithi116 ceph-mon[35411]: pgmap v9119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:52 smithi114 ceph-mon[30820]: pgmap v9119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:30:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:54 smithi116 ceph-mon[35411]: pgmap v9120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:54 smithi114 ceph-mon[30820]: pgmap v9120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:30:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:56 smithi116 ceph-mon[35411]: pgmap v9121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:30:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:56 smithi114 ceph-mon[30820]: pgmap v9121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:30:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:30:58 smithi114 ceph-mon[30820]: pgmap v9122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:30:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:30:58 smithi116 ceph-mon[35411]: pgmap v9122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:00 smithi114 ceph-mon[30820]: pgmap v9123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:00 smithi116 ceph-mon[35411]: pgmap v9123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:02 smithi114 ceph-mon[30820]: pgmap v9124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:02 smithi116 ceph-mon[35411]: pgmap v9124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:04 smithi114 ceph-mon[30820]: pgmap v9125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:04 smithi116 ceph-mon[35411]: pgmap v9125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:05 smithi114 ceph-mon[30820]: pgmap v9126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:05 smithi116 ceph-mon[35411]: pgmap v9126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:07 smithi114 ceph-mon[30820]: pgmap v9127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:07 smithi116 ceph-mon[35411]: pgmap v9127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:09 smithi114 ceph-mon[30820]: pgmap v9128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:09 smithi116 ceph-mon[35411]: pgmap v9128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:11 smithi114 ceph-mon[30820]: pgmap v9129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:11 smithi116 ceph-mon[35411]: pgmap v9129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:13 smithi114 ceph-mon[30820]: pgmap v9130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:13 smithi116 ceph-mon[35411]: pgmap v9130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:15 smithi114 ceph-mon[30820]: pgmap v9131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:15 smithi116 ceph-mon[35411]: pgmap v9131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:17 smithi114 ceph-mon[30820]: pgmap v9132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:17 smithi116 ceph-mon[35411]: pgmap v9132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:19 smithi114 ceph-mon[30820]: pgmap v9133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:19 smithi116 ceph-mon[35411]: pgmap v9133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:21.998 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:21 smithi116 ceph-mon[35411]: pgmap v9134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:21 smithi114 ceph-mon[30820]: pgmap v9134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:23 smithi114 ceph-mon[30820]: pgmap v9135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:23 smithi116 ceph-mon[35411]: pgmap v9135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:25 smithi114 ceph-mon[30820]: pgmap v9136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:25 smithi116 ceph-mon[35411]: pgmap v9136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:27 smithi114 ceph-mon[30820]: pgmap v9137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:27 smithi116 ceph-mon[35411]: pgmap v9137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:29 smithi114 ceph-mon[30820]: pgmap v9138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:29 smithi116 ceph-mon[35411]: pgmap v9138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:31 smithi114 ceph-mon[30820]: pgmap v9139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:31 smithi116 ceph-mon[35411]: pgmap v9139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:33 smithi114 ceph-mon[30820]: pgmap v9140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:33 smithi116 ceph-mon[35411]: pgmap v9140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:35 smithi114 ceph-mon[30820]: pgmap v9141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:31:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:35 smithi116 ceph-mon[35411]: pgmap v9141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:31:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:37 smithi114 ceph-mon[30820]: pgmap v9142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:31:38.005+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:31:38.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:31:38.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:31:38.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:31:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:37 smithi116 ceph-mon[35411]: pgmap v9142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:31:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:31:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:31:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:31:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:31:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-mon[30820]: pgmap v9143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-mon[30820]: pgmap v9144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:31:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:38 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:31:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:31:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:31:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:31:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:38 smithi116 ceph-mon[35411]: pgmap v9143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:31:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:38 smithi116 ceph-mon[35411]: pgmap v9144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:31:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:31:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:41 smithi114 ceph-mon[30820]: pgmap v9145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:31:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:41 smithi116 ceph-mon[35411]: pgmap v9145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:31:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:43 smithi114 ceph-mon[30820]: pgmap v9146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:31:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:43 smithi116 ceph-mon[35411]: pgmap v9146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:31:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:45 smithi114 ceph-mon[30820]: pgmap v9147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:31:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:45 smithi116 ceph-mon[35411]: pgmap v9147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:31:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:47 smithi114 ceph-mon[30820]: pgmap v9148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:31:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:47 smithi116 ceph-mon[35411]: pgmap v9148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:31:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:49 smithi114 ceph-mon[30820]: pgmap v9149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:31:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:49 smithi116 ceph-mon[35411]: pgmap v9149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:31:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:51 smithi114 ceph-mon[30820]: pgmap v9150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:51 smithi116 ceph-mon[35411]: pgmap v9150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:31:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:53 smithi114 ceph-mon[30820]: pgmap v9151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:53 smithi116 ceph-mon[35411]: pgmap v9151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:31:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:55 smithi114 ceph-mon[30820]: pgmap v9152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:55 smithi116 ceph-mon[35411]: pgmap v9152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:57 smithi114 ceph-mon[30820]: pgmap v9153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:57 smithi116 ceph-mon[35411]: pgmap v9153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:31:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:31:59 smithi114 ceph-mon[30820]: pgmap v9154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:31:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:31:59 smithi116 ceph-mon[35411]: pgmap v9154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:01 smithi114 ceph-mon[30820]: pgmap v9155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:01 smithi116 ceph-mon[35411]: pgmap v9155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:03 smithi114 ceph-mon[30820]: pgmap v9156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:32:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:03 smithi116 ceph-mon[35411]: pgmap v9156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:32:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:05 smithi114 ceph-mon[30820]: pgmap v9157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:05 smithi116 ceph-mon[35411]: pgmap v9157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:07 smithi114 ceph-mon[30820]: pgmap v9158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:32:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:07 smithi116 ceph-mon[35411]: pgmap v9158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:32:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:09 smithi114 ceph-mon[30820]: pgmap v9159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:09 smithi116 ceph-mon[35411]: pgmap v9159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:11 smithi114 ceph-mon[30820]: pgmap v9160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:11 smithi116 ceph-mon[35411]: pgmap v9160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:13 smithi114 ceph-mon[30820]: pgmap v9161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:32:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:13 smithi116 ceph-mon[35411]: pgmap v9161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:32:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:15 smithi114 ceph-mon[30820]: pgmap v9162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:15 smithi116 ceph-mon[35411]: pgmap v9162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:17 smithi114 ceph-mon[30820]: pgmap v9163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:32:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:17 smithi116 ceph-mon[35411]: pgmap v9163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:32:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:19 smithi114 ceph-mon[30820]: pgmap v9164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:19 smithi116 ceph-mon[35411]: pgmap v9164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:21 smithi114 ceph-mon[30820]: pgmap v9165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:21 smithi116 ceph-mon[35411]: pgmap v9165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:23 smithi114 ceph-mon[30820]: pgmap v9166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:32:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:23 smithi116 ceph-mon[35411]: pgmap v9166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:32:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:25 smithi114 ceph-mon[30820]: pgmap v9167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:25 smithi116 ceph-mon[35411]: pgmap v9167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:27 smithi114 ceph-mon[30820]: pgmap v9168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:32:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:27 smithi116 ceph-mon[35411]: pgmap v9168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:32:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:29 smithi114 ceph-mon[30820]: pgmap v9169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:29 smithi116 ceph-mon[35411]: pgmap v9169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:31 smithi114 ceph-mon[30820]: pgmap v9170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:31 smithi116 ceph-mon[35411]: pgmap v9170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:33 smithi116 ceph-mon[35411]: pgmap v9171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:32:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:33 smithi114 ceph-mon[30820]: pgmap v9171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:32:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:35 smithi116 ceph-mon[35411]: pgmap v9172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:35 smithi114 ceph-mon[30820]: pgmap v9172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:37 smithi116 ceph-mon[35411]: pgmap v9173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:32:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:37 smithi114 ceph-mon[30820]: pgmap v9173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:32:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:32:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:32:39.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: pgmap v9174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:39.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.569 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: pgmap v9174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:32:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:39.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:32:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:40 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:32:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:40 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:32:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:40 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:32:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:40 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:32:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:40 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:32:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:40 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:32:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:40 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:32:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:40 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:32:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-mon[30820]: pgmap v9175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:32:41.364+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:32:41.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:32:41.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:32:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:32:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:41 smithi116 ceph-mon[35411]: pgmap v9175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:32:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:42 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:32:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:32:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:32:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:32:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:32:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:42 smithi114 ceph-mon[30820]: pgmap v9176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:32:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:42 smithi114 ceph-mon[30820]: pgmap v9177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:42.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:32:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:42 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:32:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:32:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:32:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:32:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:32:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:32:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:32:42.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:42 smithi116 ceph-mon[35411]: pgmap v9176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:42 smithi116 ceph-mon[35411]: pgmap v9177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:42.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:32:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:44 smithi116 ceph-mon[35411]: pgmap v9178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:44 smithi114 ceph-mon[30820]: pgmap v9178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:46 smithi114 ceph-mon[30820]: pgmap v9179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:46 smithi116 ceph-mon[35411]: pgmap v9179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:48 smithi114 ceph-mon[30820]: pgmap v9180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:48 smithi116 ceph-mon[35411]: pgmap v9180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:50 smithi114 ceph-mon[30820]: pgmap v9181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:50 smithi116 ceph-mon[35411]: pgmap v9181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:32:52.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:52 smithi116 ceph-mon[35411]: pgmap v9182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T11:32:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:52 smithi114 ceph-mon[30820]: pgmap v9182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T11:32:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:54 smithi114 ceph-mon[30820]: pgmap v9183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-09-19T11:32:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:54 smithi116 ceph-mon[35411]: pgmap v9183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-09-19T11:32:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:56 smithi114 ceph-mon[30820]: pgmap v9184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2024-09-19T11:32:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:56 smithi116 ceph-mon[35411]: pgmap v9184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 7 op/s 2024-09-19T11:32:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:32:58 smithi114 ceph-mon[30820]: pgmap v9185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-09-19T11:32:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:32:58 smithi116 ceph-mon[35411]: pgmap v9185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-09-19T11:33:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:00 smithi114 ceph-mon[30820]: pgmap v9186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 6 op/s 2024-09-19T11:33:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:00 smithi116 ceph-mon[35411]: pgmap v9186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 6 op/s 2024-09-19T11:33:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:02 smithi114 ceph-mon[30820]: pgmap v9187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 6 op/s 2024-09-19T11:33:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:02 smithi116 ceph-mon[35411]: pgmap v9187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 6 op/s 2024-09-19T11:33:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:04 smithi114 ceph-mon[30820]: pgmap v9188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:04 smithi116 ceph-mon[35411]: pgmap v9188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:06 smithi114 ceph-mon[30820]: pgmap v9189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:33:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:06 smithi116 ceph-mon[35411]: pgmap v9189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:33:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:08 smithi114 ceph-mon[30820]: pgmap v9190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:08 smithi116 ceph-mon[35411]: pgmap v9190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:10.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:10 smithi114 ceph-mon[30820]: pgmap v9191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:33:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:10 smithi116 ceph-mon[35411]: pgmap v9191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:33:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:12 smithi114 ceph-mon[30820]: pgmap v9192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:12 smithi116 ceph-mon[35411]: pgmap v9192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:14 smithi114 ceph-mon[30820]: pgmap v9193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:14 smithi116 ceph-mon[35411]: pgmap v9193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:16 smithi114 ceph-mon[30820]: pgmap v9194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:33:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:16 smithi116 ceph-mon[35411]: pgmap v9194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:33:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:18 smithi114 ceph-mon[30820]: pgmap v9195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:18 smithi116 ceph-mon[35411]: pgmap v9195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:20 smithi114 ceph-mon[30820]: pgmap v9196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:33:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:20 smithi116 ceph-mon[35411]: pgmap v9196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:33:22.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:22 smithi116 ceph-mon[35411]: pgmap v9197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:22 smithi114 ceph-mon[30820]: pgmap v9197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:23 smithi114 ceph-mon[30820]: pgmap v9198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:23 smithi116 ceph-mon[35411]: pgmap v9198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:26 smithi114 ceph-mon[30820]: pgmap v9199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:33:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:26 smithi116 ceph-mon[35411]: pgmap v9199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:33:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:28 smithi114 ceph-mon[30820]: pgmap v9200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:28 smithi116 ceph-mon[35411]: pgmap v9200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:30 smithi114 ceph-mon[30820]: pgmap v9201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:33:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:30 smithi116 ceph-mon[35411]: pgmap v9201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:33:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:32 smithi114 ceph-mon[30820]: pgmap v9202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:32 smithi116 ceph-mon[35411]: pgmap v9202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:34 smithi114 ceph-mon[30820]: pgmap v9203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:34 smithi116 ceph-mon[35411]: pgmap v9203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:36 smithi114 ceph-mon[30820]: pgmap v9204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:33:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:36 smithi116 ceph-mon[35411]: pgmap v9204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:33:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:38 smithi114 ceph-mon[30820]: pgmap v9205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:38 smithi116 ceph-mon[35411]: pgmap v9205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:39 smithi114 ceph-mon[30820]: pgmap v9206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:33:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:39 smithi116 ceph-mon[35411]: pgmap v9206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:33:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:42 smithi114 ceph-mon[30820]: pgmap v9207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:33:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:42 smithi116 ceph-mon[35411]: pgmap v9207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:33:44.727 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-mon[30820]: pgmap v9208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:44 smithi116 ceph-mon[35411]: pgmap v9208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:33:45.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:33:44.724+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:33:45.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:33:45.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:33:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:33:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:33:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:33:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:33:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:33:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:33:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:33:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:33:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:33:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:33:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:46 smithi114 ceph-mon[30820]: pgmap v9209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:33:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:46 smithi114 ceph-mon[30820]: pgmap v9210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:33:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:33:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:33:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:33:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:33:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:33:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:33:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:33:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:33:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:33:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:33:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:46 smithi116 ceph-mon[35411]: pgmap v9209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:33:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:46 smithi116 ceph-mon[35411]: pgmap v9210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:33:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:48 smithi114 ceph-mon[30820]: pgmap v9211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:33:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:48 smithi116 ceph-mon[35411]: pgmap v9211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:33:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:50 smithi114 ceph-mon[30820]: pgmap v9212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:33:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:50 smithi116 ceph-mon[35411]: pgmap v9212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:33:52.730 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:52 smithi116 ceph-mon[35411]: pgmap v9213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:33:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:52 smithi114 ceph-mon[30820]: pgmap v9213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:33:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:53 smithi114 ceph-mon[30820]: pgmap v9214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:33:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:53 smithi116 ceph-mon[35411]: pgmap v9214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:33:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:55 smithi114 ceph-mon[30820]: pgmap v9215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:33:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:55 smithi116 ceph-mon[35411]: pgmap v9215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:33:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:57 smithi114 ceph-mon[30820]: pgmap v9216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:33:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:57 smithi116 ceph-mon[35411]: pgmap v9216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:33:59 smithi114 ceph-mon[30820]: pgmap v9217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:33:59 smithi116 ceph-mon[35411]: pgmap v9217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:01 smithi114 ceph-mon[30820]: pgmap v9218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:01 smithi116 ceph-mon[35411]: pgmap v9218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:03 smithi114 ceph-mon[30820]: pgmap v9219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:03 smithi116 ceph-mon[35411]: pgmap v9219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:05 smithi114 ceph-mon[30820]: pgmap v9220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:34:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:05 smithi116 ceph-mon[35411]: pgmap v9220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:34:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:07 smithi114 ceph-mon[30820]: pgmap v9221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:07 smithi116 ceph-mon[35411]: pgmap v9221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:09 smithi114 ceph-mon[30820]: pgmap v9222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:09 smithi116 ceph-mon[35411]: pgmap v9222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:11 smithi114 ceph-mon[30820]: pgmap v9223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:11 smithi116 ceph-mon[35411]: pgmap v9223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:13 smithi114 ceph-mon[30820]: pgmap v9224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:13 smithi116 ceph-mon[35411]: pgmap v9224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:15 smithi114 ceph-mon[30820]: pgmap v9225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:34:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:15 smithi116 ceph-mon[35411]: pgmap v9225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:34:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:17 smithi114 ceph-mon[30820]: pgmap v9226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:17 smithi116 ceph-mon[35411]: pgmap v9226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:19 smithi114 ceph-mon[30820]: pgmap v9227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:19 smithi116 ceph-mon[35411]: pgmap v9227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:21 smithi114 ceph-mon[30820]: pgmap v9228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:21 smithi116 ceph-mon[35411]: pgmap v9228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:23 smithi114 ceph-mon[30820]: pgmap v9229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:23 smithi116 ceph-mon[35411]: pgmap v9229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:25 smithi114 ceph-mon[30820]: pgmap v9230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:34:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:25 smithi116 ceph-mon[35411]: pgmap v9230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:34:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:27 smithi114 ceph-mon[30820]: pgmap v9231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:27 smithi116 ceph-mon[35411]: pgmap v9231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:29 smithi114 ceph-mon[30820]: pgmap v9232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:29 smithi116 ceph-mon[35411]: pgmap v9232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:31 smithi114 ceph-mon[30820]: pgmap v9233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:31 smithi116 ceph-mon[35411]: pgmap v9233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:33 smithi114 ceph-mon[30820]: pgmap v9234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:33 smithi116 ceph-mon[35411]: pgmap v9234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:35 smithi114 ceph-mon[30820]: pgmap v9235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:34:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:35 smithi116 ceph-mon[35411]: pgmap v9235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:34:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:37 smithi114 ceph-mon[30820]: pgmap v9236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:37 smithi116 ceph-mon[35411]: pgmap v9236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:39 smithi114 ceph-mon[30820]: pgmap v9237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:39 smithi116 ceph-mon[35411]: pgmap v9237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:34:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:41 smithi114 ceph-mon[30820]: pgmap v9238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:41 smithi116 ceph-mon[35411]: pgmap v9238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:34:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:43 smithi114 ceph-mon[30820]: pgmap v9239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:34:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:43 smithi116 ceph-mon[35411]: pgmap v9239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:34:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:34:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:34:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:45 smithi114 ceph-mon[30820]: pgmap v9240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:34:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:45 smithi116 ceph-mon[35411]: pgmap v9240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:34:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:47 smithi114 ceph-mon[30820]: pgmap v9241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:34:48.084+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:34:48.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:34:48.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:34:48.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:34:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:47 smithi116 ceph-mon[35411]: pgmap v9241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:34:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:34:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:34:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:34:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:34:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:34:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-mon[30820]: pgmap v9242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:34:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-mon[30820]: pgmap v9243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:34:49.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:34:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:48 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:34:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:34:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:34:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:34:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:34:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:34:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:34:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:34:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:34:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:34:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:34:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:34:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:48 smithi116 ceph-mon[35411]: pgmap v9242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:34:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:48 smithi116 ceph-mon[35411]: pgmap v9243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:34:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:34:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:51 smithi114 ceph-mon[30820]: pgmap v9244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:34:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:51 smithi116 ceph-mon[35411]: pgmap v9244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:34:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:53 smithi114 ceph-mon[30820]: pgmap v9245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:34:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:53 smithi116 ceph-mon[35411]: pgmap v9245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:34:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:55 smithi114 ceph-mon[30820]: pgmap v9246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:34:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:55 smithi116 ceph-mon[35411]: pgmap v9246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:34:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:57 smithi114 ceph-mon[30820]: pgmap v9247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:34:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:57 smithi116 ceph-mon[35411]: pgmap v9247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:34:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:34:59 smithi114 ceph-mon[30820]: pgmap v9248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:34:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:34:59 smithi116 ceph-mon[35411]: pgmap v9248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:35:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:01 smithi114 ceph-mon[30820]: pgmap v9249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:01 smithi116 ceph-mon[35411]: pgmap v9249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:03 smithi116 ceph-mon[35411]: pgmap v9250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:03 smithi114 ceph-mon[30820]: pgmap v9250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:05 smithi116 ceph-mon[35411]: pgmap v9251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:05 smithi114 ceph-mon[30820]: pgmap v9251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:07 smithi116 ceph-mon[35411]: pgmap v9252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:07 smithi114 ceph-mon[30820]: pgmap v9252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:09 smithi114 ceph-mon[30820]: pgmap v9253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:09 smithi116 ceph-mon[35411]: pgmap v9253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:11 smithi114 ceph-mon[30820]: pgmap v9254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:11 smithi116 ceph-mon[35411]: pgmap v9254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:13.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:13 smithi114 ceph-mon[30820]: pgmap v9255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:13 smithi116 ceph-mon[35411]: pgmap v9255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:15 smithi114 ceph-mon[30820]: pgmap v9256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:15 smithi116 ceph-mon[35411]: pgmap v9256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:17 smithi114 ceph-mon[30820]: pgmap v9257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:17 smithi116 ceph-mon[35411]: pgmap v9257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:19 smithi114 ceph-mon[30820]: pgmap v9258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:19 smithi116 ceph-mon[35411]: pgmap v9258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:21 smithi114 ceph-mon[30820]: pgmap v9259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:21 smithi116 ceph-mon[35411]: pgmap v9259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:23.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:23 smithi116 ceph-mon[35411]: pgmap v9260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:23 smithi114 ceph-mon[30820]: pgmap v9260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:25 smithi114 ceph-mon[30820]: pgmap v9261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:25 smithi116 ceph-mon[35411]: pgmap v9261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:27 smithi114 ceph-mon[30820]: pgmap v9262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:27 smithi116 ceph-mon[35411]: pgmap v9262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:29 smithi114 ceph-mon[30820]: pgmap v9263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:29 smithi116 ceph-mon[35411]: pgmap v9263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:31 smithi114 ceph-mon[30820]: pgmap v9264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:31 smithi116 ceph-mon[35411]: pgmap v9264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:33 smithi114 ceph-mon[30820]: pgmap v9265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:33 smithi116 ceph-mon[35411]: pgmap v9265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:35 smithi114 ceph-mon[30820]: pgmap v9266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:35.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:35 smithi116 ceph-mon[35411]: pgmap v9266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:37 smithi114 ceph-mon[30820]: pgmap v9267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:37 smithi116 ceph-mon[35411]: pgmap v9267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:39 smithi114 ceph-mon[30820]: pgmap v9268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:39 smithi116 ceph-mon[35411]: pgmap v9268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:41 smithi114 ceph-mon[30820]: pgmap v9269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:41 smithi116 ceph-mon[35411]: pgmap v9269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:43 smithi114 ceph-mon[30820]: pgmap v9270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:43 smithi116 ceph-mon[35411]: pgmap v9270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:35:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:45 smithi114 ceph-mon[30820]: pgmap v9271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:45 smithi116 ceph-mon[35411]: pgmap v9271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:47 smithi114 ceph-mon[30820]: pgmap v9272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:47 smithi116 ceph-mon[35411]: pgmap v9272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:35:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:35:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:35:49.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:49 smithi114 ceph-mon[30820]: pgmap v9273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:49 smithi116 ceph-mon[35411]: pgmap v9273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:35:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:50 smithi114 ceph-mon[30820]: pgmap v9274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:50 smithi116 ceph-mon[35411]: pgmap v9274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:35:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:35:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:35:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:35:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-mon[30820]: pgmap v9275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-mon[30820]: pgmap v9276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:35:51.444+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:35:51.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:35:51.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:35:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:35:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:35:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:35:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:35:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:35:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:35:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:35:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:35:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:35:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:35:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:35:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:35:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:35:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:35:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:51 smithi116 ceph-mon[35411]: pgmap v9275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:35:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:51 smithi116 ceph-mon[35411]: pgmap v9276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:35:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:53 smithi114 ceph-mon[30820]: pgmap v9277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:53 smithi116 ceph-mon[35411]: pgmap v9277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:55 smithi114 ceph-mon[30820]: pgmap v9278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:55 smithi116 ceph-mon[35411]: pgmap v9278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:57 smithi114 ceph-mon[30820]: pgmap v9279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:57 smithi116 ceph-mon[35411]: pgmap v9279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:35:59 smithi114 ceph-mon[30820]: pgmap v9280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:35:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:35:59 smithi116 ceph-mon[35411]: pgmap v9280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:36:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:01 smithi114 ceph-mon[30820]: pgmap v9281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:36:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:01 smithi116 ceph-mon[35411]: pgmap v9281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:36:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:03 smithi114 ceph-mon[30820]: pgmap v9282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:03 smithi116 ceph-mon[35411]: pgmap v9282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:05 smithi114 ceph-mon[30820]: pgmap v9283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:05 smithi116 ceph-mon[35411]: pgmap v9283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:07 smithi114 ceph-mon[30820]: pgmap v9284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:07 smithi116 ceph-mon[35411]: pgmap v9284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:09 smithi114 ceph-mon[30820]: pgmap v9285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:09 smithi116 ceph-mon[35411]: pgmap v9285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:11 smithi114 ceph-mon[30820]: pgmap v9286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:11 smithi116 ceph-mon[35411]: pgmap v9286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:13 smithi114 ceph-mon[30820]: pgmap v9287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:13 smithi116 ceph-mon[35411]: pgmap v9287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:15 smithi114 ceph-mon[30820]: pgmap v9288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:15 smithi116 ceph-mon[35411]: pgmap v9288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:17 smithi114 ceph-mon[30820]: pgmap v9289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:17 smithi116 ceph-mon[35411]: pgmap v9289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:19 smithi114 ceph-mon[30820]: pgmap v9290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:19 smithi116 ceph-mon[35411]: pgmap v9290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:21 smithi114 ceph-mon[30820]: pgmap v9291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:21 smithi116 ceph-mon[35411]: pgmap v9291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:23 smithi114 ceph-mon[30820]: pgmap v9292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:23 smithi116 ceph-mon[35411]: pgmap v9292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:25 smithi114 ceph-mon[30820]: pgmap v9293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:25 smithi116 ceph-mon[35411]: pgmap v9293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:27 smithi114 ceph-mon[30820]: pgmap v9294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:27 smithi116 ceph-mon[35411]: pgmap v9294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:29 smithi114 ceph-mon[30820]: pgmap v9295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:29 smithi116 ceph-mon[35411]: pgmap v9295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:31 smithi114 ceph-mon[30820]: pgmap v9296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:31 smithi116 ceph-mon[35411]: pgmap v9296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:33 smithi114 ceph-mon[30820]: pgmap v9297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:33 smithi116 ceph-mon[35411]: pgmap v9297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:35 smithi114 ceph-mon[30820]: pgmap v9298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:35 smithi116 ceph-mon[35411]: pgmap v9298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:37 smithi114 ceph-mon[30820]: pgmap v9299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:37 smithi116 ceph-mon[35411]: pgmap v9299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:39 smithi114 ceph-mon[30820]: pgmap v9300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:39 smithi116 ceph-mon[35411]: pgmap v9300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:41 smithi114 ceph-mon[30820]: pgmap v9301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:41 smithi116 ceph-mon[35411]: pgmap v9301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:43 smithi114 ceph-mon[30820]: pgmap v9302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:43 smithi116 ceph-mon[35411]: pgmap v9302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:45 smithi114 ceph-mon[30820]: pgmap v9303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:45 smithi116 ceph-mon[35411]: pgmap v9303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:36:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:47 smithi114 ceph-mon[30820]: pgmap v9304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:47 smithi116 ceph-mon[35411]: pgmap v9304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:49 smithi114 ceph-mon[30820]: pgmap v9305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:49 smithi116 ceph-mon[35411]: pgmap v9305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:36:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:51 smithi114 ceph-mon[30820]: pgmap v9306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:51 smithi116 ceph-mon[35411]: pgmap v9306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:36:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:36:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:36:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:53.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:53 smithi116 ceph-mon[35411]: pgmap v9307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:53 smithi114 ceph-mon[30820]: pgmap v9307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:36:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:36:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:36:54.740+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:36:55.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:36:55.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:36:55.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:36:55.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:36:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:36:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:36:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:36:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:36:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:55 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:36:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:36:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:36:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:55 smithi114 ceph-mon[30820]: pgmap v9308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T11:36:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:55 smithi114 ceph-mon[30820]: pgmap v9309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:36:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:55 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:36:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:36:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:36:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:36:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:36:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:36:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:55 smithi116 ceph-mon[35411]: pgmap v9308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T11:36:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:55 smithi116 ceph-mon[35411]: pgmap v9309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:36:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:36:57 smithi114 ceph-mon[30820]: pgmap v9310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:36:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:36:57 smithi116 ceph-mon[35411]: pgmap v9310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:37:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:00 smithi114 ceph-mon[30820]: pgmap v9311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:37:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:00 smithi116 ceph-mon[35411]: pgmap v9311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:37:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:02 smithi114 ceph-mon[30820]: pgmap v9312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:37:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:02 smithi116 ceph-mon[35411]: pgmap v9312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:37:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:04 smithi114 ceph-mon[30820]: pgmap v9313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:37:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:04 smithi116 ceph-mon[35411]: pgmap v9313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:37:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:06 smithi114 ceph-mon[30820]: pgmap v9314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:37:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:06 smithi116 ceph-mon[35411]: pgmap v9314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:37:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:08 smithi114 ceph-mon[30820]: pgmap v9315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:08 smithi116 ceph-mon[35411]: pgmap v9315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:10 smithi114 ceph-mon[30820]: pgmap v9316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:10 smithi116 ceph-mon[35411]: pgmap v9316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:12 smithi114 ceph-mon[30820]: pgmap v9317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:12 smithi116 ceph-mon[35411]: pgmap v9317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:14 smithi114 ceph-mon[30820]: pgmap v9318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:14 smithi116 ceph-mon[35411]: pgmap v9318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:16 smithi114 ceph-mon[30820]: pgmap v9319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:16 smithi116 ceph-mon[35411]: pgmap v9319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:18 smithi114 ceph-mon[30820]: pgmap v9320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:18 smithi116 ceph-mon[35411]: pgmap v9320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:20 smithi116 ceph-mon[35411]: pgmap v9321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:20 smithi114 ceph-mon[30820]: pgmap v9321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:22 smithi116 ceph-mon[35411]: pgmap v9322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:22 smithi114 ceph-mon[30820]: pgmap v9322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:24 smithi116 ceph-mon[35411]: pgmap v9323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:24 smithi114 ceph-mon[30820]: pgmap v9323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:26 smithi116 ceph-mon[35411]: pgmap v9324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:26 smithi114 ceph-mon[30820]: pgmap v9324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:28 smithi114 ceph-mon[30820]: pgmap v9325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:28 smithi116 ceph-mon[35411]: pgmap v9325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:30 smithi114 ceph-mon[30820]: pgmap v9326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:30 smithi116 ceph-mon[35411]: pgmap v9326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:32 smithi114 ceph-mon[30820]: pgmap v9327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:32 smithi116 ceph-mon[35411]: pgmap v9327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:34 smithi114 ceph-mon[30820]: pgmap v9328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:34 smithi116 ceph-mon[35411]: pgmap v9328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:36 smithi114 ceph-mon[30820]: pgmap v9329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:36 smithi116 ceph-mon[35411]: pgmap v9329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:38 smithi114 ceph-mon[30820]: pgmap v9330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:38 smithi116 ceph-mon[35411]: pgmap v9330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:40 smithi114 ceph-mon[30820]: pgmap v9331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:40 smithi116 ceph-mon[35411]: pgmap v9331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:42 smithi114 ceph-mon[30820]: pgmap v9332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:42 smithi116 ceph-mon[35411]: pgmap v9332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:44.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:44 smithi114 ceph-mon[30820]: pgmap v9333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:44 smithi116 ceph-mon[35411]: pgmap v9333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:46 smithi114 ceph-mon[30820]: pgmap v9334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:46 smithi116 ceph-mon[35411]: pgmap v9334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:48 smithi114 ceph-mon[30820]: pgmap v9335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:48 smithi116 ceph-mon[35411]: pgmap v9335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:50 smithi114 ceph-mon[30820]: pgmap v9336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:50 smithi116 ceph-mon[35411]: pgmap v9336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:52 smithi114 ceph-mon[30820]: pgmap v9337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:52 smithi116 ceph-mon[35411]: pgmap v9337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:37:54.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:54 smithi116 ceph-mon[35411]: pgmap v9338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:54 smithi114 ceph-mon[30820]: pgmap v9338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:37:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:37:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:37:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:56 smithi114 ceph-mon[30820]: pgmap v9339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:56 smithi116 ceph-mon[35411]: pgmap v9339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:37:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-mon[30820]: pgmap v9340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:37:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:37:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:37:58.100+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:37:58.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:37:58.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:37:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:37:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:58 smithi116 ceph-mon[35411]: pgmap v9340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:37:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:37:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:37:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:37:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:59 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:37:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:37:59.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:59 smithi114 ceph-mon[30820]: pgmap v9341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:37:59.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:37:59 smithi114 ceph-mon[30820]: pgmap v9342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:37:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:59 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:37:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:37:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:37:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:37:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:37:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:37:59.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:59 smithi116 ceph-mon[35411]: pgmap v9341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:37:59.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:37:59 smithi116 ceph-mon[35411]: pgmap v9342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:38:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:01 smithi114 ceph-mon[30820]: pgmap v9343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:38:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:01 smithi116 ceph-mon[35411]: pgmap v9343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:38:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:03 smithi114 ceph-mon[30820]: pgmap v9344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:38:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:03 smithi116 ceph-mon[35411]: pgmap v9344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:38:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:05 smithi114 ceph-mon[30820]: pgmap v9345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:38:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:05 smithi116 ceph-mon[35411]: pgmap v9345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:38:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:07 smithi114 ceph-mon[30820]: pgmap v9346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:38:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:07 smithi116 ceph-mon[35411]: pgmap v9346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:38:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:09 smithi114 ceph-mon[30820]: pgmap v9347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:38:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:09 smithi116 ceph-mon[35411]: pgmap v9347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:38:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:11 smithi114 ceph-mon[30820]: pgmap v9348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:11 smithi116 ceph-mon[35411]: pgmap v9348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:13 smithi114 ceph-mon[30820]: pgmap v9349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:13 smithi116 ceph-mon[35411]: pgmap v9349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:15 smithi114 ceph-mon[30820]: pgmap v9350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:15.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:15 smithi116 ceph-mon[35411]: pgmap v9350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:17 smithi114 ceph-mon[30820]: pgmap v9351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:17 smithi116 ceph-mon[35411]: pgmap v9351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:19 smithi114 ceph-mon[30820]: pgmap v9352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:19 smithi116 ceph-mon[35411]: pgmap v9352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:21 smithi114 ceph-mon[30820]: pgmap v9353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:21 smithi116 ceph-mon[35411]: pgmap v9353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:23 smithi114 ceph-mon[30820]: pgmap v9354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:23 smithi116 ceph-mon[35411]: pgmap v9354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:25 smithi114 ceph-mon[30820]: pgmap v9355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:25 smithi116 ceph-mon[35411]: pgmap v9355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:27 smithi114 ceph-mon[30820]: pgmap v9356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:27 smithi116 ceph-mon[35411]: pgmap v9356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:29 smithi114 ceph-mon[30820]: pgmap v9357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:29 smithi116 ceph-mon[35411]: pgmap v9357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:31 smithi114 ceph-mon[30820]: pgmap v9358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:31 smithi116 ceph-mon[35411]: pgmap v9358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:33 smithi114 ceph-mon[30820]: pgmap v9359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:33 smithi116 ceph-mon[35411]: pgmap v9359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:35 smithi114 ceph-mon[30820]: pgmap v9360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:35 smithi116 ceph-mon[35411]: pgmap v9360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:37 smithi114 ceph-mon[30820]: pgmap v9361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:37 smithi116 ceph-mon[35411]: pgmap v9361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:39 smithi114 ceph-mon[30820]: pgmap v9362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:39 smithi116 ceph-mon[35411]: pgmap v9362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:41 smithi114 ceph-mon[30820]: pgmap v9363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:41 smithi116 ceph-mon[35411]: pgmap v9363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:43 smithi114 ceph-mon[30820]: pgmap v9364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:43 smithi116 ceph-mon[35411]: pgmap v9364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:45 smithi114 ceph-mon[30820]: pgmap v9365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:45 smithi116 ceph-mon[35411]: pgmap v9365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:47 smithi114 ceph-mon[30820]: pgmap v9366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:47 smithi116 ceph-mon[35411]: pgmap v9366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:49 smithi114 ceph-mon[30820]: pgmap v9367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:49 smithi116 ceph-mon[35411]: pgmap v9367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:51 smithi114 ceph-mon[30820]: pgmap v9368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:51 smithi116 ceph-mon[35411]: pgmap v9368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:38:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:53 smithi114 ceph-mon[30820]: pgmap v9369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:53 smithi116 ceph-mon[35411]: pgmap v9369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:38:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:55 smithi114 ceph-mon[30820]: pgmap v9370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:55 smithi116 ceph-mon[35411]: pgmap v9370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:57 smithi114 ceph-mon[30820]: pgmap v9371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:57 smithi116 ceph-mon[35411]: pgmap v9371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:38:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:38:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:38:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:38:59 smithi114 ceph-mon[30820]: pgmap v9372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:38:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:38:59 smithi116 ceph-mon[35411]: pgmap v9372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-mon[30820]: pgmap v9373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:39:01.461+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:39:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:39:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:39:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:39:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:01 smithi116 ceph-mon[35411]: pgmap v9373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:02 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:39:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:39:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:39:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:39:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:39:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:39:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:39:02.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:39:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:39:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:39:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:39:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:39:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:39:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:02 smithi114 ceph-mon[30820]: pgmap v9374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:39:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:02 smithi114 ceph-mon[30820]: pgmap v9375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:02.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:39:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:02 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:39:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:39:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:39:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:39:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:39:02.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:02 smithi116 ceph-mon[35411]: pgmap v9374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:02 smithi116 ceph-mon[35411]: pgmap v9375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:02.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:39:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:03 smithi114 ceph-mon[30820]: pgmap v9376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:03 smithi116 ceph-mon[35411]: pgmap v9376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:05 smithi114 ceph-mon[30820]: pgmap v9377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:05 smithi116 ceph-mon[35411]: pgmap v9377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:07 smithi114 ceph-mon[30820]: pgmap v9378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:07 smithi116 ceph-mon[35411]: pgmap v9378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:09 smithi114 ceph-mon[30820]: pgmap v9379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:09 smithi116 ceph-mon[35411]: pgmap v9379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:39:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:11 smithi114 ceph-mon[30820]: pgmap v9380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:39:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:11 smithi116 ceph-mon[35411]: pgmap v9380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:39:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:13 smithi114 ceph-mon[30820]: pgmap v9381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:13 smithi116 ceph-mon[35411]: pgmap v9381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:15 smithi114 ceph-mon[30820]: pgmap v9382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:15 smithi116 ceph-mon[35411]: pgmap v9382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:17 smithi114 ceph-mon[30820]: pgmap v9383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:17 smithi116 ceph-mon[35411]: pgmap v9383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:19 smithi114 ceph-mon[30820]: pgmap v9384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:19 smithi116 ceph-mon[35411]: pgmap v9384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:21 smithi114 ceph-mon[30820]: pgmap v9385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:39:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:21 smithi116 ceph-mon[35411]: pgmap v9385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:39:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:23 smithi114 ceph-mon[30820]: pgmap v9386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:23 smithi116 ceph-mon[35411]: pgmap v9386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:25 smithi114 ceph-mon[30820]: pgmap v9387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:25 smithi116 ceph-mon[35411]: pgmap v9387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:27 smithi114 ceph-mon[30820]: pgmap v9388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:27 smithi116 ceph-mon[35411]: pgmap v9388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:29 smithi114 ceph-mon[30820]: pgmap v9389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:29 smithi116 ceph-mon[35411]: pgmap v9389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:31 smithi114 ceph-mon[30820]: pgmap v9390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:39:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:31 smithi116 ceph-mon[35411]: pgmap v9390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:39:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:33 smithi114 ceph-mon[30820]: pgmap v9391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:33 smithi116 ceph-mon[35411]: pgmap v9391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:35 smithi114 ceph-mon[30820]: pgmap v9392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:35 smithi116 ceph-mon[35411]: pgmap v9392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:37 smithi114 ceph-mon[30820]: pgmap v9393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:37 smithi116 ceph-mon[35411]: pgmap v9393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:39 smithi114 ceph-mon[30820]: pgmap v9394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:39 smithi116 ceph-mon[35411]: pgmap v9394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:41 smithi114 ceph-mon[30820]: pgmap v9395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:39:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:41 smithi116 ceph-mon[35411]: pgmap v9395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:39:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:43 smithi114 ceph-mon[30820]: pgmap v9396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:43 smithi116 ceph-mon[35411]: pgmap v9396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:45 smithi114 ceph-mon[30820]: pgmap v9397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:45 smithi116 ceph-mon[35411]: pgmap v9397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:47 smithi114 ceph-mon[30820]: pgmap v9398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:47.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:47 smithi116 ceph-mon[35411]: pgmap v9398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:49 smithi114 ceph-mon[30820]: pgmap v9399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:49 smithi116 ceph-mon[35411]: pgmap v9399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:51 smithi114 ceph-mon[30820]: pgmap v9400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:39:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:51 smithi116 ceph-mon[35411]: pgmap v9400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:39:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:53 smithi114 ceph-mon[30820]: pgmap v9401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:53 smithi116 ceph-mon[35411]: pgmap v9401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:55 smithi114 ceph-mon[30820]: pgmap v9402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:55 smithi116 ceph-mon[35411]: pgmap v9402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:39:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:57 smithi114 ceph-mon[30820]: pgmap v9403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:57 smithi116 ceph-mon[35411]: pgmap v9403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:39:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:39:59 smithi114 ceph-mon[30820]: pgmap v9404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:39:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:39:59 smithi116 ceph-mon[35411]: pgmap v9404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:40:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:40:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:01 smithi114 ceph-mon[30820]: pgmap v9405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:01 smithi116 ceph-mon[35411]: pgmap v9405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:40:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:40:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:03 smithi114 ceph-mon[30820]: pgmap v9406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:03 smithi116 ceph-mon[35411]: pgmap v9406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:40:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:40:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:40:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:40:04.820+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:40:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:40:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:40:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:40:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:40:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:40:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:40:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:40:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:40:05.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:40:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:40:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:40:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:40:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:40:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:05 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:40:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:05 smithi114 ceph-mon[30820]: pgmap v9407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:40:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:05 smithi114 ceph-mon[30820]: pgmap v9408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:40:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:05 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:40:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:40:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:40:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:40:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:40:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:40:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:40:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:40:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:40:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:40:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:40:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:40:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:40:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:05 smithi116 ceph-mon[35411]: pgmap v9407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:40:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:05 smithi116 ceph-mon[35411]: pgmap v9408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:40:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:07 smithi114 ceph-mon[30820]: pgmap v9409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:40:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:07 smithi116 ceph-mon[35411]: pgmap v9409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:40:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:09 smithi114 ceph-mon[30820]: pgmap v9410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:40:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:09 smithi116 ceph-mon[35411]: pgmap v9410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:40:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:11 smithi114 ceph-mon[30820]: pgmap v9411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:40:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:11 smithi116 ceph-mon[35411]: pgmap v9411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:40:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:13 smithi114 ceph-mon[30820]: pgmap v9412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:40:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:13 smithi116 ceph-mon[35411]: pgmap v9412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:40:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:15 smithi114 ceph-mon[30820]: pgmap v9413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:40:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:15 smithi116 ceph-mon[35411]: pgmap v9413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:40:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:17 smithi114 ceph-mon[30820]: pgmap v9414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:18.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:17 smithi116 ceph-mon[35411]: pgmap v9414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:19 smithi114 ceph-mon[30820]: pgmap v9415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:19 smithi116 ceph-mon[35411]: pgmap v9415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:21 smithi114 ceph-mon[30820]: pgmap v9416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:40:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:21 smithi116 ceph-mon[35411]: pgmap v9416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:40:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:23 smithi116 ceph-mon[35411]: pgmap v9417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:23 smithi114 ceph-mon[30820]: pgmap v9417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:25 smithi116 ceph-mon[35411]: pgmap v9418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:25 smithi114 ceph-mon[30820]: pgmap v9418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:27 smithi116 ceph-mon[35411]: pgmap v9419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:27 smithi114 ceph-mon[30820]: pgmap v9419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:29 smithi114 ceph-mon[30820]: pgmap v9420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:29 smithi116 ceph-mon[35411]: pgmap v9420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:31 smithi114 ceph-mon[30820]: pgmap v9421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:40:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:31 smithi116 ceph-mon[35411]: pgmap v9421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:40:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:33 smithi114 ceph-mon[30820]: pgmap v9422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:33 smithi116 ceph-mon[35411]: pgmap v9422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:35 smithi114 ceph-mon[30820]: pgmap v9423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:35 smithi116 ceph-mon[35411]: pgmap v9423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:37 smithi114 ceph-mon[30820]: pgmap v9424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:37 smithi116 ceph-mon[35411]: pgmap v9424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:39 smithi114 ceph-mon[30820]: pgmap v9425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:39 smithi116 ceph-mon[35411]: pgmap v9425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:41 smithi114 ceph-mon[30820]: pgmap v9426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:40:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:41 smithi116 ceph-mon[35411]: pgmap v9426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:40:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:43 smithi114 ceph-mon[30820]: pgmap v9427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:43 smithi116 ceph-mon[35411]: pgmap v9427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:46.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:45 smithi114 ceph-mon[30820]: pgmap v9428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:45 smithi116 ceph-mon[35411]: pgmap v9428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:47 smithi114 ceph-mon[30820]: pgmap v9429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:47 smithi116 ceph-mon[35411]: pgmap v9429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:49 smithi114 ceph-mon[30820]: pgmap v9430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:49 smithi116 ceph-mon[35411]: pgmap v9430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:51 smithi114 ceph-mon[30820]: pgmap v9431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:40:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:51 smithi116 ceph-mon[35411]: pgmap v9431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:40:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:53 smithi114 ceph-mon[30820]: pgmap v9432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:53 smithi116 ceph-mon[35411]: pgmap v9432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:40:56.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:55 smithi116 ceph-mon[35411]: pgmap v9433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:55 smithi114 ceph-mon[30820]: pgmap v9433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:40:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:57 smithi114 ceph-mon[30820]: pgmap v9434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:40:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:57 smithi116 ceph-mon[35411]: pgmap v9434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:40:59 smithi114 ceph-mon[30820]: pgmap v9435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:40:59 smithi116 ceph-mon[35411]: pgmap v9435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:01 smithi114 ceph-mon[30820]: pgmap v9436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:01 smithi116 ceph-mon[35411]: pgmap v9436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:04 smithi114 ceph-mon[30820]: pgmap v9437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:04 smithi116 ceph-mon[35411]: pgmap v9437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:41:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:41:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:06 smithi114 ceph-mon[30820]: pgmap v9438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:06 smithi116 ceph-mon[35411]: pgmap v9438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-mon[30820]: pgmap v9439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:41:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:41:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:41:08.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:41:08.180+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:41:08.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:41:08.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:41:08.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:41:08.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:41:08.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:41:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:41:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:08 smithi116 ceph-mon[35411]: pgmap v9439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:41:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:41:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:41:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:09 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:41:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:41:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:41:09.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:41:09.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:41:09.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:41:09.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:41:09.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:41:09.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:41:09.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:41:09.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:09 smithi114 ceph-mon[30820]: pgmap v9440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:41:09.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:09 smithi114 ceph-mon[30820]: pgmap v9441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:41:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:09 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:41:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:41:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:41:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:41:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:41:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:41:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:41:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:41:09.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:41:09.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:41:09.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:09 smithi116 ceph-mon[35411]: pgmap v9440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:41:09.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:09 smithi116 ceph-mon[35411]: pgmap v9441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:41:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:11 smithi114 ceph-mon[30820]: pgmap v9442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:41:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:11 smithi116 ceph-mon[35411]: pgmap v9442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:41:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:13 smithi114 ceph-mon[30820]: pgmap v9443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:41:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:13 smithi116 ceph-mon[35411]: pgmap v9443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:41:15.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:15 smithi114 ceph-mon[30820]: pgmap v9444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:41:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:15 smithi116 ceph-mon[35411]: pgmap v9444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:41:17.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:17 smithi114 ceph-mon[30820]: pgmap v9445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:41:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:17 smithi116 ceph-mon[35411]: pgmap v9445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:41:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:19 smithi114 ceph-mon[30820]: pgmap v9446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:41:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:19 smithi116 ceph-mon[35411]: pgmap v9446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:41:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:21 smithi114 ceph-mon[30820]: pgmap v9447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:21 smithi116 ceph-mon[35411]: pgmap v9447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:23 smithi114 ceph-mon[30820]: pgmap v9448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:23 smithi116 ceph-mon[35411]: pgmap v9448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:25 smithi114 ceph-mon[30820]: pgmap v9449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:25 smithi116 ceph-mon[35411]: pgmap v9449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:27 smithi114 ceph-mon[30820]: pgmap v9450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:27 smithi116 ceph-mon[35411]: pgmap v9450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:29 smithi114 ceph-mon[30820]: pgmap v9451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:29 smithi116 ceph-mon[35411]: pgmap v9451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:31 smithi114 ceph-mon[30820]: pgmap v9452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:31 smithi116 ceph-mon[35411]: pgmap v9452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:33 smithi114 ceph-mon[30820]: pgmap v9453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:33 smithi116 ceph-mon[35411]: pgmap v9453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:35 smithi114 ceph-mon[30820]: pgmap v9454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:35 smithi116 ceph-mon[35411]: pgmap v9454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:37 smithi114 ceph-mon[30820]: pgmap v9455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:37 smithi116 ceph-mon[35411]: pgmap v9455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:39 smithi114 ceph-mon[30820]: pgmap v9456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:39 smithi116 ceph-mon[35411]: pgmap v9456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:41 smithi114 ceph-mon[30820]: pgmap v9457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:41 smithi116 ceph-mon[35411]: pgmap v9457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:43 smithi114 ceph-mon[30820]: pgmap v9458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:43 smithi116 ceph-mon[35411]: pgmap v9458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:45 smithi114 ceph-mon[30820]: pgmap v9459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:45 smithi116 ceph-mon[35411]: pgmap v9459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:47 smithi114 ceph-mon[30820]: pgmap v9460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:47 smithi116 ceph-mon[35411]: pgmap v9460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:49 smithi114 ceph-mon[30820]: pgmap v9461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:49 smithi116 ceph-mon[35411]: pgmap v9461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:51 smithi114 ceph-mon[30820]: pgmap v9462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:51 smithi116 ceph-mon[35411]: pgmap v9462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:41:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:53 smithi114 ceph-mon[30820]: pgmap v9463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:53 smithi116 ceph-mon[35411]: pgmap v9463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:41:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:55 smithi114 ceph-mon[30820]: pgmap v9464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:55 smithi116 ceph-mon[35411]: pgmap v9464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:57 smithi114 ceph-mon[30820]: pgmap v9465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:57 smithi116 ceph-mon[35411]: pgmap v9465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:41:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:41:59 smithi114 ceph-mon[30820]: pgmap v9466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:41:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:41:59 smithi116 ceph-mon[35411]: pgmap v9466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:01 smithi114 ceph-mon[30820]: pgmap v9467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:01 smithi116 ceph-mon[35411]: pgmap v9467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:03 smithi114 ceph-mon[30820]: pgmap v9468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:42:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:03 smithi116 ceph-mon[35411]: pgmap v9468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:42:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:05 smithi114 ceph-mon[30820]: pgmap v9469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:05 smithi116 ceph-mon[35411]: pgmap v9469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:07 smithi114 ceph-mon[30820]: pgmap v9470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:07 smithi116 ceph-mon[35411]: pgmap v9470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:09 smithi114 ceph-mon[30820]: pgmap v9471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:42:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:09 smithi116 ceph-mon[35411]: pgmap v9471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:42:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-mon[30820]: pgmap v9472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:11.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:42:11.540+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:42:11.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:42:11.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:42:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:42:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:11 smithi116 ceph-mon[35411]: pgmap v9472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:12 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:42:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:42:12.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:42:12.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:12 smithi114 ceph-mon[30820]: pgmap v9473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:12 smithi114 ceph-mon[30820]: pgmap v9474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:12.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:42:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:12 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:42:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:42:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:42:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:42:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:42:12.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:12 smithi116 ceph-mon[35411]: pgmap v9473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:12 smithi116 ceph-mon[35411]: pgmap v9474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:12.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:42:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:13 smithi114 ceph-mon[30820]: pgmap v9475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:13 smithi116 ceph-mon[35411]: pgmap v9475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:15 smithi114 ceph-mon[30820]: pgmap v9476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:15.933 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:15 smithi116 ceph-mon[35411]: pgmap v9476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:17 smithi114 ceph-mon[30820]: pgmap v9477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:17 smithi116 ceph-mon[35411]: pgmap v9477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:19.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:19 smithi114 ceph-mon[30820]: pgmap v9478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:19 smithi116 ceph-mon[35411]: pgmap v9478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:42:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:21 smithi114 ceph-mon[30820]: pgmap v9479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:42:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:21 smithi116 ceph-mon[35411]: pgmap v9479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:42:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:23 smithi114 ceph-mon[30820]: pgmap v9480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:23 smithi116 ceph-mon[35411]: pgmap v9480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:25 smithi114 ceph-mon[30820]: pgmap v9481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:25 smithi116 ceph-mon[35411]: pgmap v9481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:27 smithi114 ceph-mon[30820]: pgmap v9482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:27 smithi116 ceph-mon[35411]: pgmap v9482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:29 smithi114 ceph-mon[30820]: pgmap v9483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:29 smithi116 ceph-mon[35411]: pgmap v9483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:31 smithi114 ceph-mon[30820]: pgmap v9484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:42:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:31 smithi116 ceph-mon[35411]: pgmap v9484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:33 smithi114 ceph-mon[30820]: pgmap v9485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:33 smithi116 ceph-mon[35411]: pgmap v9485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:35 smithi114 ceph-mon[30820]: pgmap v9486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:35 smithi116 ceph-mon[35411]: pgmap v9486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:37 smithi114 ceph-mon[30820]: pgmap v9487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:37 smithi116 ceph-mon[35411]: pgmap v9487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:39 smithi114 ceph-mon[30820]: pgmap v9488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:39 smithi116 ceph-mon[35411]: pgmap v9488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:42 smithi114 ceph-mon[30820]: pgmap v9489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:42:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:42 smithi116 ceph-mon[35411]: pgmap v9489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:42:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:44 smithi114 ceph-mon[30820]: pgmap v9490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:44 smithi116 ceph-mon[35411]: pgmap v9490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:46 smithi114 ceph-mon[30820]: pgmap v9491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:46 smithi116 ceph-mon[35411]: pgmap v9491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:48 smithi114 ceph-mon[30820]: pgmap v9492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:48 smithi116 ceph-mon[35411]: pgmap v9492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:50 smithi114 ceph-mon[30820]: pgmap v9493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:50 smithi116 ceph-mon[35411]: pgmap v9493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:42:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:52 smithi114 ceph-mon[30820]: pgmap v9494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:42:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:52 smithi116 ceph-mon[35411]: pgmap v9494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:42:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:54 smithi114 ceph-mon[30820]: pgmap v9495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:54 smithi116 ceph-mon[35411]: pgmap v9495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:56 smithi114 ceph-mon[30820]: pgmap v9496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:56 smithi116 ceph-mon[35411]: pgmap v9496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:42:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:42:58 smithi114 ceph-mon[30820]: pgmap v9497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:42:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:42:58 smithi116 ceph-mon[35411]: pgmap v9497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:00 smithi114 ceph-mon[30820]: pgmap v9498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:00 smithi116 ceph-mon[35411]: pgmap v9498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:02 smithi114 ceph-mon[30820]: pgmap v9499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:02 smithi116 ceph-mon[35411]: pgmap v9499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:04 smithi114 ceph-mon[30820]: pgmap v9500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:04 smithi116 ceph-mon[35411]: pgmap v9500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:06 smithi114 ceph-mon[30820]: pgmap v9501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:43:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:06 smithi116 ceph-mon[35411]: pgmap v9501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:43:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:08 smithi114 ceph-mon[30820]: pgmap v9502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:08 smithi116 ceph-mon[35411]: pgmap v9502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:10 smithi114 ceph-mon[30820]: pgmap v9503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:10 smithi116 ceph-mon[35411]: pgmap v9503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: pgmap v9504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:43:12.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:12 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:43:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: pgmap v9504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:43:13.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:12 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:43:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:14 smithi116 ceph-mon[35411]: pgmap v9505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-mon[30820]: pgmap v9505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:15.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:43:14.901+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:43:15.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:43:15.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:43:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:43:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:15 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:43:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:43:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:43:15.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:15 smithi116 ceph-mon[35411]: pgmap v9506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:15 smithi116 ceph-mon[35411]: pgmap v9507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:43:15.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:43:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:15 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:43:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:43:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:43:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:43:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:15 smithi114 ceph-mon[30820]: pgmap v9506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:43:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:15 smithi114 ceph-mon[30820]: pgmap v9507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:43:16.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:43:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:17 smithi114 ceph-mon[30820]: pgmap v9508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:43:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:17 smithi116 ceph-mon[35411]: pgmap v9508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:43:20.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:19 smithi114 ceph-mon[30820]: pgmap v9509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:43:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:19 smithi116 ceph-mon[35411]: pgmap v9509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:43:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:21 smithi114 ceph-mon[30820]: pgmap v9510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:43:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:21 smithi116 ceph-mon[35411]: pgmap v9510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:43:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:23 smithi114 ceph-mon[30820]: pgmap v9511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:43:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:23 smithi116 ceph-mon[35411]: pgmap v9511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:43:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:25 smithi114 ceph-mon[30820]: pgmap v9512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:43:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:25 smithi116 ceph-mon[35411]: pgmap v9512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:43:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:27 smithi114 ceph-mon[30820]: pgmap v9513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:27 smithi116 ceph-mon[35411]: pgmap v9513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:29 smithi114 ceph-mon[30820]: pgmap v9514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:29 smithi116 ceph-mon[35411]: pgmap v9514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:31 smithi114 ceph-mon[30820]: pgmap v9515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:43:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:31 smithi116 ceph-mon[35411]: pgmap v9515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:43:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:33 smithi114 ceph-mon[30820]: pgmap v9516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:33 smithi116 ceph-mon[35411]: pgmap v9516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:35 smithi114 ceph-mon[30820]: pgmap v9517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:35 smithi116 ceph-mon[35411]: pgmap v9517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:38 smithi114 ceph-mon[30820]: pgmap v9518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:38 smithi116 ceph-mon[35411]: pgmap v9518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:40 smithi114 ceph-mon[30820]: pgmap v9519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:40 smithi116 ceph-mon[35411]: pgmap v9519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:42 smithi114 ceph-mon[30820]: pgmap v9520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:43:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:42 smithi116 ceph-mon[35411]: pgmap v9520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:43:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:44 smithi114 ceph-mon[30820]: pgmap v9521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:44 smithi116 ceph-mon[35411]: pgmap v9521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:46 smithi114 ceph-mon[30820]: pgmap v9522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:46 smithi116 ceph-mon[35411]: pgmap v9522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:48 smithi114 ceph-mon[30820]: pgmap v9523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:48 smithi116 ceph-mon[35411]: pgmap v9523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:50 smithi114 ceph-mon[30820]: pgmap v9524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:50 smithi116 ceph-mon[35411]: pgmap v9524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:52 smithi114 ceph-mon[30820]: pgmap v9525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:43:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:52 smithi116 ceph-mon[35411]: pgmap v9525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:43:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:54 smithi114 ceph-mon[30820]: pgmap v9526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:54 smithi116 ceph-mon[35411]: pgmap v9526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:43:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:56 smithi114 ceph-mon[30820]: pgmap v9527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:56 smithi116 ceph-mon[35411]: pgmap v9527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:43:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:43:58 smithi114 ceph-mon[30820]: pgmap v9528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:43:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:43:58 smithi116 ceph-mon[35411]: pgmap v9528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:00 smithi114 ceph-mon[30820]: pgmap v9529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:00 smithi116 ceph-mon[35411]: pgmap v9529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:02 smithi114 ceph-mon[30820]: pgmap v9530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:44:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:02 smithi116 ceph-mon[35411]: pgmap v9530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:44:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:04 smithi114 ceph-mon[30820]: pgmap v9531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:44:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:04 smithi116 ceph-mon[35411]: pgmap v9531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:44:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:06 smithi114 ceph-mon[30820]: pgmap v9532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:44:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:06 smithi116 ceph-mon[35411]: pgmap v9532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:44:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:08 smithi114 ceph-mon[30820]: pgmap v9533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:08 smithi116 ceph-mon[35411]: pgmap v9533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:10 smithi114 ceph-mon[30820]: pgmap v9534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:44:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:10 smithi116 ceph-mon[35411]: pgmap v9534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:44:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:12 smithi114 ceph-mon[30820]: pgmap v9535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:44:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:12 smithi116 ceph-mon[35411]: pgmap v9535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:44:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:14 smithi114 ceph-mon[30820]: pgmap v9536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:14 smithi116 ceph-mon[35411]: pgmap v9536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:44:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:44:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:16 smithi114 ceph-mon[30820]: pgmap v9537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:44:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:16 smithi116 ceph-mon[35411]: pgmap v9537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:44:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-mon[30820]: pgmap v9538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:18.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:44:18.260+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:44:18.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:44:18.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:44:18.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:44:18.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:44:18.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:44:18.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:44:18.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:44:18.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:44:18.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:44:18.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:44:18.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:44:18.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:44:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:44:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:18 smithi116 ceph-mon[35411]: pgmap v9538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:44:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:19 smithi114 ceph-mon[30820]: pgmap v9539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:19 smithi114 ceph-mon[30820]: pgmap v9540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:44:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:44:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:44:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:44:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:44:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:44:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:44:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:44:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:44:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:44:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:44:19.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:19 smithi116 ceph-mon[35411]: pgmap v9539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:44:19.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:19 smithi116 ceph-mon[35411]: pgmap v9540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:44:19.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:44:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:21 smithi114 ceph-mon[30820]: pgmap v9541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:44:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:21 smithi116 ceph-mon[35411]: pgmap v9541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:44:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:23 smithi114 ceph-mon[30820]: pgmap v9542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:44:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:23 smithi116 ceph-mon[35411]: pgmap v9542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:44:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:25 smithi114 ceph-mon[30820]: pgmap v9543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:44:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:25 smithi116 ceph-mon[35411]: pgmap v9543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:44:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:27 smithi114 ceph-mon[30820]: pgmap v9544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:44:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:27 smithi116 ceph-mon[35411]: pgmap v9544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:44:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:29 smithi114 ceph-mon[30820]: pgmap v9545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:44:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:29 smithi116 ceph-mon[35411]: pgmap v9545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:44:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:31 smithi114 ceph-mon[30820]: pgmap v9546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T11:44:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:31 smithi116 ceph-mon[35411]: pgmap v9546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T11:44:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:33 smithi114 ceph-mon[30820]: pgmap v9547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:44:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:33 smithi116 ceph-mon[35411]: pgmap v9547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:44:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:35 smithi114 ceph-mon[30820]: pgmap v9548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:44:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:35 smithi116 ceph-mon[35411]: pgmap v9548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:44:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:37 smithi114 ceph-mon[30820]: pgmap v9549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:37 smithi116 ceph-mon[35411]: pgmap v9549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:39 smithi114 ceph-mon[30820]: pgmap v9550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:39 smithi116 ceph-mon[35411]: pgmap v9550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:41 smithi114 ceph-mon[30820]: pgmap v9551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:44:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:41 smithi116 ceph-mon[35411]: pgmap v9551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:44:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:43 smithi114 ceph-mon[30820]: pgmap v9552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:44:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:43 smithi116 ceph-mon[35411]: pgmap v9552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:44:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:45 smithi114 ceph-mon[30820]: pgmap v9553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:44:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:45 smithi116 ceph-mon[35411]: pgmap v9553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:44:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:47 smithi114 ceph-mon[30820]: pgmap v9554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:47 smithi116 ceph-mon[35411]: pgmap v9554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:49 smithi114 ceph-mon[30820]: pgmap v9555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:49 smithi116 ceph-mon[35411]: pgmap v9555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:51 smithi114 ceph-mon[30820]: pgmap v9556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:44:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:51 smithi116 ceph-mon[35411]: pgmap v9556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:44:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:53 smithi114 ceph-mon[30820]: pgmap v9557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:44:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:53 smithi116 ceph-mon[35411]: pgmap v9557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:44:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:55 smithi114 ceph-mon[30820]: pgmap v9558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:44:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:55 smithi116 ceph-mon[35411]: pgmap v9558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:44:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:57 smithi114 ceph-mon[30820]: pgmap v9559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:57 smithi116 ceph-mon[35411]: pgmap v9559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:44:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:44:59 smithi114 ceph-mon[30820]: pgmap v9560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:44:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:44:59 smithi116 ceph-mon[35411]: pgmap v9560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:01 smithi114 ceph-mon[30820]: pgmap v9561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:45:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:01 smithi116 ceph-mon[35411]: pgmap v9561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:45:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:03 smithi114 ceph-mon[30820]: pgmap v9562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:45:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:03 smithi116 ceph-mon[35411]: pgmap v9562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:45:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:05 smithi114 ceph-mon[30820]: pgmap v9563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:45:05.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:05 smithi116 ceph-mon[35411]: pgmap v9563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:45:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:07 smithi114 ceph-mon[30820]: pgmap v9564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:45:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:07 smithi116 ceph-mon[35411]: pgmap v9564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:45:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:09 smithi114 ceph-mon[30820]: pgmap v9565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:09 smithi116 ceph-mon[35411]: pgmap v9565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:11 smithi114 ceph-mon[30820]: pgmap v9566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:45:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:11 smithi116 ceph-mon[35411]: pgmap v9566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:45:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:13 smithi116 ceph-mon[35411]: pgmap v9567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:45:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:13 smithi114 ceph-mon[30820]: pgmap v9567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:45:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:15 smithi116 ceph-mon[35411]: pgmap v9568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:45:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:15 smithi114 ceph-mon[30820]: pgmap v9568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:45:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:17 smithi116 ceph-mon[35411]: pgmap v9569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:45:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:17 smithi114 ceph-mon[30820]: pgmap v9569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:45:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:45:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:45:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:19 smithi114 ceph-mon[30820]: pgmap v9570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:19 smithi116 ceph-mon[35411]: pgmap v9570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:21.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-mon[30820]: pgmap v9571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:45:21.620+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:45:21.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:45:21.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:45:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:45:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:21 smithi116 ceph-mon[35411]: pgmap v9571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:45:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:45:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:45:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:45:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:45:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:45:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:45:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:23 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:45:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:45:23.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:23 smithi114 ceph-mon[30820]: pgmap v9572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T11:45:23.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:23 smithi114 ceph-mon[30820]: pgmap v9573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:23 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:45:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:45:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:45:23.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:45:23.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:23 smithi116 ceph-mon[35411]: pgmap v9572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T11:45:23.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:23 smithi116 ceph-mon[35411]: pgmap v9573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:45:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:25 smithi114 ceph-mon[30820]: pgmap v9574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:45:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:25 smithi116 ceph-mon[35411]: pgmap v9574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:45:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:27 smithi114 ceph-mon[30820]: pgmap v9575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:45:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:27 smithi116 ceph-mon[35411]: pgmap v9575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:45:29.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:29 smithi116 ceph-mon[35411]: pgmap v9576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:45:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:29 smithi114 ceph-mon[30820]: pgmap v9576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:45:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:31 smithi114 ceph-mon[30820]: pgmap v9577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:45:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:31 smithi116 ceph-mon[35411]: pgmap v9577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:45:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:33 smithi114 ceph-mon[30820]: pgmap v9578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:45:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:33 smithi116 ceph-mon[35411]: pgmap v9578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:45:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:35 smithi114 ceph-mon[30820]: pgmap v9579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:35 smithi116 ceph-mon[35411]: pgmap v9579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:37 smithi114 ceph-mon[30820]: pgmap v9580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:45:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:37 smithi116 ceph-mon[35411]: pgmap v9580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:45:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:39 smithi114 ceph-mon[30820]: pgmap v9581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:39 smithi116 ceph-mon[35411]: pgmap v9581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:41 smithi114 ceph-mon[30820]: pgmap v9582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:45:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:41 smithi116 ceph-mon[35411]: pgmap v9582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:45:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:43 smithi114 ceph-mon[30820]: pgmap v9583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:45:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:43 smithi116 ceph-mon[35411]: pgmap v9583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:45:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:45 smithi114 ceph-mon[30820]: pgmap v9584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:45 smithi116 ceph-mon[35411]: pgmap v9584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:47 smithi114 ceph-mon[30820]: pgmap v9585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:45:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:47 smithi116 ceph-mon[35411]: pgmap v9585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:45:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:49 smithi114 ceph-mon[30820]: pgmap v9586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:49 smithi116 ceph-mon[35411]: pgmap v9586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:51 smithi114 ceph-mon[30820]: pgmap v9587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:45:51.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:51 smithi116 ceph-mon[35411]: pgmap v9587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:45:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:53 smithi114 ceph-mon[30820]: pgmap v9588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:45:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:53 smithi116 ceph-mon[35411]: pgmap v9588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:45:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:55 smithi114 ceph-mon[30820]: pgmap v9589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:55 smithi116 ceph-mon[35411]: pgmap v9589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:57 smithi114 ceph-mon[30820]: pgmap v9590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:45:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:57 smithi116 ceph-mon[35411]: pgmap v9590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:45:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:45:59 smithi114 ceph-mon[30820]: pgmap v9591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:45:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:45:59 smithi116 ceph-mon[35411]: pgmap v9591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:01 smithi114 ceph-mon[30820]: pgmap v9592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:01 smithi116 ceph-mon[35411]: pgmap v9592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:03 smithi114 ceph-mon[30820]: pgmap v9593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:03 smithi116 ceph-mon[35411]: pgmap v9593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:05 smithi114 ceph-mon[30820]: pgmap v9594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:05 smithi116 ceph-mon[35411]: pgmap v9594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:07 smithi114 ceph-mon[30820]: pgmap v9595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:46:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:07 smithi116 ceph-mon[35411]: pgmap v9595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:46:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:09 smithi114 ceph-mon[30820]: pgmap v9596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:09 smithi116 ceph-mon[35411]: pgmap v9596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:11 smithi114 ceph-mon[30820]: pgmap v9597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:11 smithi116 ceph-mon[35411]: pgmap v9597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:13 smithi114 ceph-mon[30820]: pgmap v9598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:13 smithi116 ceph-mon[35411]: pgmap v9598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:15 smithi114 ceph-mon[30820]: pgmap v9599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:15 smithi116 ceph-mon[35411]: pgmap v9599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:17 smithi114 ceph-mon[30820]: pgmap v9600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:46:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:17 smithi116 ceph-mon[35411]: pgmap v9600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:46:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:19 smithi114 ceph-mon[30820]: pgmap v9601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:19 smithi116 ceph-mon[35411]: pgmap v9601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:21 smithi114 ceph-mon[30820]: pgmap v9602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:21 smithi116 ceph-mon[35411]: pgmap v9602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:46:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:46:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:23 smithi114 ceph-mon[30820]: pgmap v9603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:23 smithi116 ceph-mon[35411]: pgmap v9603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:24.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-mon[30820]: pgmap v9604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:46:24.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:24.983 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:24 smithi116 ceph-mon[35411]: pgmap v9604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:46:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:46:24.980+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:46:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:46:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:46:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:46:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:46:25.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:46:25.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:46:25.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:46:25.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:46:25.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:46:25.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:46:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:46:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:25 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:46:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:46:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:46:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:46:26.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:46:26.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:46:26.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:46:26.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:25 smithi114 ceph-mon[30820]: pgmap v9605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:46:26.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:25 smithi114 ceph-mon[30820]: pgmap v9606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T11:46:26.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:25 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:46:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:46:26.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:46:26.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:46:26.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:46:26.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:46:26.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:25 smithi116 ceph-mon[35411]: pgmap v9605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:46:26.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:25 smithi116 ceph-mon[35411]: pgmap v9606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T11:46:26.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:46:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:28 smithi114 ceph-mon[30820]: pgmap v9607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:46:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:28 smithi116 ceph-mon[35411]: pgmap v9607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:30 smithi114 ceph-mon[30820]: pgmap v9608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T11:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:30 smithi116 ceph-mon[35411]: pgmap v9608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T11:46:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:32 smithi114 ceph-mon[30820]: pgmap v9609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:46:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:32 smithi116 ceph-mon[35411]: pgmap v9609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:46:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:34 smithi114 ceph-mon[30820]: pgmap v9610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:46:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:34 smithi116 ceph-mon[35411]: pgmap v9610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:46:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:36 smithi114 ceph-mon[30820]: pgmap v9611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:46:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:36 smithi116 ceph-mon[35411]: pgmap v9611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:46:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:38 smithi114 ceph-mon[30820]: pgmap v9612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:38 smithi116 ceph-mon[35411]: pgmap v9612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:40 smithi114 ceph-mon[30820]: pgmap v9613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:40 smithi116 ceph-mon[35411]: pgmap v9613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:42 smithi114 ceph-mon[30820]: pgmap v9614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:46:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:42 smithi116 ceph-mon[35411]: pgmap v9614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:46:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:44 smithi114 ceph-mon[30820]: pgmap v9615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:44 smithi116 ceph-mon[35411]: pgmap v9615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:46 smithi114 ceph-mon[30820]: pgmap v9616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:46 smithi116 ceph-mon[35411]: pgmap v9616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:48 smithi114 ceph-mon[30820]: pgmap v9617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:48 smithi116 ceph-mon[35411]: pgmap v9617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:50 smithi114 ceph-mon[30820]: pgmap v9618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:50 smithi116 ceph-mon[35411]: pgmap v9618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:52 smithi114 ceph-mon[30820]: pgmap v9619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:46:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:52 smithi116 ceph-mon[35411]: pgmap v9619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:46:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:54 smithi114 ceph-mon[30820]: pgmap v9620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:54 smithi116 ceph-mon[35411]: pgmap v9620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:46:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:56 smithi114 ceph-mon[30820]: pgmap v9621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:56 smithi116 ceph-mon[35411]: pgmap v9621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:46:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:46:58 smithi114 ceph-mon[30820]: pgmap v9622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:46:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:46:58 smithi116 ceph-mon[35411]: pgmap v9622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:00 smithi114 ceph-mon[30820]: pgmap v9623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:00 smithi116 ceph-mon[35411]: pgmap v9623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:02 smithi114 ceph-mon[30820]: pgmap v9624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:47:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:02 smithi116 ceph-mon[35411]: pgmap v9624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:47:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:04 smithi114 ceph-mon[30820]: pgmap v9625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:04 smithi116 ceph-mon[35411]: pgmap v9625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:06.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:06 smithi114 ceph-mon[30820]: pgmap v9626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:47:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:06 smithi116 ceph-mon[35411]: pgmap v9626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:47:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:08 smithi114 ceph-mon[30820]: pgmap v9627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:08 smithi116 ceph-mon[35411]: pgmap v9627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:10 smithi114 ceph-mon[30820]: pgmap v9628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:10 smithi116 ceph-mon[35411]: pgmap v9628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:12 smithi114 ceph-mon[30820]: pgmap v9629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:47:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:12 smithi116 ceph-mon[35411]: pgmap v9629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:47:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:14 smithi114 ceph-mon[30820]: pgmap v9630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:14 smithi116 ceph-mon[35411]: pgmap v9630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:16 smithi114 ceph-mon[30820]: pgmap v9631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:47:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:16 smithi116 ceph-mon[35411]: pgmap v9631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:47:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:18 smithi114 ceph-mon[30820]: pgmap v9632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:18 smithi116 ceph-mon[35411]: pgmap v9632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:20 smithi114 ceph-mon[30820]: pgmap v9633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:20 smithi116 ceph-mon[35411]: pgmap v9633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:22 smithi114 ceph-mon[30820]: pgmap v9634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:47:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:22 smithi116 ceph-mon[35411]: pgmap v9634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:47:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:24 smithi114 ceph-mon[30820]: pgmap v9635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:47:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:24 smithi116 ceph-mon[35411]: pgmap v9635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:47:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:47:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:47:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:26 smithi114 ceph-mon[30820]: pgmap v9636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:47:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:26 smithi116 ceph-mon[35411]: pgmap v9636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:47:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:27 smithi114 ceph-mon[30820]: pgmap v9637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:27 smithi116 ceph-mon[35411]: pgmap v9637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:28.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:47:28.276+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:47:28.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:47:28.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:47:28.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:47:28.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:47:28.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:47:28.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:47:28.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:47:28.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:47:28.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:47:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:47:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:47:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:47:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:47:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-mon[30820]: pgmap v9638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-mon[30820]: pgmap v9639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:47:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:47:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:47:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:47:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:47:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:28 smithi116 ceph-mon[35411]: pgmap v9638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:28 smithi116 ceph-mon[35411]: pgmap v9639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T11:47:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:47:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:31 smithi114 ceph-mon[30820]: pgmap v9640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:47:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:31 smithi116 ceph-mon[35411]: pgmap v9640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:47:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:33 smithi114 ceph-mon[30820]: pgmap v9641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:47:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:33 smithi116 ceph-mon[35411]: pgmap v9641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:47:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:35 smithi114 ceph-mon[30820]: pgmap v9642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:47:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:35 smithi116 ceph-mon[35411]: pgmap v9642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:47:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:37 smithi114 ceph-mon[30820]: pgmap v9643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:47:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:37 smithi116 ceph-mon[35411]: pgmap v9643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:47:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:39 smithi114 ceph-mon[30820]: pgmap v9644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:47:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:39 smithi116 ceph-mon[35411]: pgmap v9644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:47:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:41 smithi114 ceph-mon[30820]: pgmap v9645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:47:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:41 smithi116 ceph-mon[35411]: pgmap v9645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:47:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:43 smithi114 ceph-mon[30820]: pgmap v9646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:47:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:43 smithi116 ceph-mon[35411]: pgmap v9646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:47:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:45 smithi114 ceph-mon[30820]: pgmap v9647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:47:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:45 smithi116 ceph-mon[35411]: pgmap v9647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:47:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:47 smithi114 ceph-mon[30820]: pgmap v9648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:47 smithi116 ceph-mon[35411]: pgmap v9648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:49 smithi114 ceph-mon[30820]: pgmap v9649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:49 smithi116 ceph-mon[35411]: pgmap v9649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:51 smithi114 ceph-mon[30820]: pgmap v9650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:47:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:51 smithi116 ceph-mon[35411]: pgmap v9650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:47:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:53 smithi114 ceph-mon[30820]: pgmap v9651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:47:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:53 smithi116 ceph-mon[35411]: pgmap v9651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:47:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:55 smithi114 ceph-mon[30820]: pgmap v9652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:47:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:55 smithi116 ceph-mon[35411]: pgmap v9652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:47:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:57 smithi114 ceph-mon[30820]: pgmap v9653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:57 smithi116 ceph-mon[35411]: pgmap v9653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:47:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:47:59 smithi114 ceph-mon[30820]: pgmap v9654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:47:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:47:59 smithi116 ceph-mon[35411]: pgmap v9654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:01 smithi114 ceph-mon[30820]: pgmap v9655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:48:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:01 smithi116 ceph-mon[35411]: pgmap v9655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:48:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:03 smithi114 ceph-mon[30820]: pgmap v9656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:48:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:03 smithi116 ceph-mon[35411]: pgmap v9656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:48:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:05 smithi114 ceph-mon[30820]: pgmap v9657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:48:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:05 smithi116 ceph-mon[35411]: pgmap v9657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:48:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:07 smithi114 ceph-mon[30820]: pgmap v9658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:48:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:07 smithi116 ceph-mon[35411]: pgmap v9658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:48:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:09 smithi114 ceph-mon[30820]: pgmap v9659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:09 smithi116 ceph-mon[35411]: pgmap v9659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:11 smithi114 ceph-mon[30820]: pgmap v9660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:48:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:11 smithi116 ceph-mon[35411]: pgmap v9660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:48:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:13 smithi116 ceph-mon[35411]: pgmap v9661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:48:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:13 smithi114 ceph-mon[30820]: pgmap v9661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:48:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:15 smithi116 ceph-mon[35411]: pgmap v9662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:48:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:15 smithi114 ceph-mon[30820]: pgmap v9662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:48:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:17 smithi116 ceph-mon[35411]: pgmap v9663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:48:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:17 smithi114 ceph-mon[30820]: pgmap v9663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:48:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:19 smithi114 ceph-mon[30820]: pgmap v9664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:19 smithi116 ceph-mon[35411]: pgmap v9664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:21 smithi114 ceph-mon[30820]: pgmap v9665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:48:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:21 smithi116 ceph-mon[35411]: pgmap v9665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:48:23.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:23 smithi114 ceph-mon[30820]: pgmap v9666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:48:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:23 smithi116 ceph-mon[35411]: pgmap v9666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:48:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:25 smithi114 ceph-mon[30820]: pgmap v9667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:48:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:25 smithi116 ceph-mon[35411]: pgmap v9667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:48:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:27 smithi114 ceph-mon[30820]: pgmap v9668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:48:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:27 smithi116 ceph-mon[35411]: pgmap v9668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:48:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:48:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:48:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:29 smithi114 ceph-mon[30820]: pgmap v9669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:48:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:48:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:29 smithi116 ceph-mon[35411]: pgmap v9669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:48:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:48:31.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:31 smithi116 ceph-mon[35411]: pgmap v9670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:48:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-mon[30820]: pgmap v9670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:48:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:48:31.637+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:48:31.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:48:31.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:48:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:48:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:48:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:48:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:48:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:48:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:48:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:48:31.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:48:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:48:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:48:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:48:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:48:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:48:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:48:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:48:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:33 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:48:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:48:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:48:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:48:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:48:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:48:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:48:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:48:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:48:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:48:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:33 smithi114 ceph-mon[30820]: pgmap v9671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T11:48:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:33 smithi114 ceph-mon[30820]: pgmap v9672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:48:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:33 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:48:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:48:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:48:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:48:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:48:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:48:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:48:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:48:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:48:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:48:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:33 smithi116 ceph-mon[35411]: pgmap v9671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T11:48:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:33 smithi116 ceph-mon[35411]: pgmap v9672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:48:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:35 smithi114 ceph-mon[30820]: pgmap v9673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:48:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:35 smithi116 ceph-mon[35411]: pgmap v9673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:48:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:37 smithi114 ceph-mon[30820]: pgmap v9674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:48:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:37 smithi116 ceph-mon[35411]: pgmap v9674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:48:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:39 smithi114 ceph-mon[30820]: pgmap v9675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:48:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:39 smithi116 ceph-mon[35411]: pgmap v9675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:48:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:41 smithi114 ceph-mon[30820]: pgmap v9676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:48:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:41 smithi116 ceph-mon[35411]: pgmap v9676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:48:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:43 smithi114 ceph-mon[30820]: pgmap v9677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:48:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:43 smithi116 ceph-mon[35411]: pgmap v9677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:48:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:45 smithi114 ceph-mon[30820]: pgmap v9678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:45 smithi116 ceph-mon[35411]: pgmap v9678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:47 smithi114 ceph-mon[30820]: pgmap v9679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:48:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:47 smithi116 ceph-mon[35411]: pgmap v9679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:48:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:49 smithi114 ceph-mon[30820]: pgmap v9680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:49 smithi116 ceph-mon[35411]: pgmap v9680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:51 smithi114 ceph-mon[30820]: pgmap v9681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:48:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:51 smithi116 ceph-mon[35411]: pgmap v9681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:48:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:53 smithi114 ceph-mon[30820]: pgmap v9682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:48:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:53 smithi116 ceph-mon[35411]: pgmap v9682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:48:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:55 smithi114 ceph-mon[30820]: pgmap v9683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:55 smithi116 ceph-mon[35411]: pgmap v9683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:57 smithi114 ceph-mon[30820]: pgmap v9684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:48:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:57 smithi116 ceph-mon[35411]: pgmap v9684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:48:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:48:59 smithi114 ceph-mon[30820]: pgmap v9685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:48:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:48:59 smithi116 ceph-mon[35411]: pgmap v9685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:01 smithi114 ceph-mon[30820]: pgmap v9686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:01 smithi116 ceph-mon[35411]: pgmap v9686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:03 smithi114 ceph-mon[30820]: pgmap v9687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:03 smithi116 ceph-mon[35411]: pgmap v9687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:05 smithi114 ceph-mon[30820]: pgmap v9688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:05 smithi116 ceph-mon[35411]: pgmap v9688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:07 smithi114 ceph-mon[30820]: pgmap v9689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:49:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:07 smithi116 ceph-mon[35411]: pgmap v9689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:49:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:09 smithi114 ceph-mon[30820]: pgmap v9690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:09 smithi116 ceph-mon[35411]: pgmap v9690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:11 smithi114 ceph-mon[30820]: pgmap v9691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:11.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:11 smithi116 ceph-mon[35411]: pgmap v9691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:13 smithi114 ceph-mon[30820]: pgmap v9692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:13 smithi116 ceph-mon[35411]: pgmap v9692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:15 smithi114 ceph-mon[30820]: pgmap v9693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:15 smithi116 ceph-mon[35411]: pgmap v9693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:17 smithi114 ceph-mon[30820]: pgmap v9694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:49:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:17 smithi116 ceph-mon[35411]: pgmap v9694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:49:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:19 smithi114 ceph-mon[30820]: pgmap v9695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:19 smithi116 ceph-mon[35411]: pgmap v9695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:21 smithi114 ceph-mon[30820]: pgmap v9696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:21 smithi116 ceph-mon[35411]: pgmap v9696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:23 smithi114 ceph-mon[30820]: pgmap v9697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:23 smithi116 ceph-mon[35411]: pgmap v9697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:25 smithi114 ceph-mon[30820]: pgmap v9698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:25 smithi116 ceph-mon[35411]: pgmap v9698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:27 smithi114 ceph-mon[30820]: pgmap v9699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:49:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:27 smithi116 ceph-mon[35411]: pgmap v9699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:49:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:29 smithi114 ceph-mon[30820]: pgmap v9700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:29 smithi116 ceph-mon[35411]: pgmap v9700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:31 smithi114 ceph-mon[30820]: pgmap v9701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:31 smithi116 ceph-mon[35411]: pgmap v9701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:49:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:49:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:33 smithi114 ceph-mon[30820]: pgmap v9702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:33 smithi116 ceph-mon[35411]: pgmap v9702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:49:34.996+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:49:35.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:49:35.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:49:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:49:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:35 smithi114 ceph-mon[30820]: pgmap v9703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:49:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:49:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:49:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:35 smithi116 ceph-mon[35411]: pgmap v9703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:49:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:49:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:49:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:49:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:49:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:49:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:49:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:49:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:49:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:49:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:49:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:49:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:49:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:36 smithi114 ceph-mon[30820]: pgmap v9704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:49:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:36 smithi114 ceph-mon[30820]: pgmap v9705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:49:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:49:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:49:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:49:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:49:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:49:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:49:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:49:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:49:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:49:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:49:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:49:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:36 smithi116 ceph-mon[35411]: pgmap v9704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:49:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:36 smithi116 ceph-mon[35411]: pgmap v9705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:49:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:38 smithi114 ceph-mon[30820]: pgmap v9706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:49:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:38 smithi116 ceph-mon[35411]: pgmap v9706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:49:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:40 smithi114 ceph-mon[30820]: pgmap v9707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:49:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:40 smithi116 ceph-mon[35411]: pgmap v9707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:49:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:42 smithi114 ceph-mon[30820]: pgmap v9708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:49:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:42 smithi116 ceph-mon[35411]: pgmap v9708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:49:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:44 smithi114 ceph-mon[30820]: pgmap v9709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:49:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:44 smithi116 ceph-mon[35411]: pgmap v9709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:49:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:46 smithi114 ceph-mon[30820]: pgmap v9710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:49:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:46 smithi116 ceph-mon[35411]: pgmap v9710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:49:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:48 smithi114 ceph-mon[30820]: pgmap v9711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:48 smithi116 ceph-mon[35411]: pgmap v9711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:50 smithi114 ceph-mon[30820]: pgmap v9712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:50 smithi116 ceph-mon[35411]: pgmap v9712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:52 smithi114 ceph-mon[30820]: pgmap v9713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:49:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:52 smithi116 ceph-mon[35411]: pgmap v9713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:49:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:54 smithi114 ceph-mon[30820]: pgmap v9714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:54 smithi116 ceph-mon[35411]: pgmap v9714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:49:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:56 smithi114 ceph-mon[30820]: pgmap v9715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:56 smithi116 ceph-mon[35411]: pgmap v9715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:49:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:49:58 smithi114 ceph-mon[30820]: pgmap v9716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:49:58.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:49:58 smithi116 ceph-mon[35411]: pgmap v9716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:00 smithi114 ceph-mon[30820]: pgmap v9717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:50:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:00 smithi116 ceph-mon[35411]: pgmap v9717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T11:50:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:02 smithi114 ceph-mon[30820]: pgmap v9718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:50:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:02 smithi116 ceph-mon[35411]: pgmap v9718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:50:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:03 smithi114 ceph-mon[30820]: pgmap v9719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:03 smithi116 ceph-mon[35411]: pgmap v9719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:06 smithi114 ceph-mon[30820]: pgmap v9720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:06 smithi116 ceph-mon[35411]: pgmap v9720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:08 smithi114 ceph-mon[30820]: pgmap v9721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:08 smithi116 ceph-mon[35411]: pgmap v9721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:10 smithi114 ceph-mon[30820]: pgmap v9722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:10 smithi116 ceph-mon[35411]: pgmap v9722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:12 smithi114 ceph-mon[30820]: pgmap v9723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:50:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:12 smithi116 ceph-mon[35411]: pgmap v9723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:50:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:14 smithi114 ceph-mon[30820]: pgmap v9724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:14 smithi116 ceph-mon[35411]: pgmap v9724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:16.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:16 smithi114 ceph-mon[30820]: pgmap v9725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:16 smithi116 ceph-mon[35411]: pgmap v9725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:18 smithi114 ceph-mon[30820]: pgmap v9726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:18 smithi116 ceph-mon[35411]: pgmap v9726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:20 smithi114 ceph-mon[30820]: pgmap v9727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:20 smithi116 ceph-mon[35411]: pgmap v9727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:22 smithi114 ceph-mon[30820]: pgmap v9728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:50:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:22 smithi116 ceph-mon[35411]: pgmap v9728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:50:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:24 smithi114 ceph-mon[30820]: pgmap v9729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:24 smithi116 ceph-mon[35411]: pgmap v9729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:26 smithi114 ceph-mon[30820]: pgmap v9730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:26 smithi116 ceph-mon[35411]: pgmap v9730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:28 smithi114 ceph-mon[30820]: pgmap v9731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:28 smithi116 ceph-mon[35411]: pgmap v9731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:30 smithi114 ceph-mon[30820]: pgmap v9732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:30 smithi116 ceph-mon[35411]: pgmap v9732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:32 smithi114 ceph-mon[30820]: pgmap v9733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:50:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:32 smithi116 ceph-mon[35411]: pgmap v9733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:50:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:34 smithi114 ceph-mon[30820]: pgmap v9734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:34 smithi116 ceph-mon[35411]: pgmap v9734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:36 smithi114 ceph-mon[30820]: pgmap v9735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:50:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:36 smithi116 ceph-mon[35411]: pgmap v9735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:50:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-mon[30820]: pgmap v9736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:50:38.292+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:50:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:50:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:50:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:50:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:50:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:50:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:50:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:50:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:50:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:50:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:50:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:50:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:50:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:50:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:50:38.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:50:38.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:50:38.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:50:38.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:50:38.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:50:38.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:50:38.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:50:38.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:50:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:50:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:38 smithi116 ceph-mon[35411]: pgmap v9736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:50:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:50:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:50:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:50:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:50:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:50:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:50:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:50:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:50:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:50:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:50:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:50:39.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:50:39.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:50:39.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:50:39.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:50:39.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:50:39.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:50:39.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:50:39.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:50:39.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:50:39.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:50:39.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:50:39.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:50:39.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:50:39.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:39 smithi114 ceph-mon[30820]: pgmap v9737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:50:39.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:39 smithi114 ceph-mon[30820]: pgmap v9738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:50:39.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:50:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:50:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:50:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:50:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:50:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:50:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:50:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:50:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:50:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:50:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:50:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:50:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:50:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:50:39.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:50:39.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:50:39.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:50:39.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:50:39.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:50:39.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:50:39.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:50:39.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:50:39.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:50:39.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:50:39.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:50:39.433 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:50:39.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:39 smithi116 ceph-mon[35411]: pgmap v9737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:50:39.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:39 smithi116 ceph-mon[35411]: pgmap v9738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:50:39.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:50:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:41 smithi114 ceph-mon[30820]: pgmap v9739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:50:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:41 smithi116 ceph-mon[35411]: pgmap v9739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:50:43.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:43 smithi114 ceph-mon[30820]: pgmap v9740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:50:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:43 smithi116 ceph-mon[35411]: pgmap v9740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:50:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:45 smithi114 ceph-mon[30820]: pgmap v9741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:50:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:45 smithi116 ceph-mon[35411]: pgmap v9741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:50:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:47 smithi114 ceph-mon[30820]: pgmap v9742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:50:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:47 smithi116 ceph-mon[35411]: pgmap v9742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:49 smithi114 ceph-mon[30820]: pgmap v9743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:49 smithi116 ceph-mon[35411]: pgmap v9743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:50:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:51 smithi114 ceph-mon[30820]: pgmap v9744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:51 smithi116 ceph-mon[35411]: pgmap v9744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:50:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:53 smithi114 ceph-mon[30820]: pgmap v9745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:50:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:53 smithi116 ceph-mon[35411]: pgmap v9745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:50:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:55 smithi114 ceph-mon[30820]: pgmap v9746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:55 smithi116 ceph-mon[35411]: pgmap v9746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:50:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:57 smithi114 ceph-mon[30820]: pgmap v9747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:57 smithi116 ceph-mon[35411]: pgmap v9747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:50:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:50:59 smithi114 ceph-mon[30820]: pgmap v9748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:50:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:50:59 smithi116 ceph-mon[35411]: pgmap v9748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:01 smithi114 ceph-mon[30820]: pgmap v9749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:01 smithi116 ceph-mon[35411]: pgmap v9749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:03 smithi114 ceph-mon[30820]: pgmap v9750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:51:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:03 smithi116 ceph-mon[35411]: pgmap v9750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:51:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:05 smithi114 ceph-mon[30820]: pgmap v9751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:51:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:05 smithi116 ceph-mon[35411]: pgmap v9751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:51:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:07 smithi114 ceph-mon[30820]: pgmap v9752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:51:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:07 smithi116 ceph-mon[35411]: pgmap v9752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:51:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:09 smithi114 ceph-mon[30820]: pgmap v9753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:09 smithi116 ceph-mon[35411]: pgmap v9753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:11 smithi114 ceph-mon[30820]: pgmap v9754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:11 smithi116 ceph-mon[35411]: pgmap v9754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:13 smithi114 ceph-mon[30820]: pgmap v9755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:51:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:13 smithi116 ceph-mon[35411]: pgmap v9755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:51:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:15 smithi116 ceph-mon[35411]: pgmap v9756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:51:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:15 smithi114 ceph-mon[30820]: pgmap v9756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:51:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:17 smithi116 ceph-mon[35411]: pgmap v9757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:51:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:17 smithi114 ceph-mon[30820]: pgmap v9757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:51:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:19 smithi116 ceph-mon[35411]: pgmap v9758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:19 smithi114 ceph-mon[30820]: pgmap v9758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:21 smithi114 ceph-mon[30820]: pgmap v9759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:21 smithi116 ceph-mon[35411]: pgmap v9759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:23 smithi114 ceph-mon[30820]: pgmap v9760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:51:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:23 smithi116 ceph-mon[35411]: pgmap v9760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:51:25.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:25 smithi114 ceph-mon[30820]: pgmap v9761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:51:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:25 smithi116 ceph-mon[35411]: pgmap v9761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:51:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:27 smithi114 ceph-mon[30820]: pgmap v9762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:51:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:27 smithi116 ceph-mon[35411]: pgmap v9762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:51:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:29 smithi114 ceph-mon[30820]: pgmap v9763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:29 smithi116 ceph-mon[35411]: pgmap v9763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:31 smithi114 ceph-mon[30820]: pgmap v9764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:31 smithi116 ceph-mon[35411]: pgmap v9764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:33.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:33 smithi116 ceph-mon[35411]: pgmap v9765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:51:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:33 smithi114 ceph-mon[30820]: pgmap v9765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T11:51:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:35 smithi114 ceph-mon[30820]: pgmap v9766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:51:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:35 smithi116 ceph-mon[35411]: pgmap v9766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:51:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:37 smithi114 ceph-mon[30820]: pgmap v9767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:51:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:37 smithi116 ceph-mon[35411]: pgmap v9767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:51:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:51:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:51:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:39 smithi114 ceph-mon[30820]: pgmap v9768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:39 smithi116 ceph-mon[35411]: pgmap v9768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-mon[30820]: pgmap v9769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:41.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:51:41.653+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:51:41.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:51:41.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:51:41.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:51:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:51:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:51:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:51:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:51:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:51:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:51:41.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:51:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:51:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:41 smithi116 ceph-mon[35411]: pgmap v9769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T11:51:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:42 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:51:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:51:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:51:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:51:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:51:42.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:51:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:42 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:51:42.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:51:42.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:51:42.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:51:42.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:51:42.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:51:42.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:51:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:43 smithi114 ceph-mon[30820]: pgmap v9770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T11:51:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:43 smithi114 ceph-mon[30820]: pgmap v9771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:51:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:43 smithi116 ceph-mon[35411]: pgmap v9770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T11:51:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:43 smithi116 ceph-mon[35411]: pgmap v9771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:51:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:45 smithi114 ceph-mon[30820]: pgmap v9772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:51:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:45 smithi116 ceph-mon[35411]: pgmap v9772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:51:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:47 smithi114 ceph-mon[30820]: pgmap v9773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:51:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:47 smithi116 ceph-mon[35411]: pgmap v9773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:51:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:49 smithi114 ceph-mon[30820]: pgmap v9774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:51:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:49 smithi116 ceph-mon[35411]: pgmap v9774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:51:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:51 smithi114 ceph-mon[30820]: pgmap v9775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:51:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:51 smithi116 ceph-mon[35411]: pgmap v9775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T11:51:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:53 smithi114 ceph-mon[30820]: pgmap v9776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:51:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:53 smithi116 ceph-mon[35411]: pgmap v9776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:51:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:55 smithi114 ceph-mon[30820]: pgmap v9777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:55 smithi116 ceph-mon[35411]: pgmap v9777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:57 smithi114 ceph-mon[30820]: pgmap v9778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:51:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:57 smithi116 ceph-mon[35411]: pgmap v9778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:51:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:51:59 smithi114 ceph-mon[30820]: pgmap v9779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:51:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:51:59 smithi116 ceph-mon[35411]: pgmap v9779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:01 smithi114 ceph-mon[30820]: pgmap v9780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:01 smithi116 ceph-mon[35411]: pgmap v9780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:03 smithi114 ceph-mon[30820]: pgmap v9781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:03 smithi116 ceph-mon[35411]: pgmap v9781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:05 smithi114 ceph-mon[30820]: pgmap v9782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:05 smithi116 ceph-mon[35411]: pgmap v9782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:07 smithi114 ceph-mon[30820]: pgmap v9783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:52:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:07 smithi116 ceph-mon[35411]: pgmap v9783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:52:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:09 smithi114 ceph-mon[30820]: pgmap v9784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:09 smithi116 ceph-mon[35411]: pgmap v9784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:11 smithi114 ceph-mon[30820]: pgmap v9785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:11 smithi116 ceph-mon[35411]: pgmap v9785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:13 smithi114 ceph-mon[30820]: pgmap v9786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:13 smithi116 ceph-mon[35411]: pgmap v9786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:15 smithi114 ceph-mon[30820]: pgmap v9787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:15 smithi116 ceph-mon[35411]: pgmap v9787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:17 smithi114 ceph-mon[30820]: pgmap v9788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:52:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:17 smithi116 ceph-mon[35411]: pgmap v9788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:52:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:19 smithi114 ceph-mon[30820]: pgmap v9789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:19 smithi116 ceph-mon[35411]: pgmap v9789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:21 smithi114 ceph-mon[30820]: pgmap v9790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:21 smithi116 ceph-mon[35411]: pgmap v9790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:23 smithi114 ceph-mon[30820]: pgmap v9791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:23 smithi116 ceph-mon[35411]: pgmap v9791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:25 smithi114 ceph-mon[30820]: pgmap v9792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:25 smithi116 ceph-mon[35411]: pgmap v9792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:27 smithi114 ceph-mon[30820]: pgmap v9793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:52:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:27 smithi116 ceph-mon[35411]: pgmap v9793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:52:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:29 smithi114 ceph-mon[30820]: pgmap v9794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:29 smithi116 ceph-mon[35411]: pgmap v9794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:31 smithi114 ceph-mon[30820]: pgmap v9795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:31 smithi116 ceph-mon[35411]: pgmap v9795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:33 smithi114 ceph-mon[30820]: pgmap v9796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:33 smithi116 ceph-mon[35411]: pgmap v9796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:35 smithi114 ceph-mon[30820]: pgmap v9797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:35 smithi116 ceph-mon[35411]: pgmap v9797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:37 smithi114 ceph-mon[30820]: pgmap v9798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:52:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:37 smithi116 ceph-mon[35411]: pgmap v9798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:52:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:39 smithi114 ceph-mon[30820]: pgmap v9799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:39 smithi116 ceph-mon[35411]: pgmap v9799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:41 smithi114 ceph-mon[30820]: pgmap v9800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:41 smithi116 ceph-mon[35411]: pgmap v9800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:52:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:52:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:52:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:43 smithi114 ceph-mon[30820]: pgmap v9801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:43 smithi116 ceph-mon[35411]: pgmap v9801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:52:45.013+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:52:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:52:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:52:45.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:52:45.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:52:45.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:52:45.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:52:45.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:52:45.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:52:45.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:52:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-mon[30820]: pgmap v9802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:52:45.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:52:45.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:52:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:45 smithi116 ceph-mon[35411]: pgmap v9802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:52:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:52:45.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:52:45.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:52:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:52:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:52:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:52:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:52:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:52:46.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:52:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:52:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:52:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:52:46.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:46 smithi114 ceph-mon[30820]: pgmap v9803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:52:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:52:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:52:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:52:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:52:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:52:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:52:46.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:52:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:52:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:52:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:52:46.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:46 smithi116 ceph-mon[35411]: pgmap v9803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:52:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:48 smithi114 ceph-mon[30820]: pgmap v9804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:52:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:48 smithi116 ceph-mon[35411]: pgmap v9804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:52:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:50 smithi114 ceph-mon[30820]: pgmap v9805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:52:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:50 smithi116 ceph-mon[35411]: pgmap v9805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:52:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:52 smithi114 ceph-mon[30820]: pgmap v9806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:52:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:52 smithi116 ceph-mon[35411]: pgmap v9806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T11:52:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:54 smithi114 ceph-mon[30820]: pgmap v9807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:52:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:54 smithi116 ceph-mon[35411]: pgmap v9807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:52:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:56 smithi114 ceph-mon[30820]: pgmap v9808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:52:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:56 smithi116 ceph-mon[35411]: pgmap v9808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:52:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:52:58 smithi114 ceph-mon[30820]: pgmap v9809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:52:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:52:58 smithi116 ceph-mon[35411]: pgmap v9809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:00 smithi114 ceph-mon[30820]: pgmap v9810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:00 smithi116 ceph-mon[35411]: pgmap v9810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:02 smithi114 ceph-mon[30820]: pgmap v9811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:02 smithi116 ceph-mon[35411]: pgmap v9811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:04 smithi114 ceph-mon[30820]: pgmap v9812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:04 smithi116 ceph-mon[35411]: pgmap v9812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:06 smithi114 ceph-mon[30820]: pgmap v9813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:06 smithi116 ceph-mon[35411]: pgmap v9813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:08 smithi114 ceph-mon[30820]: pgmap v9814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:08 smithi116 ceph-mon[35411]: pgmap v9814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:10 smithi114 ceph-mon[30820]: pgmap v9815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:10 smithi116 ceph-mon[35411]: pgmap v9815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:12 smithi114 ceph-mon[30820]: pgmap v9816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:12 smithi116 ceph-mon[35411]: pgmap v9816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:14 smithi114 ceph-mon[30820]: pgmap v9817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:14 smithi116 ceph-mon[35411]: pgmap v9817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:16 smithi114 ceph-mon[30820]: pgmap v9818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:16 smithi116 ceph-mon[35411]: pgmap v9818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:18 smithi114 ceph-mon[30820]: pgmap v9819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:18 smithi116 ceph-mon[35411]: pgmap v9819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:20 smithi114 ceph-mon[30820]: pgmap v9820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:20 smithi116 ceph-mon[35411]: pgmap v9820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:22 smithi114 ceph-mon[30820]: pgmap v9821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:22 smithi116 ceph-mon[35411]: pgmap v9821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:24 smithi116 ceph-mon[35411]: pgmap v9822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:24 smithi114 ceph-mon[30820]: pgmap v9822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:26 smithi116 ceph-mon[35411]: pgmap v9823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:26.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:26 smithi114 ceph-mon[30820]: pgmap v9823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:28 smithi116 ceph-mon[35411]: pgmap v9824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:28 smithi114 ceph-mon[30820]: pgmap v9824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:29 smithi114 ceph-mon[30820]: pgmap v9825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:30.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:29 smithi116 ceph-mon[35411]: pgmap v9825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:32 smithi114 ceph-mon[30820]: pgmap v9826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:32 smithi116 ceph-mon[35411]: pgmap v9826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:34 smithi114 ceph-mon[30820]: pgmap v9827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:34 smithi116 ceph-mon[35411]: pgmap v9827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:36 smithi114 ceph-mon[30820]: pgmap v9828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:36 smithi116 ceph-mon[35411]: pgmap v9828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:38 smithi114 ceph-mon[30820]: pgmap v9829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:38 smithi116 ceph-mon[35411]: pgmap v9829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:40 smithi114 ceph-mon[30820]: pgmap v9830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:40 smithi116 ceph-mon[35411]: pgmap v9830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:42 smithi114 ceph-mon[30820]: pgmap v9831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:42 smithi116 ceph-mon[35411]: pgmap v9831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:53:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:44 smithi114 ceph-mon[30820]: pgmap v9832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:44 smithi116 ceph-mon[35411]: pgmap v9832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:46 smithi116 ceph-mon[35411]: pgmap v9833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:53:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:46 smithi114 ceph-mon[30820]: pgmap v9833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:53:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:53:48.201 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-mon[30820]: pgmap v9834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:48 smithi116 ceph-mon[35411]: pgmap v9834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:53:48.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:53:48.372+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:53:48.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:53:48.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:53:48.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:53:48.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:53:48.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:53:48.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:53:48.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:53:48.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:53:48.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:53:48.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:53:48.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:53:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:53:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:53:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:53:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:53:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:53:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:53:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:49.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:53:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:53:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:53:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:53:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:53:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:53:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:53:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:53:49.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:53:49.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:53:49.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:49.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:49.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:49.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:49.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:49.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:50 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:53:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:50 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:53:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:50 smithi114 ceph-mon[30820]: pgmap v9835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:50 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T11:53:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:50 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T11:53:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:50 smithi116 ceph-mon[35411]: pgmap v9835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:53:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: pgmap v9836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 7 op/s 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: pgmap v9837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T11:53:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T11:53:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T11:53:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:53:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:53:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: pgmap v9836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 7 op/s 2024-09-19T11:53:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: pgmap v9837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:53:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:53 smithi114 ceph-mon[30820]: pgmap v9838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:53 smithi116 ceph-mon[35411]: pgmap v9838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:55 smithi114 ceph-mon[30820]: pgmap v9839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:55 smithi116 ceph-mon[35411]: pgmap v9839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:57 smithi114 ceph-mon[30820]: pgmap v9840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:57 smithi116 ceph-mon[35411]: pgmap v9840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:53:59 smithi114 ceph-mon[30820]: pgmap v9841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:53:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:53:59 smithi116 ceph-mon[35411]: pgmap v9841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:54:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:01 smithi114 ceph-mon[30820]: pgmap v9842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:54:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:01 smithi116 ceph-mon[35411]: pgmap v9842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:54:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:03 smithi114 ceph-mon[30820]: pgmap v9843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:03 smithi116 ceph-mon[35411]: pgmap v9843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:05 smithi114 ceph-mon[30820]: pgmap v9844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:54:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:05 smithi116 ceph-mon[35411]: pgmap v9844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:54:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:07 smithi114 ceph-mon[30820]: pgmap v9845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:07 smithi116 ceph-mon[35411]: pgmap v9845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:09 smithi114 ceph-mon[30820]: pgmap v9846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:09 smithi116 ceph-mon[35411]: pgmap v9846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:11 smithi114 ceph-mon[30820]: pgmap v9847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:11 smithi116 ceph-mon[35411]: pgmap v9847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:13 smithi114 ceph-mon[30820]: pgmap v9848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:13 smithi116 ceph-mon[35411]: pgmap v9848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:15 smithi114 ceph-mon[30820]: pgmap v9849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:54:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:15 smithi116 ceph-mon[35411]: pgmap v9849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:54:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:17 smithi114 ceph-mon[30820]: pgmap v9850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:17 smithi116 ceph-mon[35411]: pgmap v9850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:19 smithi114 ceph-mon[30820]: pgmap v9851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:54:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:19 smithi116 ceph-mon[35411]: pgmap v9851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:54:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:21 smithi114 ceph-mon[30820]: pgmap v9852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:21 smithi116 ceph-mon[35411]: pgmap v9852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:23 smithi114 ceph-mon[30820]: pgmap v9853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:54:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:23 smithi116 ceph-mon[35411]: pgmap v9853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:54:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:25 smithi114 ceph-mon[30820]: pgmap v9854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:54:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:25 smithi116 ceph-mon[35411]: pgmap v9854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:54:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:27 smithi114 ceph-mon[30820]: pgmap v9855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:27 smithi116 ceph-mon[35411]: pgmap v9855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:29 smithi116 ceph-mon[35411]: pgmap v9856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:29 smithi114 ceph-mon[30820]: pgmap v9856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:31 smithi116 ceph-mon[35411]: pgmap v9857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:31 smithi114 ceph-mon[30820]: pgmap v9857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:33 smithi116 ceph-mon[35411]: pgmap v9858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:33 smithi114 ceph-mon[30820]: pgmap v9858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:35 smithi114 ceph-mon[30820]: pgmap v9859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:54:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:35 smithi116 ceph-mon[35411]: pgmap v9859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:54:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:37 smithi114 ceph-mon[30820]: pgmap v9860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:37 smithi116 ceph-mon[35411]: pgmap v9860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:39 smithi114 ceph-mon[30820]: pgmap v9861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:39 smithi116 ceph-mon[35411]: pgmap v9861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:41 smithi114 ceph-mon[30820]: pgmap v9862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:41 smithi116 ceph-mon[35411]: pgmap v9862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:42 smithi114 ceph-mon[30820]: pgmap v9863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:42 smithi116 ceph-mon[35411]: pgmap v9863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:45 smithi114 ceph-mon[30820]: pgmap v9864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:54:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:45 smithi116 ceph-mon[35411]: pgmap v9864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:54:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:47 smithi114 ceph-mon[30820]: pgmap v9865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:47 smithi116 ceph-mon[35411]: pgmap v9865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:54:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:49 smithi114 ceph-mon[30820]: pgmap v9866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:49 smithi116 ceph-mon[35411]: pgmap v9866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:54:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-mon[30820]: pgmap v9867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:54:51.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:54:51.796+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:54:51.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:54:51.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:54:51.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:54:51.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:54:51.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:54:51.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:54:51.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:54:51.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:54:51.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:54:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:54:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:51 smithi116 ceph-mon[35411]: pgmap v9867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:54:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:54:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:54:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:54:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:54:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:54:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:54:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:54:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:53 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:54:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:54:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:54:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:54:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:54:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:54:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:54:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:54:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:54:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:54:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:54:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:53 smithi114 ceph-mon[30820]: pgmap v9868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:54:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:53 smithi114 ceph-mon[30820]: pgmap v9869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:54:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:53 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:54:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:54:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:54:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:54:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:54:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:54:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:54:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:54:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:54:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:54:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:54:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:53 smithi116 ceph-mon[35411]: pgmap v9868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:54:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:53 smithi116 ceph-mon[35411]: pgmap v9869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:54:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:55 smithi114 ceph-mon[30820]: pgmap v9870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:54:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:55 smithi116 ceph-mon[35411]: pgmap v9870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:54:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:57 smithi114 ceph-mon[30820]: pgmap v9871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:54:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:57 smithi116 ceph-mon[35411]: pgmap v9871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:54:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:54:59 smithi114 ceph-mon[30820]: pgmap v9872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:54:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:54:59 smithi116 ceph-mon[35411]: pgmap v9872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:55:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:01 smithi114 ceph-mon[30820]: pgmap v9873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:55:01.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:01 smithi116 ceph-mon[35411]: pgmap v9873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:55:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:03 smithi114 ceph-mon[30820]: pgmap v9874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:55:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:03 smithi116 ceph-mon[35411]: pgmap v9874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:55:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:05 smithi114 ceph-mon[30820]: pgmap v9875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:05 smithi116 ceph-mon[35411]: pgmap v9875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:07 smithi114 ceph-mon[30820]: pgmap v9876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:07 smithi116 ceph-mon[35411]: pgmap v9876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:09 smithi116 ceph-mon[35411]: pgmap v9877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:09 smithi114 ceph-mon[30820]: pgmap v9877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:11 smithi116 ceph-mon[35411]: pgmap v9878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:11 smithi114 ceph-mon[30820]: pgmap v9878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:13 smithi116 ceph-mon[35411]: pgmap v9879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:13 smithi114 ceph-mon[30820]: pgmap v9879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:15 smithi114 ceph-mon[30820]: pgmap v9880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:15 smithi116 ceph-mon[35411]: pgmap v9880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:17 smithi114 ceph-mon[30820]: pgmap v9881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:17 smithi116 ceph-mon[35411]: pgmap v9881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:19 smithi114 ceph-mon[30820]: pgmap v9882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:19 smithi116 ceph-mon[35411]: pgmap v9882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:21 smithi114 ceph-mon[30820]: pgmap v9883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:21 smithi116 ceph-mon[35411]: pgmap v9883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:23 smithi114 ceph-mon[30820]: pgmap v9884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:23 smithi116 ceph-mon[35411]: pgmap v9884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:24 smithi114 ceph-mon[30820]: pgmap v9885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:24 smithi116 ceph-mon[35411]: pgmap v9885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:26 smithi114 ceph-mon[30820]: pgmap v9886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:26 smithi116 ceph-mon[35411]: pgmap v9886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:28 smithi114 ceph-mon[30820]: pgmap v9887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:28 smithi116 ceph-mon[35411]: pgmap v9887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:31.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:30 smithi114 ceph-mon[30820]: pgmap v9888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:30 smithi116 ceph-mon[35411]: pgmap v9888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:32 smithi114 ceph-mon[30820]: pgmap v9889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:32 smithi116 ceph-mon[35411]: pgmap v9889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:34 smithi114 ceph-mon[30820]: pgmap v9890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:34 smithi116 ceph-mon[35411]: pgmap v9890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:36 smithi114 ceph-mon[30820]: pgmap v9891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:36 smithi116 ceph-mon[35411]: pgmap v9891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:38 smithi114 ceph-mon[30820]: pgmap v9892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:38 smithi116 ceph-mon[35411]: pgmap v9892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:40 smithi114 ceph-mon[30820]: pgmap v9893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:40 smithi116 ceph-mon[35411]: pgmap v9893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:42 smithi114 ceph-mon[30820]: pgmap v9894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:42 smithi116 ceph-mon[35411]: pgmap v9894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:44 smithi114 ceph-mon[30820]: pgmap v9895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:44 smithi116 ceph-mon[35411]: pgmap v9895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:46 smithi114 ceph-mon[30820]: pgmap v9896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:47.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:46 smithi116 ceph-mon[35411]: pgmap v9896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:55:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:48 smithi114 ceph-mon[30820]: pgmap v9897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:48 smithi116 ceph-mon[35411]: pgmap v9897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:50 smithi116 ceph-mon[35411]: pgmap v9898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:50 smithi114 ceph-mon[30820]: pgmap v9898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:55:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:55:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:55:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:52 smithi114 ceph-mon[30820]: pgmap v9899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:52 smithi116 ceph-mon[35411]: pgmap v9899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:55:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:54 smithi114 ceph-mon[30820]: pgmap v9900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:55.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:55:55.093+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:55:55.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:55:55.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:55:55.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:55:55.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:55:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:54 smithi116 ceph-mon[35411]: pgmap v9900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:55:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:55:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:55:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:55:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:55:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:55:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:55:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:55:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:55:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:55:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:55:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:55:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:55:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:55:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-mon[30820]: pgmap v9901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T11:55:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-mon[30820]: pgmap v9902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:55:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:55:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:55 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:55:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:55:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:55:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:55:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:55:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:55:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:55:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:55:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:55:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:55:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:55:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:55:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:55 smithi116 ceph-mon[35411]: pgmap v9901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T11:55:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:55 smithi116 ceph-mon[35411]: pgmap v9902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:55:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:55:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:55:58 smithi114 ceph-mon[30820]: pgmap v9903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:55:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:55:58 smithi116 ceph-mon[35411]: pgmap v9903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:56:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:00 smithi114 ceph-mon[30820]: pgmap v9904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:56:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:00 smithi116 ceph-mon[35411]: pgmap v9904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:56:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:02 smithi114 ceph-mon[30820]: pgmap v9905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:56:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:02 smithi116 ceph-mon[35411]: pgmap v9905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:56:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:04 smithi116 ceph-mon[35411]: pgmap v9906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:56:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:04 smithi114 ceph-mon[30820]: pgmap v9906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:56:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:06 smithi116 ceph-mon[35411]: pgmap v9907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:56:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:06 smithi114 ceph-mon[30820]: pgmap v9907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:56:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:08 smithi116 ceph-mon[35411]: pgmap v9908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:08 smithi114 ceph-mon[30820]: pgmap v9908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:10 smithi114 ceph-mon[30820]: pgmap v9909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:10 smithi116 ceph-mon[35411]: pgmap v9909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:12 smithi114 ceph-mon[30820]: pgmap v9910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:12 smithi116 ceph-mon[35411]: pgmap v9910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:14 smithi114 ceph-mon[30820]: pgmap v9911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:14 smithi116 ceph-mon[35411]: pgmap v9911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:16 smithi114 ceph-mon[30820]: pgmap v9912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:56:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:16 smithi116 ceph-mon[35411]: pgmap v9912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:56:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:18 smithi114 ceph-mon[30820]: pgmap v9913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:18 smithi116 ceph-mon[35411]: pgmap v9913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:20 smithi114 ceph-mon[30820]: pgmap v9914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:20 smithi116 ceph-mon[35411]: pgmap v9914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:22 smithi114 ceph-mon[30820]: pgmap v9915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:22 smithi116 ceph-mon[35411]: pgmap v9915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:24 smithi114 ceph-mon[30820]: pgmap v9916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:24 smithi116 ceph-mon[35411]: pgmap v9916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:26 smithi114 ceph-mon[30820]: pgmap v9917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:56:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:26 smithi116 ceph-mon[35411]: pgmap v9917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:56:28.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:28 smithi114 ceph-mon[30820]: pgmap v9918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:28 smithi116 ceph-mon[35411]: pgmap v9918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:30 smithi114 ceph-mon[30820]: pgmap v9919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:30 smithi116 ceph-mon[35411]: pgmap v9919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:32 smithi114 ceph-mon[30820]: pgmap v9920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:32 smithi116 ceph-mon[35411]: pgmap v9920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:34 smithi114 ceph-mon[30820]: pgmap v9921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:56:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:34 smithi116 ceph-mon[35411]: pgmap v9921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:56:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:36 smithi114 ceph-mon[30820]: pgmap v9922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:56:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:36 smithi116 ceph-mon[35411]: pgmap v9922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T11:56:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:38 smithi114 ceph-mon[30820]: pgmap v9923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:38 smithi116 ceph-mon[35411]: pgmap v9923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:40 smithi114 ceph-mon[30820]: pgmap v9924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:56:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:40 smithi116 ceph-mon[35411]: pgmap v9924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T11:56:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:42 smithi114 ceph-mon[30820]: pgmap v9925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:42 smithi116 ceph-mon[35411]: pgmap v9925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:44 smithi114 ceph-mon[30820]: pgmap v9926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:44 smithi116 ceph-mon[35411]: pgmap v9926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:46 smithi114 ceph-mon[30820]: pgmap v9927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:56:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:46 smithi116 ceph-mon[35411]: pgmap v9927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:56:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:48 smithi114 ceph-mon[30820]: pgmap v9928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:48 smithi116 ceph-mon[35411]: pgmap v9928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:50 smithi116 ceph-mon[35411]: pgmap v9929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:50 smithi114 ceph-mon[30820]: pgmap v9929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:52 smithi116 ceph-mon[35411]: pgmap v9930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:52 smithi114 ceph-mon[30820]: pgmap v9930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:56:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:54 smithi116 ceph-mon[35411]: pgmap v9931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:54 smithi114 ceph-mon[30820]: pgmap v9931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:56:55.899 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:56:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:56:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:56 smithi114 ceph-mon[30820]: pgmap v9932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:56:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:56 smithi116 ceph-mon[35411]: pgmap v9932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:56:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:57 smithi114 ceph-mon[30820]: pgmap v9933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:58.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:57 smithi116 ceph-mon[35411]: pgmap v9933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:56:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:58.738 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:56:58.452+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:56:58.738 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:56:58.738 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:56:58.738 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:56:58.738 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:56:58.739 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:56:58.739 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:56:58.739 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:56:58.739 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:56:58.739 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:56:58.740 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:56:58.740 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:56:58.740 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:56:58.740 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:56:58.740 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:56:58.741 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:56:58.741 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:56:58.741 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:56:58.741 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:56:58.741 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:56:58.741 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:56:58.742 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:56:58.742 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:56:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:56:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:56:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:56:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:56:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:56:59.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-mon[30820]: pgmap v9934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-mon[30820]: pgmap v9935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:56:59.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:56:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:56:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:58 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:56:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:56:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:56:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:56:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:56:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:56:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:56:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:56:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:56:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:56:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:56:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:56:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:56:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:56:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:56:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:56:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:56:59.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:58 smithi116 ceph-mon[35411]: pgmap v9934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:58 smithi116 ceph-mon[35411]: pgmap v9935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:56:59.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:56:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:57:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:01 smithi114 ceph-mon[30820]: pgmap v9936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:57:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:01 smithi116 ceph-mon[35411]: pgmap v9936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:57:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:03 smithi114 ceph-mon[30820]: pgmap v9937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:57:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:03 smithi116 ceph-mon[35411]: pgmap v9937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:57:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:05 smithi114 ceph-mon[30820]: pgmap v9938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:57:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:05 smithi116 ceph-mon[35411]: pgmap v9938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:57:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:07 smithi114 ceph-mon[30820]: pgmap v9939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:57:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:07 smithi116 ceph-mon[35411]: pgmap v9939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:57:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:09 smithi114 ceph-mon[30820]: pgmap v9940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:57:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:09 smithi116 ceph-mon[35411]: pgmap v9940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:57:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:11 smithi114 ceph-mon[30820]: pgmap v9941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:11 smithi116 ceph-mon[35411]: pgmap v9941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:13 smithi114 ceph-mon[30820]: pgmap v9942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:13 smithi116 ceph-mon[35411]: pgmap v9942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:15 smithi114 ceph-mon[30820]: pgmap v9943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:15 smithi116 ceph-mon[35411]: pgmap v9943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:17 smithi114 ceph-mon[30820]: pgmap v9944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:17 smithi116 ceph-mon[35411]: pgmap v9944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:19 smithi114 ceph-mon[30820]: pgmap v9945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:19 smithi116 ceph-mon[35411]: pgmap v9945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:21 smithi114 ceph-mon[30820]: pgmap v9946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:21 smithi116 ceph-mon[35411]: pgmap v9946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:23 smithi114 ceph-mon[30820]: pgmap v9947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:23 smithi116 ceph-mon[35411]: pgmap v9947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:25 smithi114 ceph-mon[30820]: pgmap v9948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:25 smithi116 ceph-mon[35411]: pgmap v9948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:27 smithi114 ceph-mon[30820]: pgmap v9949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:27 smithi116 ceph-mon[35411]: pgmap v9949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:29 smithi114 ceph-mon[30820]: pgmap v9950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:29 smithi116 ceph-mon[35411]: pgmap v9950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:31 smithi114 ceph-mon[30820]: pgmap v9951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:31 smithi116 ceph-mon[35411]: pgmap v9951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:33 smithi114 ceph-mon[30820]: pgmap v9952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:33 smithi116 ceph-mon[35411]: pgmap v9952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:35 smithi114 ceph-mon[30820]: pgmap v9953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:35 smithi116 ceph-mon[35411]: pgmap v9953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:37 smithi114 ceph-mon[30820]: pgmap v9954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:37 smithi116 ceph-mon[35411]: pgmap v9954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:39 smithi114 ceph-mon[30820]: pgmap v9955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:39 smithi116 ceph-mon[35411]: pgmap v9955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:41 smithi114 ceph-mon[30820]: pgmap v9956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:41 smithi116 ceph-mon[35411]: pgmap v9956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:43 smithi114 ceph-mon[30820]: pgmap v9957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:43 smithi116 ceph-mon[35411]: pgmap v9957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:45 smithi114 ceph-mon[30820]: pgmap v9958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:45 smithi116 ceph-mon[35411]: pgmap v9958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:47 smithi114 ceph-mon[30820]: pgmap v9959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:47 smithi116 ceph-mon[35411]: pgmap v9959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:49 smithi114 ceph-mon[30820]: pgmap v9960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:49 smithi116 ceph-mon[35411]: pgmap v9960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:51 smithi114 ceph-mon[30820]: pgmap v9961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:51 smithi116 ceph-mon[35411]: pgmap v9961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:57:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:53 smithi114 ceph-mon[30820]: pgmap v9962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:53 smithi116 ceph-mon[35411]: pgmap v9962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:55 smithi114 ceph-mon[30820]: pgmap v9963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:55 smithi116 ceph-mon[35411]: pgmap v9963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:57:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:57 smithi114 ceph-mon[30820]: pgmap v9964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:57 smithi116 ceph-mon[35411]: pgmap v9964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:57:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:57:58.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:57:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:59 smithi114 ceph-mon[30820]: pgmap v9965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:57:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:57:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:57:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:57:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:57:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:59 smithi116 ceph-mon[35411]: pgmap v9965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:57:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:57:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:57:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:57:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:57:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:58:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-mon[30820]: pgmap v9966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:01.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:58:01.748+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:58:01.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:58:01.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:58:01.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:58:01.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:58:01.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:58:01.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:58:01.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:58:01.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:58:01.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:58:01.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:58:01.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:58:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:58:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:01 smithi116 ceph-mon[35411]: pgmap v9966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:58:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:58:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:58:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:58:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:58:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:58:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:58:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:58:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:58:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:58:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:58:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:58:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:58:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:58:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:03 smithi114 ceph-mon[30820]: pgmap v9967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:58:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:03 smithi114 ceph-mon[30820]: pgmap v9968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:58:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:58:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:58:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:58:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:58:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:58:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:58:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:58:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:58:03.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:58:03.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:58:03.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:03 smithi116 ceph-mon[35411]: pgmap v9967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T11:58:03.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:03 smithi116 ceph-mon[35411]: pgmap v9968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:58:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:05 smithi114 ceph-mon[30820]: pgmap v9969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:58:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:05 smithi116 ceph-mon[35411]: pgmap v9969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:58:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:07 smithi114 ceph-mon[30820]: pgmap v9970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:58:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:07 smithi116 ceph-mon[35411]: pgmap v9970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:58:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:09 smithi114 ceph-mon[30820]: pgmap v9971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:58:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:09 smithi116 ceph-mon[35411]: pgmap v9971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:58:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:11 smithi114 ceph-mon[30820]: pgmap v9972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:58:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:11 smithi116 ceph-mon[35411]: pgmap v9972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:58:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:13 smithi114 ceph-mon[30820]: pgmap v9973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:58:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:13 smithi116 ceph-mon[35411]: pgmap v9973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:58:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:15 smithi114 ceph-mon[30820]: pgmap v9974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:15 smithi116 ceph-mon[35411]: pgmap v9974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:17 smithi114 ceph-mon[30820]: pgmap v9975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:17 smithi116 ceph-mon[35411]: pgmap v9975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:19 smithi116 ceph-mon[35411]: pgmap v9976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:19 smithi114 ceph-mon[30820]: pgmap v9976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:21 smithi116 ceph-mon[35411]: pgmap v9977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:58:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:21 smithi114 ceph-mon[30820]: pgmap v9977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:58:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:23 smithi116 ceph-mon[35411]: pgmap v9978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:58:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:23 smithi114 ceph-mon[30820]: pgmap v9978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:58:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:25 smithi114 ceph-mon[30820]: pgmap v9979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:25 smithi116 ceph-mon[35411]: pgmap v9979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:27 smithi114 ceph-mon[30820]: pgmap v9980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:27 smithi116 ceph-mon[35411]: pgmap v9980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:29.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:29 smithi114 ceph-mon[30820]: pgmap v9981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:29 smithi116 ceph-mon[35411]: pgmap v9981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:31 smithi114 ceph-mon[30820]: pgmap v9982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:58:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:31 smithi116 ceph-mon[35411]: pgmap v9982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:58:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:33 smithi114 ceph-mon[30820]: pgmap v9983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:58:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:33 smithi116 ceph-mon[35411]: pgmap v9983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:58:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:35 smithi114 ceph-mon[30820]: pgmap v9984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:35 smithi116 ceph-mon[35411]: pgmap v9984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:37.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:37 smithi116 ceph-mon[35411]: pgmap v9985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:37 smithi114 ceph-mon[30820]: pgmap v9985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:39 smithi114 ceph-mon[30820]: pgmap v9986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:39 smithi116 ceph-mon[35411]: pgmap v9986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:41 smithi114 ceph-mon[30820]: pgmap v9987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:58:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:41 smithi116 ceph-mon[35411]: pgmap v9987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:58:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:43 smithi114 ceph-mon[30820]: pgmap v9988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:58:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:43 smithi116 ceph-mon[35411]: pgmap v9988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:58:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:45 smithi114 ceph-mon[30820]: pgmap v9989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:45 smithi116 ceph-mon[35411]: pgmap v9989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:47 smithi114 ceph-mon[30820]: pgmap v9990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:47 smithi116 ceph-mon[35411]: pgmap v9990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:49 smithi114 ceph-mon[30820]: pgmap v9991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:49 smithi116 ceph-mon[35411]: pgmap v9991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:51 smithi114 ceph-mon[30820]: pgmap v9992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:58:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:51 smithi116 ceph-mon[35411]: pgmap v9992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:58:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:53 smithi114 ceph-mon[30820]: pgmap v9993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:58:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:53 smithi116 ceph-mon[35411]: pgmap v9993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:58:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:55 smithi114 ceph-mon[30820]: pgmap v9994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:55 smithi116 ceph-mon[35411]: pgmap v9994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:56 smithi114 ceph-mon[30820]: pgmap v9995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:56 smithi116 ceph-mon[35411]: pgmap v9995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:58:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:58:58 smithi114 ceph-mon[30820]: pgmap v9996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:58:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:58:58 smithi116 ceph-mon[35411]: pgmap v9996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:00 smithi114 ceph-mon[30820]: pgmap v9997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:00 smithi116 ceph-mon[35411]: pgmap v9997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:02 smithi114 ceph-mon[30820]: pgmap v9998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:59:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:02 smithi116 ceph-mon[35411]: pgmap v9998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T11:59:05.111 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:04 smithi114 ceph-mon[30820]: pgmap v9999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:04 smithi116 ceph-mon[35411]: pgmap v9999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T11:59:05.109+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:59:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T11:59:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:59:05.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T11:59:05.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:59:05.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:59:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:59:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T11:59:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:59:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:59:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-mon[30820]: pgmap v10000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:59:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-mon[30820]: pgmap v10001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:59:06.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:05 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T11:59:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T11:59:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T11:59:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T11:59:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T11:59:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:05 smithi116 ceph-mon[35411]: pgmap v10000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T11:59:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:05 smithi116 ceph-mon[35411]: pgmap v10001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T11:59:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T11:59:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:08 smithi116 ceph-mon[35411]: pgmap v10002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:59:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:08 smithi114 ceph-mon[30820]: pgmap v10002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:59:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:10 smithi116 ceph-mon[35411]: pgmap v10003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:59:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:10 smithi114 ceph-mon[30820]: pgmap v10003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T11:59:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:12 smithi116 ceph-mon[35411]: pgmap v10004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:59:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:12 smithi114 ceph-mon[30820]: pgmap v10004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:59:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:14 smithi114 ceph-mon[30820]: pgmap v10005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:59:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:14 smithi116 ceph-mon[35411]: pgmap v10005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T11:59:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:16 smithi114 ceph-mon[30820]: pgmap v10006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:59:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:16 smithi116 ceph-mon[35411]: pgmap v10006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T11:59:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:18 smithi114 ceph-mon[30820]: pgmap v10007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:18.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:18 smithi116 ceph-mon[35411]: pgmap v10007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:20 smithi114 ceph-mon[30820]: pgmap v10008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:20 smithi116 ceph-mon[35411]: pgmap v10008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:22 smithi114 ceph-mon[30820]: pgmap v10009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:59:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:22 smithi116 ceph-mon[35411]: pgmap v10009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:59:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:24 smithi114 ceph-mon[30820]: pgmap v10010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:24.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:24 smithi116 ceph-mon[35411]: pgmap v10010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:26 smithi114 ceph-mon[30820]: pgmap v10011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:26 smithi116 ceph-mon[35411]: pgmap v10011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:28 smithi114 ceph-mon[30820]: pgmap v10012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:28 smithi116 ceph-mon[35411]: pgmap v10012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:30.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:30 smithi114 ceph-mon[30820]: pgmap v10013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:30 smithi116 ceph-mon[35411]: pgmap v10013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:32 smithi114 ceph-mon[30820]: pgmap v10014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:59:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:32 smithi116 ceph-mon[35411]: pgmap v10014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:59:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:34 smithi114 ceph-mon[30820]: pgmap v10015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:34 smithi116 ceph-mon[35411]: pgmap v10015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:36 smithi114 ceph-mon[30820]: pgmap v10016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:36 smithi116 ceph-mon[35411]: pgmap v10016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:38.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:38 smithi116 ceph-mon[35411]: pgmap v10017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:38 smithi114 ceph-mon[30820]: pgmap v10017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:40 smithi114 ceph-mon[30820]: pgmap v10018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:40 smithi116 ceph-mon[35411]: pgmap v10018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:42 smithi114 ceph-mon[30820]: pgmap v10019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:59:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:42 smithi116 ceph-mon[35411]: pgmap v10019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:59:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:44 smithi114 ceph-mon[30820]: pgmap v10020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:44 smithi116 ceph-mon[35411]: pgmap v10020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:46 smithi114 ceph-mon[30820]: pgmap v10021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:46 smithi116 ceph-mon[35411]: pgmap v10021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:48 smithi114 ceph-mon[30820]: pgmap v10022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:48 smithi116 ceph-mon[35411]: pgmap v10022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:50 smithi114 ceph-mon[30820]: pgmap v10023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:50 smithi116 ceph-mon[35411]: pgmap v10023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:52 smithi114 ceph-mon[30820]: pgmap v10024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:59:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:52 smithi116 ceph-mon[35411]: pgmap v10024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T11:59:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:54 smithi114 ceph-mon[30820]: pgmap v10025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:54 smithi116 ceph-mon[35411]: pgmap v10025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T11:59:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:56 smithi114 ceph-mon[30820]: pgmap v10026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:56 smithi116 ceph-mon[35411]: pgmap v10026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T11:59:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 11:59:58 smithi114 ceph-mon[30820]: pgmap v10027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T11:59:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 11:59:58 smithi116 ceph-mon[35411]: pgmap v10027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:00 smithi114 ceph-mon[30820]: pgmap v10028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:00:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:00 smithi116 ceph-mon[35411]: pgmap v10028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:00:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:02 smithi114 ceph-mon[30820]: pgmap v10029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:02 smithi116 ceph-mon[35411]: pgmap v10029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:04 smithi116 ceph-mon[35411]: pgmap v10030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:04 smithi114 ceph-mon[30820]: pgmap v10030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:00:05.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:00:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:06 smithi114 ceph-mon[30820]: pgmap v10031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:06 smithi116 ceph-mon[35411]: pgmap v10031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:08.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:08 smithi116 ceph-mon[35411]: pgmap v10032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-mon[30820]: pgmap v10032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:00:08.468+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:00:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:00:08.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:00:08.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:00:08.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:00:08.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:00:08.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:00:08.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:00:08.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:00:08.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:00:08.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:00:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:00:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:09 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:00:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:00:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:00:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:00:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:00:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:00:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:00:09.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:00:09.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:00:09.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:00:09.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:00:09.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:00:09.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:00:09.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:09 smithi114 ceph-mon[30820]: pgmap v10033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:00:09.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:09 smithi114 ceph-mon[30820]: pgmap v10034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:09.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:00:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:09 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:00:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:00:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:00:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:00:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:00:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:00:09.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:00:09.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:00:09.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:00:09.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:00:09.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:00:09.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:09 smithi116 ceph-mon[35411]: pgmap v10033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:00:09.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:09 smithi116 ceph-mon[35411]: pgmap v10034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:09.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:00:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:11 smithi114 ceph-mon[30820]: pgmap v10035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:11 smithi116 ceph-mon[35411]: pgmap v10035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:13 smithi114 ceph-mon[30820]: pgmap v10036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:13 smithi116 ceph-mon[35411]: pgmap v10036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:15 smithi114 ceph-mon[30820]: pgmap v10037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:15 smithi116 ceph-mon[35411]: pgmap v10037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:17 smithi114 ceph-mon[30820]: pgmap v10038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:17 smithi116 ceph-mon[35411]: pgmap v10038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:19 smithi114 ceph-mon[30820]: pgmap v10039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:00:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:19 smithi116 ceph-mon[35411]: pgmap v10039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:00:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:21 smithi114 ceph-mon[30820]: pgmap v10040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:21 smithi116 ceph-mon[35411]: pgmap v10040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:23 smithi114 ceph-mon[30820]: pgmap v10041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:23 smithi116 ceph-mon[35411]: pgmap v10041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:25 smithi114 ceph-mon[30820]: pgmap v10042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:25 smithi116 ceph-mon[35411]: pgmap v10042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:27 smithi114 ceph-mon[30820]: pgmap v10043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:27 smithi116 ceph-mon[35411]: pgmap v10043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:29 smithi114 ceph-mon[30820]: pgmap v10044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:29 smithi116 ceph-mon[35411]: pgmap v10044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:31 smithi114 ceph-mon[30820]: pgmap v10045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:31 smithi116 ceph-mon[35411]: pgmap v10045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:33.845 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:33 smithi114 ceph-mon[30820]: pgmap v10046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:33 smithi116 ceph-mon[35411]: pgmap v10046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:35 smithi114 ceph-mon[30820]: pgmap v10047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:35.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:35 smithi116 ceph-mon[35411]: pgmap v10047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:37 smithi114 ceph-mon[30820]: pgmap v10048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:37 smithi116 ceph-mon[35411]: pgmap v10048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:39 smithi114 ceph-mon[30820]: pgmap v10049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:39 smithi116 ceph-mon[35411]: pgmap v10049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:41 smithi114 ceph-mon[30820]: pgmap v10050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:41 smithi116 ceph-mon[35411]: pgmap v10050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:43 smithi114 ceph-mon[30820]: pgmap v10051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:43 smithi116 ceph-mon[35411]: pgmap v10051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:45 smithi114 ceph-mon[30820]: pgmap v10052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:45 smithi116 ceph-mon[35411]: pgmap v10052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:47 smithi114 ceph-mon[30820]: pgmap v10053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:47 smithi116 ceph-mon[35411]: pgmap v10053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:49 smithi114 ceph-mon[30820]: pgmap v10054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:49 smithi116 ceph-mon[35411]: pgmap v10054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:51 smithi114 ceph-mon[30820]: pgmap v10055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:51 smithi116 ceph-mon[35411]: pgmap v10055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:00:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:53 smithi114 ceph-mon[30820]: pgmap v10056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:53 smithi116 ceph-mon[35411]: pgmap v10056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:55 smithi114 ceph-mon[30820]: pgmap v10057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:55 smithi116 ceph-mon[35411]: pgmap v10057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:00:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:57 smithi114 ceph-mon[30820]: pgmap v10058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:57 smithi116 ceph-mon[35411]: pgmap v10058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:00:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:00:59 smithi114 ceph-mon[30820]: pgmap v10059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:00:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:00:59 smithi116 ceph-mon[35411]: pgmap v10059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:01.849 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:01 smithi114 ceph-mon[30820]: pgmap v10060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:01 smithi116 ceph-mon[35411]: pgmap v10060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:03 smithi114 ceph-mon[30820]: pgmap v10061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:03 smithi116 ceph-mon[35411]: pgmap v10061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:05 smithi114 ceph-mon[30820]: pgmap v10062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:01:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:05 smithi116 ceph-mon[35411]: pgmap v10062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:01:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:07 smithi114 ceph-mon[30820]: pgmap v10063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:01:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:07 smithi116 ceph-mon[35411]: pgmap v10063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:01:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:09 smithi114 ceph-mon[30820]: pgmap v10064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:01:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:09 smithi116 ceph-mon[35411]: pgmap v10064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:01:11.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-mon[30820]: pgmap v10065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:11 smithi116 ceph-mon[35411]: pgmap v10065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:01:11.829+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:01:12.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:01:12.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:01:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:01:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:01:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:01:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:01:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:01:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:01:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:01:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:13 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:01:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:01:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:01:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:01:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:01:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:01:13.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:01:13.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:01:13.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:01:13.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:01:13.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:13 smithi114 ceph-mon[30820]: pgmap v10066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:01:13.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:13 smithi114 ceph-mon[30820]: pgmap v10067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:13 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:01:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:01:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:01:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:13 smithi116 ceph-mon[35411]: pgmap v10066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:01:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:13 smithi116 ceph-mon[35411]: pgmap v10067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:01:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:15 smithi114 ceph-mon[30820]: pgmap v10068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T12:01:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:15 smithi116 ceph-mon[35411]: pgmap v10068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T12:01:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:17 smithi114 ceph-mon[30820]: pgmap v10069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:01:17.947 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:17 smithi116 ceph-mon[35411]: pgmap v10069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:01:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:19 smithi114 ceph-mon[30820]: pgmap v10070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:01:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:19 smithi116 ceph-mon[35411]: pgmap v10070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:01:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:21 smithi114 ceph-mon[30820]: pgmap v10071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:01:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:21 smithi116 ceph-mon[35411]: pgmap v10071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:01:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:23 smithi114 ceph-mon[30820]: pgmap v10072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:01:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:23 smithi116 ceph-mon[35411]: pgmap v10072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:01:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:25 smithi114 ceph-mon[30820]: pgmap v10073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:25 smithi116 ceph-mon[35411]: pgmap v10073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:27 smithi114 ceph-mon[30820]: pgmap v10074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:27 smithi116 ceph-mon[35411]: pgmap v10074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:29 smithi114 ceph-mon[30820]: pgmap v10075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:29 smithi116 ceph-mon[35411]: pgmap v10075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:31 smithi114 ceph-mon[30820]: pgmap v10076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:01:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:31 smithi116 ceph-mon[35411]: pgmap v10076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:01:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:33 smithi114 ceph-mon[30820]: pgmap v10077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:01:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:33 smithi116 ceph-mon[35411]: pgmap v10077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:01:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:35 smithi114 ceph-mon[30820]: pgmap v10078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:35 smithi116 ceph-mon[35411]: pgmap v10078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:37 smithi114 ceph-mon[30820]: pgmap v10079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:37 smithi116 ceph-mon[35411]: pgmap v10079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:39 smithi114 ceph-mon[30820]: pgmap v10080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:39 smithi116 ceph-mon[35411]: pgmap v10080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:41 smithi114 ceph-mon[30820]: pgmap v10081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:01:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:41 smithi116 ceph-mon[35411]: pgmap v10081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:01:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:43 smithi114 ceph-mon[30820]: pgmap v10082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:01:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:43 smithi116 ceph-mon[35411]: pgmap v10082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:01:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:45 smithi114 ceph-mon[30820]: pgmap v10083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:45 smithi116 ceph-mon[35411]: pgmap v10083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:47 smithi114 ceph-mon[30820]: pgmap v10084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:47.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:47 smithi116 ceph-mon[35411]: pgmap v10084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:49 smithi114 ceph-mon[30820]: pgmap v10085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:49 smithi116 ceph-mon[35411]: pgmap v10085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:51 smithi114 ceph-mon[30820]: pgmap v10086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:01:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:51 smithi116 ceph-mon[35411]: pgmap v10086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:01:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:53 smithi114 ceph-mon[30820]: pgmap v10087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:01:53.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:53 smithi116 ceph-mon[35411]: pgmap v10087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:01:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:55 smithi114 ceph-mon[30820]: pgmap v10088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:55 smithi116 ceph-mon[35411]: pgmap v10088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:57 smithi114 ceph-mon[30820]: pgmap v10089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:57 smithi116 ceph-mon[35411]: pgmap v10089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:01:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:01:59 smithi114 ceph-mon[30820]: pgmap v10090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:01:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:01:59 smithi116 ceph-mon[35411]: pgmap v10090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:01 smithi114 ceph-mon[30820]: pgmap v10091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:01 smithi116 ceph-mon[35411]: pgmap v10091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:03 smithi114 ceph-mon[30820]: pgmap v10092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:03 smithi116 ceph-mon[35411]: pgmap v10092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:05 smithi116 ceph-mon[35411]: pgmap v10093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:05 smithi114 ceph-mon[30820]: pgmap v10093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:07 smithi116 ceph-mon[35411]: pgmap v10094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:02:08.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:07 smithi114 ceph-mon[30820]: pgmap v10094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:02:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:09 smithi116 ceph-mon[35411]: pgmap v10095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:09 smithi114 ceph-mon[30820]: pgmap v10095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:12.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:11 smithi114 ceph-mon[30820]: pgmap v10096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:11 smithi116 ceph-mon[35411]: pgmap v10096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:13.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:02:13.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:02:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:13 smithi114 ceph-mon[30820]: pgmap v10097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:13 smithi116 ceph-mon[35411]: pgmap v10097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:15.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:02:15.188+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:02:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:02:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:02:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-mon[30820]: pgmap v10098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:02:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:02:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:02:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:15 smithi116 ceph-mon[35411]: pgmap v10098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:02:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:02:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:02:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:16 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:02:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:02:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:02:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:02:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:02:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:02:17.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:02:17.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:02:17.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:02:17.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:02:17.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:16 smithi114 ceph-mon[30820]: pgmap v10099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:02:17.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:16 smithi114 ceph-mon[30820]: pgmap v10100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:16 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:02:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:02:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:02:17.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:16 smithi116 ceph-mon[35411]: pgmap v10099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:02:17.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:16 smithi116 ceph-mon[35411]: pgmap v10100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:02:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:18 smithi114 ceph-mon[30820]: pgmap v10101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:02:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:18 smithi116 ceph-mon[35411]: pgmap v10101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:02:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:20 smithi114 ceph-mon[30820]: pgmap v10102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:02:21.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:20 smithi116 ceph-mon[35411]: pgmap v10102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:02:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:22 smithi114 ceph-mon[30820]: pgmap v10103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:02:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:22 smithi116 ceph-mon[35411]: pgmap v10103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:02:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:24 smithi114 ceph-mon[30820]: pgmap v10104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:02:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:24 smithi116 ceph-mon[35411]: pgmap v10104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:02:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:26 smithi114 ceph-mon[30820]: pgmap v10105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:02:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:26 smithi116 ceph-mon[35411]: pgmap v10105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:02:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:28 smithi114 ceph-mon[30820]: pgmap v10106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:29.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:28 smithi116 ceph-mon[35411]: pgmap v10106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:30 smithi114 ceph-mon[30820]: pgmap v10107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:30 smithi116 ceph-mon[35411]: pgmap v10107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:32 smithi114 ceph-mon[30820]: pgmap v10108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:32 smithi116 ceph-mon[35411]: pgmap v10108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:34 smithi114 ceph-mon[30820]: pgmap v10109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:34 smithi116 ceph-mon[35411]: pgmap v10109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:36 smithi114 ceph-mon[30820]: pgmap v10110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:36 smithi116 ceph-mon[35411]: pgmap v10110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:38 smithi114 ceph-mon[30820]: pgmap v10111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:38 smithi116 ceph-mon[35411]: pgmap v10111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:40 smithi114 ceph-mon[30820]: pgmap v10112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:40 smithi116 ceph-mon[35411]: pgmap v10112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:42 smithi114 ceph-mon[30820]: pgmap v10113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:42 smithi116 ceph-mon[35411]: pgmap v10113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:44 smithi114 ceph-mon[30820]: pgmap v10114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:44 smithi116 ceph-mon[35411]: pgmap v10114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:46 smithi114 ceph-mon[30820]: pgmap v10115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:46 smithi116 ceph-mon[35411]: pgmap v10115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:48 smithi114 ceph-mon[30820]: pgmap v10116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:48 smithi116 ceph-mon[35411]: pgmap v10116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:50 smithi114 ceph-mon[30820]: pgmap v10117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:50 smithi116 ceph-mon[35411]: pgmap v10117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:02:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:52 smithi114 ceph-mon[30820]: pgmap v10118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:52 smithi116 ceph-mon[35411]: pgmap v10118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:54 smithi114 ceph-mon[30820]: pgmap v10119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:54 smithi116 ceph-mon[35411]: pgmap v10119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:02:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:56 smithi114 ceph-mon[30820]: pgmap v10120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:02:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:56 smithi116 ceph-mon[35411]: pgmap v10120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:02:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:02:58 smithi114 ceph-mon[30820]: pgmap v10121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:02:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:02:58 smithi116 ceph-mon[35411]: pgmap v10121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:00 smithi114 ceph-mon[30820]: pgmap v10122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:00 smithi116 ceph-mon[35411]: pgmap v10122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:02 smithi114 ceph-mon[30820]: pgmap v10123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:03.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:02 smithi116 ceph-mon[35411]: pgmap v10123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:03 smithi114 ceph-mon[30820]: pgmap v10124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:04.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:03 smithi116 ceph-mon[35411]: pgmap v10124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:06 smithi114 ceph-mon[30820]: pgmap v10125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:03:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:06 smithi116 ceph-mon[35411]: pgmap v10125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:03:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:08 smithi114 ceph-mon[30820]: pgmap v10126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:03:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:08 smithi116 ceph-mon[35411]: pgmap v10126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:03:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:10 smithi114 ceph-mon[30820]: pgmap v10127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:10 smithi116 ceph-mon[35411]: pgmap v10127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:12 smithi114 ceph-mon[30820]: pgmap v10128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:12 smithi116 ceph-mon[35411]: pgmap v10128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:14 smithi114 ceph-mon[30820]: pgmap v10129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:14 smithi116 ceph-mon[35411]: pgmap v10129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:16 smithi114 ceph-mon[30820]: pgmap v10130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:03:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:03:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:16 smithi116 ceph-mon[35411]: pgmap v10130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:03:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:03:18.551 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-mon[30820]: pgmap v10131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:03:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:18 smithi116 ceph-mon[35411]: pgmap v10131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:03:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:03:18.549+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:03:18.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:03:18.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:03:18.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:03:18.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:03:18.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:03:18.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:03:18.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:03:18.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:03:18.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:03:18.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:03:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:03:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:03:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:03:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:03:19.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:03:19.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:03:19.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:19 smithi114 ceph-mon[30820]: pgmap v10132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:03:19.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:19 smithi114 ceph-mon[30820]: pgmap v10133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:03:19.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:03:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:03:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:03:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:03:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:03:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:03:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:03:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:03:19.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:19 smithi116 ceph-mon[35411]: pgmap v10132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:19 smithi116 ceph-mon[35411]: pgmap v10133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:03:19.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:03:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:21 smithi114 ceph-mon[30820]: pgmap v10134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:03:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:21 smithi116 ceph-mon[35411]: pgmap v10134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:03:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:23 smithi114 ceph-mon[30820]: pgmap v10135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:03:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:23 smithi116 ceph-mon[35411]: pgmap v10135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:03:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:25 smithi114 ceph-mon[30820]: pgmap v10136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:03:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:25 smithi116 ceph-mon[35411]: pgmap v10136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:03:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:27 smithi114 ceph-mon[30820]: pgmap v10137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:03:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:27 smithi116 ceph-mon[35411]: pgmap v10137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:03:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:29 smithi114 ceph-mon[30820]: pgmap v10138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:03:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:29 smithi116 ceph-mon[35411]: pgmap v10138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:03:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:31 smithi114 ceph-mon[30820]: pgmap v10139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:03:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:31 smithi116 ceph-mon[35411]: pgmap v10139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:03:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:33 smithi114 ceph-mon[30820]: pgmap v10140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:33 smithi116 ceph-mon[35411]: pgmap v10140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:35 smithi114 ceph-mon[30820]: pgmap v10141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:35 smithi116 ceph-mon[35411]: pgmap v10141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:37 smithi114 ceph-mon[30820]: pgmap v10142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:37 smithi116 ceph-mon[35411]: pgmap v10142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:39 smithi114 ceph-mon[30820]: pgmap v10143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:39.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:39 smithi116 ceph-mon[35411]: pgmap v10143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:41 smithi114 ceph-mon[30820]: pgmap v10144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:03:41.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:41 smithi116 ceph-mon[35411]: pgmap v10144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:03:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:43 smithi114 ceph-mon[30820]: pgmap v10145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:43 smithi116 ceph-mon[35411]: pgmap v10145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:45 smithi114 ceph-mon[30820]: pgmap v10146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:45 smithi116 ceph-mon[35411]: pgmap v10146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:47 smithi114 ceph-mon[30820]: pgmap v10147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:47 smithi116 ceph-mon[35411]: pgmap v10147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:49 smithi114 ceph-mon[30820]: pgmap v10148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:49.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:49 smithi116 ceph-mon[35411]: pgmap v10148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:51 smithi116 ceph-mon[35411]: pgmap v10149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:03:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:51 smithi114 ceph-mon[30820]: pgmap v10149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:03:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:53 smithi116 ceph-mon[35411]: pgmap v10150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:53 smithi114 ceph-mon[30820]: pgmap v10150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:03:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:55 smithi116 ceph-mon[35411]: pgmap v10151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:55 smithi114 ceph-mon[30820]: pgmap v10151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:03:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:57 smithi114 ceph-mon[30820]: pgmap v10152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:03:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:57 smithi116 ceph-mon[35411]: pgmap v10152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:03:59 smithi114 ceph-mon[30820]: pgmap v10153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:03:59 smithi116 ceph-mon[35411]: pgmap v10153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:01 smithi114 ceph-mon[30820]: pgmap v10154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:01 smithi116 ceph-mon[35411]: pgmap v10154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:03.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:03 smithi114 ceph-mon[30820]: pgmap v10155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:03 smithi116 ceph-mon[35411]: pgmap v10155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:05 smithi114 ceph-mon[30820]: pgmap v10156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:04:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:05 smithi116 ceph-mon[35411]: pgmap v10156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:04:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:07 smithi114 ceph-mon[30820]: pgmap v10157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:07 smithi116 ceph-mon[35411]: pgmap v10157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:10.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:09 smithi114 ceph-mon[30820]: pgmap v10158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:09 smithi116 ceph-mon[35411]: pgmap v10158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:11.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:11 smithi116 ceph-mon[35411]: pgmap v10159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:11 smithi114 ceph-mon[30820]: pgmap v10159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:13 smithi114 ceph-mon[30820]: pgmap v10160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:13 smithi116 ceph-mon[35411]: pgmap v10160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:15 smithi114 ceph-mon[30820]: pgmap v10161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:04:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:15 smithi116 ceph-mon[35411]: pgmap v10161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:04:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:17 smithi114 ceph-mon[30820]: pgmap v10162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:17 smithi116 ceph-mon[35411]: pgmap v10162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:19.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:04:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: pgmap v10163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: pgmap v10163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:04:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:20 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:04:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:20 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:04:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:20 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:04:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:20 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:04:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:20 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:04:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:20 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:04:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:20 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:04:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:20 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:04:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-mon[30820]: pgmap v10164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:04:21.908+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:04:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:04:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:04:22.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:04:22.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:04:22.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:04:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:04:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:21 smithi116 ceph-mon[35411]: pgmap v10164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:04:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:04:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:04:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:04:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:04:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:23 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:04:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:04:24.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:23 smithi114 ceph-mon[30820]: pgmap v10165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:04:24.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:23 smithi114 ceph-mon[30820]: pgmap v10166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:23 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:04:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:04:24.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:04:24.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:23 smithi116 ceph-mon[35411]: pgmap v10165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:04:24.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:23 smithi116 ceph-mon[35411]: pgmap v10166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:04:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:25 smithi114 ceph-mon[30820]: pgmap v10167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:04:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:25 smithi116 ceph-mon[35411]: pgmap v10167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:04:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:27 smithi114 ceph-mon[30820]: pgmap v10168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:04:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:27 smithi116 ceph-mon[35411]: pgmap v10168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:04:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:29 smithi114 ceph-mon[30820]: pgmap v10169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:04:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:29 smithi116 ceph-mon[35411]: pgmap v10169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:04:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:31 smithi114 ceph-mon[30820]: pgmap v10170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T12:04:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:31 smithi116 ceph-mon[35411]: pgmap v10170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 4 op/s 2024-09-19T12:04:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:33 smithi114 ceph-mon[30820]: pgmap v10171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:04:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:33 smithi116 ceph-mon[35411]: pgmap v10171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:04:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:35 smithi114 ceph-mon[30820]: pgmap v10172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:04:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:35 smithi116 ceph-mon[35411]: pgmap v10172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:04:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:37 smithi114 ceph-mon[30820]: pgmap v10173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:37 smithi116 ceph-mon[35411]: pgmap v10173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:39 smithi114 ceph-mon[30820]: pgmap v10174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:39 smithi116 ceph-mon[35411]: pgmap v10174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:41 smithi114 ceph-mon[30820]: pgmap v10175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:04:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:41 smithi116 ceph-mon[35411]: pgmap v10175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:04:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:43 smithi114 ceph-mon[30820]: pgmap v10176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:43 smithi116 ceph-mon[35411]: pgmap v10176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:46.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:45 smithi114 ceph-mon[30820]: pgmap v10177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:45 smithi116 ceph-mon[35411]: pgmap v10177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:47 smithi114 ceph-mon[30820]: pgmap v10178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:47 smithi116 ceph-mon[35411]: pgmap v10178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:49 smithi114 ceph-mon[30820]: pgmap v10179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:49 smithi116 ceph-mon[35411]: pgmap v10179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:52.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:51 smithi114 ceph-mon[30820]: pgmap v10180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:04:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:51 smithi116 ceph-mon[35411]: pgmap v10180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:04:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:53 smithi114 ceph-mon[30820]: pgmap v10181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:53 smithi116 ceph-mon[35411]: pgmap v10181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:04:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:55 smithi114 ceph-mon[30820]: pgmap v10182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:55 smithi116 ceph-mon[35411]: pgmap v10182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:58.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:57 smithi114 ceph-mon[30820]: pgmap v10183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:57 smithi116 ceph-mon[35411]: pgmap v10183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:04:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:04:58 smithi114 ceph-mon[30820]: pgmap v10184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:04:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:04:58 smithi116 ceph-mon[35411]: pgmap v10184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:00 smithi114 ceph-mon[30820]: pgmap v10185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:00 smithi116 ceph-mon[35411]: pgmap v10185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:02 smithi114 ceph-mon[30820]: pgmap v10186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:02 smithi116 ceph-mon[35411]: pgmap v10186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:05 smithi114 ceph-mon[30820]: pgmap v10187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:05 smithi116 ceph-mon[35411]: pgmap v10187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:07 smithi114 ceph-mon[30820]: pgmap v10188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:05:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:07 smithi116 ceph-mon[35411]: pgmap v10188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:05:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:09 smithi114 ceph-mon[30820]: pgmap v10189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:09 smithi116 ceph-mon[35411]: pgmap v10189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:11 smithi114 ceph-mon[30820]: pgmap v10190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:11 smithi116 ceph-mon[35411]: pgmap v10190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:13 smithi114 ceph-mon[30820]: pgmap v10191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:13 smithi116 ceph-mon[35411]: pgmap v10191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:15 smithi114 ceph-mon[30820]: pgmap v10192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:15 smithi116 ceph-mon[35411]: pgmap v10192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:17 smithi114 ceph-mon[30820]: pgmap v10193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:05:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:17 smithi116 ceph-mon[35411]: pgmap v10193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:05:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:19 smithi114 ceph-mon[30820]: pgmap v10194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:19 smithi116 ceph-mon[35411]: pgmap v10194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:21 smithi114 ceph-mon[30820]: pgmap v10195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:21 smithi116 ceph-mon[35411]: pgmap v10195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:05:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:05:23.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:23 smithi114 ceph-mon[30820]: pgmap v10196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:23 smithi116 ceph-mon[35411]: pgmap v10196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:05:25.204+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:05:25.535 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:05:25.536 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:05:25.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:05:25.537 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:05:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-mon[30820]: pgmap v10197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:05:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:05:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:05:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:25 smithi116 ceph-mon[35411]: pgmap v10197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:05:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:05:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:05:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:05:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:05:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:05:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:05:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:05:26.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:05:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:05:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:05:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:05:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:05:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:05:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:26 smithi114 ceph-mon[30820]: pgmap v10198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T12:05:26.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:26 smithi114 ceph-mon[30820]: pgmap v10199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:05:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:05:26.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:05:26.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:05:26.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:26 smithi116 ceph-mon[35411]: pgmap v10198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T12:05:26.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:26 smithi116 ceph-mon[35411]: pgmap v10199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:05:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:28 smithi114 ceph-mon[30820]: pgmap v10200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:05:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:28 smithi116 ceph-mon[35411]: pgmap v10200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:05:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:30 smithi114 ceph-mon[30820]: pgmap v10201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:05:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:30 smithi116 ceph-mon[35411]: pgmap v10201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:05:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:32 smithi114 ceph-mon[30820]: pgmap v10202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:05:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:32 smithi116 ceph-mon[35411]: pgmap v10202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:05:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:34 smithi114 ceph-mon[30820]: pgmap v10203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:05:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:34 smithi116 ceph-mon[35411]: pgmap v10203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:05:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:36 smithi114 ceph-mon[30820]: pgmap v10204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:05:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:36 smithi116 ceph-mon[35411]: pgmap v10204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:05:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:38 smithi114 ceph-mon[30820]: pgmap v10205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:05:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:38 smithi116 ceph-mon[35411]: pgmap v10205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:05:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:40 smithi114 ceph-mon[30820]: pgmap v10206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:40 smithi116 ceph-mon[35411]: pgmap v10206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:42 smithi114 ceph-mon[30820]: pgmap v10207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:42 smithi116 ceph-mon[35411]: pgmap v10207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:44 smithi114 ceph-mon[30820]: pgmap v10208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:44 smithi116 ceph-mon[35411]: pgmap v10208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:46 smithi114 ceph-mon[30820]: pgmap v10209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:05:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:46 smithi116 ceph-mon[35411]: pgmap v10209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:05:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:48 smithi114 ceph-mon[30820]: pgmap v10210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:05:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:48 smithi116 ceph-mon[35411]: pgmap v10210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:05:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:50 smithi114 ceph-mon[30820]: pgmap v10211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:50 smithi116 ceph-mon[35411]: pgmap v10211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:05:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:52 smithi114 ceph-mon[30820]: pgmap v10212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:52 smithi116 ceph-mon[35411]: pgmap v10212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:05:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:54 smithi114 ceph-mon[30820]: pgmap v10213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:54 smithi116 ceph-mon[35411]: pgmap v10213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:05:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:56 smithi114 ceph-mon[30820]: pgmap v10214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:05:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:56 smithi116 ceph-mon[35411]: pgmap v10214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:05:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:05:58 smithi114 ceph-mon[30820]: pgmap v10215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:05:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:05:58 smithi116 ceph-mon[35411]: pgmap v10215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:06:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:00 smithi114 ceph-mon[30820]: pgmap v10216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:06:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:00 smithi116 ceph-mon[35411]: pgmap v10216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:06:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:02 smithi114 ceph-mon[30820]: pgmap v10217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:02 smithi116 ceph-mon[35411]: pgmap v10217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:04 smithi114 ceph-mon[30820]: pgmap v10218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:06:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:04 smithi116 ceph-mon[35411]: pgmap v10218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:06:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:06 smithi114 ceph-mon[30820]: pgmap v10219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:06:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:06 smithi116 ceph-mon[35411]: pgmap v10219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:06:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:08 smithi114 ceph-mon[30820]: pgmap v10220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:06:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:08 smithi116 ceph-mon[35411]: pgmap v10220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:06:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:10 smithi114 ceph-mon[30820]: pgmap v10221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:06:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:10 smithi116 ceph-mon[35411]: pgmap v10221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:06:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:12 smithi114 ceph-mon[30820]: pgmap v10222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:12 smithi116 ceph-mon[35411]: pgmap v10222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:14 smithi114 ceph-mon[30820]: pgmap v10223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:06:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:14 smithi116 ceph-mon[35411]: pgmap v10223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:06:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:16 smithi114 ceph-mon[30820]: pgmap v10224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:06:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:16 smithi116 ceph-mon[35411]: pgmap v10224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:06:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:18 smithi114 ceph-mon[30820]: pgmap v10225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:06:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:18 smithi116 ceph-mon[35411]: pgmap v10225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:06:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:20 smithi114 ceph-mon[30820]: pgmap v10226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:06:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:20 smithi116 ceph-mon[35411]: pgmap v10226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:06:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:22 smithi114 ceph-mon[30820]: pgmap v10227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:22 smithi116 ceph-mon[35411]: pgmap v10227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:24 smithi114 ceph-mon[30820]: pgmap v10228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:06:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:24 smithi116 ceph-mon[35411]: pgmap v10228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:06:25.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:06:25.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:06:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:26 smithi114 ceph-mon[30820]: pgmap v10229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:06:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:26 smithi116 ceph-mon[35411]: pgmap v10229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:06:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-mon[30820]: pgmap v10230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:06:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:06:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:06:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:06:28.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:06:28.565+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:06:28.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:06:28.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:06:28 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:06:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:28 smithi116 ceph-mon[35411]: pgmap v10230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:06:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:06:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:06:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:06:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:29 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:06:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:06:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:06:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:06:29.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:06:29.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:29 smithi114 ceph-mon[30820]: pgmap v10231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:06:29.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:29 smithi114 ceph-mon[30820]: pgmap v10232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:06:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:29 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:06:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:06:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:06:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:06:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:06:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:06:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:06:29.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:06:29.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:06:29.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:06:29.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:29 smithi116 ceph-mon[35411]: pgmap v10231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:06:29.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:29 smithi116 ceph-mon[35411]: pgmap v10232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:06:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:31 smithi116 ceph-mon[35411]: pgmap v10233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:06:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:31 smithi114 ceph-mon[30820]: pgmap v10233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:06:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:33 smithi116 ceph-mon[35411]: pgmap v10234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:06:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:33 smithi114 ceph-mon[30820]: pgmap v10234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:06:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:35 smithi116 ceph-mon[35411]: pgmap v10235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:06:35.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:35 smithi114 ceph-mon[30820]: pgmap v10235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:06:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:37 smithi114 ceph-mon[30820]: pgmap v10236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:06:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:37 smithi116 ceph-mon[35411]: pgmap v10236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:06:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:39 smithi114 ceph-mon[30820]: pgmap v10237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:06:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:39 smithi116 ceph-mon[35411]: pgmap v10237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:06:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:41 smithi114 ceph-mon[30820]: pgmap v10238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:06:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:41 smithi116 ceph-mon[35411]: pgmap v10238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:06:43.980 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:43 smithi116 ceph-mon[35411]: pgmap v10239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:06:44.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:43 smithi114 ceph-mon[30820]: pgmap v10239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:06:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:45 smithi114 ceph-mon[30820]: pgmap v10240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:06:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:45 smithi116 ceph-mon[35411]: pgmap v10240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:06:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:47 smithi114 ceph-mon[30820]: pgmap v10241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:47 smithi116 ceph-mon[35411]: pgmap v10241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:49 smithi114 ceph-mon[30820]: pgmap v10242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:06:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:49 smithi116 ceph-mon[35411]: pgmap v10242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:06:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:51 smithi114 ceph-mon[30820]: pgmap v10243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:06:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:51 smithi116 ceph-mon[35411]: pgmap v10243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:06:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:53 smithi114 ceph-mon[30820]: pgmap v10244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:06:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:53 smithi116 ceph-mon[35411]: pgmap v10244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:06:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:55 smithi114 ceph-mon[30820]: pgmap v10245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:06:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:55 smithi116 ceph-mon[35411]: pgmap v10245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:06:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:57 smithi114 ceph-mon[30820]: pgmap v10246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:06:58.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:57 smithi116 ceph-mon[35411]: pgmap v10246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:06:59 smithi114 ceph-mon[30820]: pgmap v10247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:06:59 smithi116 ceph-mon[35411]: pgmap v10247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:01 smithi114 ceph-mon[30820]: pgmap v10248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:01 smithi116 ceph-mon[35411]: pgmap v10248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:03 smithi114 ceph-mon[30820]: pgmap v10249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:03 smithi116 ceph-mon[35411]: pgmap v10249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:05 smithi114 ceph-mon[30820]: pgmap v10250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:07:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:05 smithi116 ceph-mon[35411]: pgmap v10250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:07:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:07 smithi114 ceph-mon[30820]: pgmap v10251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:07 smithi116 ceph-mon[35411]: pgmap v10251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:09 smithi114 ceph-mon[30820]: pgmap v10252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:10.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:09 smithi116 ceph-mon[35411]: pgmap v10252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:11 smithi114 ceph-mon[30820]: pgmap v10253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:11 smithi116 ceph-mon[35411]: pgmap v10253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:13 smithi114 ceph-mon[30820]: pgmap v10254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:13 smithi116 ceph-mon[35411]: pgmap v10254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:15 smithi114 ceph-mon[30820]: pgmap v10255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:07:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:15 smithi116 ceph-mon[35411]: pgmap v10255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:07:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:17 smithi114 ceph-mon[30820]: pgmap v10256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:17 smithi116 ceph-mon[35411]: pgmap v10256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:19 smithi114 ceph-mon[30820]: pgmap v10257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:19 smithi116 ceph-mon[35411]: pgmap v10257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:21 smithi114 ceph-mon[30820]: pgmap v10258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:21 smithi116 ceph-mon[35411]: pgmap v10258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:23 smithi114 ceph-mon[30820]: pgmap v10259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:23 smithi116 ceph-mon[35411]: pgmap v10259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:25 smithi114 ceph-mon[30820]: pgmap v10260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:07:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:25 smithi116 ceph-mon[35411]: pgmap v10260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:07:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:27 smithi114 ceph-mon[30820]: pgmap v10261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:27 smithi116 ceph-mon[35411]: pgmap v10261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:29.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:07:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:07:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:29 smithi114 ceph-mon[30820]: pgmap v10262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:29.928 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:29 smithi116 ceph-mon[35411]: pgmap v10262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-mon[30820]: pgmap v10263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:31 smithi116 ceph-mon[35411]: pgmap v10263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:32.224 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:07:31.925+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:07:32.224 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:07:32.225 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:07:32.226 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:07:32.226 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:07:32.226 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:07:32.226 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:07:32.226 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:07:32.226 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:07:32.226 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:07:32.226 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:07:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:07:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:07:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:07:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:07:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:07:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:07:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:33 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:07:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:07:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:07:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:07:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:07:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:07:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:07:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:07:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:07:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:07:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:07:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:33 smithi114 ceph-mon[30820]: pgmap v10264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:07:33.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:33 smithi114 ceph-mon[30820]: pgmap v10265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:07:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:33 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:07:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:07:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:07:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:07:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:07:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:07:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:07:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:07:33.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:07:33.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:07:33.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:33 smithi116 ceph-mon[35411]: pgmap v10264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:07:33.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:33 smithi116 ceph-mon[35411]: pgmap v10265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:07:35.485 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:35 smithi114 ceph-mon[30820]: pgmap v10266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:07:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:35 smithi116 ceph-mon[35411]: pgmap v10266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:07:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:37 smithi114 ceph-mon[30820]: pgmap v10267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:07:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:37 smithi116 ceph-mon[35411]: pgmap v10267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:07:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:39 smithi114 ceph-mon[30820]: pgmap v10268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:07:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:39 smithi116 ceph-mon[35411]: pgmap v10268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:07:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:41 smithi114 ceph-mon[30820]: pgmap v10269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:07:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:41 smithi116 ceph-mon[35411]: pgmap v10269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:07:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:43 smithi114 ceph-mon[30820]: pgmap v10270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:07:43.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:43 smithi116 ceph-mon[35411]: pgmap v10270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:07:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:45 smithi114 ceph-mon[30820]: pgmap v10271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:45 smithi116 ceph-mon[35411]: pgmap v10271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:47 smithi114 ceph-mon[30820]: pgmap v10272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:47 smithi116 ceph-mon[35411]: pgmap v10272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:49 smithi114 ceph-mon[30820]: pgmap v10273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:49 smithi116 ceph-mon[35411]: pgmap v10273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:51 smithi114 ceph-mon[30820]: pgmap v10274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:07:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:51 smithi116 ceph-mon[35411]: pgmap v10274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:07:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:53 smithi114 ceph-mon[30820]: pgmap v10275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:53 smithi116 ceph-mon[35411]: pgmap v10275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:07:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:55 smithi114 ceph-mon[30820]: pgmap v10276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:55 smithi116 ceph-mon[35411]: pgmap v10276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:57 smithi114 ceph-mon[30820]: pgmap v10277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:57.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:57 smithi116 ceph-mon[35411]: pgmap v10277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:07:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:07:59 smithi114 ceph-mon[30820]: pgmap v10278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:07:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:07:59 smithi116 ceph-mon[35411]: pgmap v10278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:01 smithi114 ceph-mon[30820]: pgmap v10279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:01 smithi116 ceph-mon[35411]: pgmap v10279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:03 smithi114 ceph-mon[30820]: pgmap v10280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:03 smithi116 ceph-mon[35411]: pgmap v10280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:05 smithi114 ceph-mon[30820]: pgmap v10281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:05 smithi116 ceph-mon[35411]: pgmap v10281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:07 smithi114 ceph-mon[30820]: pgmap v10282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:08:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:07 smithi116 ceph-mon[35411]: pgmap v10282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:08:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:09 smithi114 ceph-mon[30820]: pgmap v10283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:09 smithi116 ceph-mon[35411]: pgmap v10283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:11 smithi114 ceph-mon[30820]: pgmap v10284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:11 smithi116 ceph-mon[35411]: pgmap v10284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:13 smithi114 ceph-mon[30820]: pgmap v10285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:13 smithi116 ceph-mon[35411]: pgmap v10285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:15 smithi114 ceph-mon[30820]: pgmap v10286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:15 smithi116 ceph-mon[35411]: pgmap v10286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:17 smithi114 ceph-mon[30820]: pgmap v10287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:08:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:17 smithi116 ceph-mon[35411]: pgmap v10287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:08:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:19 smithi114 ceph-mon[30820]: pgmap v10288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:19 smithi116 ceph-mon[35411]: pgmap v10288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:21 smithi114 ceph-mon[30820]: pgmap v10289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:21 smithi116 ceph-mon[35411]: pgmap v10289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:23 smithi114 ceph-mon[30820]: pgmap v10290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:23 smithi116 ceph-mon[35411]: pgmap v10290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:25 smithi114 ceph-mon[30820]: pgmap v10291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:25.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:25 smithi116 ceph-mon[35411]: pgmap v10291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:27 smithi114 ceph-mon[30820]: pgmap v10292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:08:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:27 smithi116 ceph-mon[35411]: pgmap v10292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:08:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:29 smithi114 ceph-mon[30820]: pgmap v10293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:29 smithi116 ceph-mon[35411]: pgmap v10293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:31 smithi114 ceph-mon[30820]: pgmap v10294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:31 smithi116 ceph-mon[35411]: pgmap v10294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:08:32.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:08:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:33 smithi114 ceph-mon[30820]: pgmap v10295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:33 smithi116 ceph-mon[35411]: pgmap v10295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-mon[30820]: pgmap v10296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:08:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:08:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:35.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:08:35.220+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:08:35.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:08:35.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:08:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:08:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:35 smithi116 ceph-mon[35411]: pgmap v10296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:08:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:08:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:08:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:08:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:08:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:08:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:08:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:08:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:08:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:08:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:08:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:08:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:08:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:36 smithi114 ceph-mon[30820]: pgmap v10297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T12:08:36.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:36 smithi114 ceph-mon[30820]: pgmap v10298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:36 smithi116 ceph-mon[35411]: pgmap v10297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T12:08:36.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:36 smithi116 ceph-mon[35411]: pgmap v10298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:08:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:38 smithi114 ceph-mon[30820]: pgmap v10299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:08:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:38 smithi116 ceph-mon[35411]: pgmap v10299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:08:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:40 smithi114 ceph-mon[30820]: pgmap v10300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:08:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:40 smithi116 ceph-mon[35411]: pgmap v10300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:08:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:42 smithi114 ceph-mon[30820]: pgmap v10301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:08:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:42 smithi116 ceph-mon[35411]: pgmap v10301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:08:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:44 smithi114 ceph-mon[30820]: pgmap v10302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:08:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:44 smithi116 ceph-mon[35411]: pgmap v10302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:08:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:46 smithi114 ceph-mon[30820]: pgmap v10303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:08:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:46 smithi116 ceph-mon[35411]: pgmap v10303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:08:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:48 smithi114 ceph-mon[30820]: pgmap v10304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:08:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:48 smithi116 ceph-mon[35411]: pgmap v10304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:08:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:50 smithi114 ceph-mon[30820]: pgmap v10305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:50 smithi116 ceph-mon[35411]: pgmap v10305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:08:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:52 smithi114 ceph-mon[30820]: pgmap v10306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:52 smithi116 ceph-mon[35411]: pgmap v10306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:08:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:54 smithi114 ceph-mon[30820]: pgmap v10307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:54 smithi116 ceph-mon[35411]: pgmap v10307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:08:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:56 smithi114 ceph-mon[30820]: pgmap v10308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:08:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:56 smithi116 ceph-mon[35411]: pgmap v10308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:08:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:08:58 smithi114 ceph-mon[30820]: pgmap v10309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:08:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:08:58 smithi116 ceph-mon[35411]: pgmap v10309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:00 smithi114 ceph-mon[30820]: pgmap v10310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:00 smithi116 ceph-mon[35411]: pgmap v10310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:02 smithi114 ceph-mon[30820]: pgmap v10311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:02 smithi116 ceph-mon[35411]: pgmap v10311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:04 smithi114 ceph-mon[30820]: pgmap v10312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:04 smithi116 ceph-mon[35411]: pgmap v10312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:06 smithi114 ceph-mon[30820]: pgmap v10313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:09:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:06 smithi116 ceph-mon[35411]: pgmap v10313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:09:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:08 smithi114 ceph-mon[30820]: pgmap v10314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:08 smithi116 ceph-mon[35411]: pgmap v10314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:10 smithi114 ceph-mon[30820]: pgmap v10315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:10 smithi116 ceph-mon[35411]: pgmap v10315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:12 smithi114 ceph-mon[30820]: pgmap v10316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:12 smithi116 ceph-mon[35411]: pgmap v10316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:14 smithi114 ceph-mon[30820]: pgmap v10317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:14 smithi116 ceph-mon[35411]: pgmap v10317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:16 smithi114 ceph-mon[30820]: pgmap v10318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:09:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:16 smithi116 ceph-mon[35411]: pgmap v10318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:09:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:18 smithi114 ceph-mon[30820]: pgmap v10319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:18 smithi116 ceph-mon[35411]: pgmap v10319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:20 smithi114 ceph-mon[30820]: pgmap v10320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:20 smithi116 ceph-mon[35411]: pgmap v10320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:22 smithi114 ceph-mon[30820]: pgmap v10321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:22 smithi116 ceph-mon[35411]: pgmap v10321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:24 smithi114 ceph-mon[30820]: pgmap v10322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:24 smithi116 ceph-mon[35411]: pgmap v10322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:26 smithi114 ceph-mon[30820]: pgmap v10323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:09:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:26 smithi116 ceph-mon[35411]: pgmap v10323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:09:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:28 smithi114 ceph-mon[30820]: pgmap v10324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:28 smithi116 ceph-mon[35411]: pgmap v10324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:30 smithi114 ceph-mon[30820]: pgmap v10325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:30 smithi116 ceph-mon[35411]: pgmap v10325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:32 smithi114 ceph-mon[30820]: pgmap v10326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:32 smithi116 ceph-mon[35411]: pgmap v10326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:34 smithi114 ceph-mon[30820]: pgmap v10327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:34 smithi116 ceph-mon[35411]: pgmap v10327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:09:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:09:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:36 smithi114 ceph-mon[30820]: pgmap v10328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:09:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:36 smithi116 ceph-mon[35411]: pgmap v10328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:09:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-mon[30820]: pgmap v10329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:09:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:09:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:09:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:09:38.516+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:09:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:09:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:09:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:09:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:09:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:09:38.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:09:38.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:09:38.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:09:38.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:09:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:09:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:38 smithi116 ceph-mon[35411]: pgmap v10329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:09:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:09:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:09:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:09:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:09:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:09:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:09:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:09:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:09:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:09:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:09:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:09:39.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:09:39.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:09:39.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:09:39.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:09:39.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:09:39.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:09:39.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:09:39.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:39 smithi114 ceph-mon[30820]: pgmap v10330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:09:39.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:39 smithi114 ceph-mon[30820]: pgmap v10331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:09:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:09:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:09:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:09:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:09:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:09:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:09:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:09:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:09:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:09:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:09:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:09:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:09:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:09:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:09:39.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:39 smithi116 ceph-mon[35411]: pgmap v10330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:09:39.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:39 smithi116 ceph-mon[35411]: pgmap v10331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:09:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:41 smithi114 ceph-mon[30820]: pgmap v10332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:09:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:41 smithi116 ceph-mon[35411]: pgmap v10332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:09:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:43 smithi114 ceph-mon[30820]: pgmap v10333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:09:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:43 smithi116 ceph-mon[35411]: pgmap v10333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:09:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:45 smithi114 ceph-mon[30820]: pgmap v10334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T12:09:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:45 smithi116 ceph-mon[35411]: pgmap v10334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T12:09:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:47 smithi114 ceph-mon[30820]: pgmap v10335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:09:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:47 smithi116 ceph-mon[35411]: pgmap v10335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:09:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:49 smithi114 ceph-mon[30820]: pgmap v10336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:09:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:49 smithi116 ceph-mon[35411]: pgmap v10336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:09:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:51 smithi114 ceph-mon[30820]: pgmap v10337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:09:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:51 smithi116 ceph-mon[35411]: pgmap v10337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:09:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:53 smithi114 ceph-mon[30820]: pgmap v10338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:53 smithi116 ceph-mon[35411]: pgmap v10338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:09:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:55 smithi116 ceph-mon[35411]: pgmap v10339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:55 smithi114 ceph-mon[30820]: pgmap v10339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:09:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:57 smithi116 ceph-mon[35411]: pgmap v10340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:57 smithi114 ceph-mon[30820]: pgmap v10340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:09:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:09:59 smithi116 ceph-mon[35411]: pgmap v10341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:09:59 smithi114 ceph-mon[30820]: pgmap v10341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:10:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:10:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:01 smithi114 ceph-mon[30820]: pgmap v10342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:01 smithi116 ceph-mon[35411]: pgmap v10342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:03 smithi114 ceph-mon[30820]: pgmap v10343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:03 smithi116 ceph-mon[35411]: pgmap v10343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:05 smithi114 ceph-mon[30820]: pgmap v10344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:10:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:05 smithi116 ceph-mon[35411]: pgmap v10344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:10:07.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:07 smithi114 ceph-mon[30820]: pgmap v10345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:10:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:07 smithi116 ceph-mon[35411]: pgmap v10345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:10:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:09 smithi114 ceph-mon[30820]: pgmap v10346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:09 smithi116 ceph-mon[35411]: pgmap v10346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:11 smithi114 ceph-mon[30820]: pgmap v10347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:11 smithi116 ceph-mon[35411]: pgmap v10347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:13 smithi114 ceph-mon[30820]: pgmap v10348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:13 smithi116 ceph-mon[35411]: pgmap v10348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:15.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:15 smithi116 ceph-mon[35411]: pgmap v10349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:10:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:15 smithi114 ceph-mon[30820]: pgmap v10349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:10:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:17 smithi114 ceph-mon[30820]: pgmap v10350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:10:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:17 smithi116 ceph-mon[35411]: pgmap v10350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:10:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:19 smithi114 ceph-mon[30820]: pgmap v10351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:19 smithi116 ceph-mon[35411]: pgmap v10351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:21 smithi114 ceph-mon[30820]: pgmap v10352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:21 smithi116 ceph-mon[35411]: pgmap v10352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:23 smithi114 ceph-mon[30820]: pgmap v10353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:23 smithi116 ceph-mon[35411]: pgmap v10353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:25 smithi114 ceph-mon[30820]: pgmap v10354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:10:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:25 smithi116 ceph-mon[35411]: pgmap v10354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:10:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:27 smithi114 ceph-mon[30820]: pgmap v10355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:10:28.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:27 smithi116 ceph-mon[35411]: pgmap v10355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:10:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:29 smithi114 ceph-mon[30820]: pgmap v10356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:29 smithi116 ceph-mon[35411]: pgmap v10356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:31 smithi114 ceph-mon[30820]: pgmap v10357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:31 smithi116 ceph-mon[35411]: pgmap v10357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:33 smithi114 ceph-mon[30820]: pgmap v10358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:33 smithi116 ceph-mon[35411]: pgmap v10358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:35 smithi114 ceph-mon[30820]: pgmap v10359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:10:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:35 smithi116 ceph-mon[35411]: pgmap v10359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:10:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:37 smithi114 ceph-mon[30820]: pgmap v10360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:10:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:37 smithi116 ceph-mon[35411]: pgmap v10360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:10:39.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:10:39.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:10:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:39 smithi114 ceph-mon[30820]: pgmap v10361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:39 smithi116 ceph-mon[35411]: pgmap v10361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-mon[30820]: pgmap v10362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:10:41.876+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:10:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:10:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:10:42.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:10:42.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:10:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:10:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:10:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:10:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:10:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:10:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:10:42.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:10:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:10:42.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:41 smithi116 ceph-mon[35411]: pgmap v10362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:10:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:10:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:10:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:10:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:10:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:10:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:43 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:10:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:10:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:10:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:10:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:10:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:10:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:10:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:10:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:10:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:10:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:10:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:10:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:43 smithi114 ceph-mon[30820]: pgmap v10363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:10:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:43 smithi114 ceph-mon[30820]: pgmap v10364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:10:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:43 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:10:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:10:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:10:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:10:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:10:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:10:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:10:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:10:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:10:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:10:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:10:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:10:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:10:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:10:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:43 smithi116 ceph-mon[35411]: pgmap v10363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:10:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:43 smithi116 ceph-mon[35411]: pgmap v10364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:10:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:45 smithi114 ceph-mon[30820]: pgmap v10365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:10:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:45 smithi116 ceph-mon[35411]: pgmap v10365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:10:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:47 smithi114 ceph-mon[30820]: pgmap v10366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:10:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:47 smithi116 ceph-mon[35411]: pgmap v10366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:10:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:49 smithi114 ceph-mon[30820]: pgmap v10367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:10:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:49 smithi116 ceph-mon[35411]: pgmap v10367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:10:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:51 smithi114 ceph-mon[30820]: pgmap v10368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:10:52.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:51 smithi116 ceph-mon[35411]: pgmap v10368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:10:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:53 smithi114 ceph-mon[30820]: pgmap v10369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:10:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:53 smithi116 ceph-mon[35411]: pgmap v10369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:10:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:55 smithi114 ceph-mon[30820]: pgmap v10370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:55 smithi116 ceph-mon[35411]: pgmap v10370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:10:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:57 smithi114 ceph-mon[30820]: pgmap v10371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:10:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:57 smithi116 ceph-mon[35411]: pgmap v10371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:10:59 smithi114 ceph-mon[30820]: pgmap v10372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:10:59 smithi116 ceph-mon[35411]: pgmap v10372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:01 smithi114 ceph-mon[30820]: pgmap v10373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:01 smithi116 ceph-mon[35411]: pgmap v10373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:03 smithi114 ceph-mon[30820]: pgmap v10374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:03 smithi116 ceph-mon[35411]: pgmap v10374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:05 smithi114 ceph-mon[30820]: pgmap v10375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:05 smithi116 ceph-mon[35411]: pgmap v10375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:07 smithi114 ceph-mon[30820]: pgmap v10376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:07 smithi116 ceph-mon[35411]: pgmap v10376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:09 smithi114 ceph-mon[30820]: pgmap v10377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:09 smithi116 ceph-mon[35411]: pgmap v10377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:11 smithi114 ceph-mon[30820]: pgmap v10378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:11 smithi116 ceph-mon[35411]: pgmap v10378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:13 smithi114 ceph-mon[30820]: pgmap v10379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:13 smithi116 ceph-mon[35411]: pgmap v10379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:15 smithi114 ceph-mon[30820]: pgmap v10380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:15 smithi116 ceph-mon[35411]: pgmap v10380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:17 smithi114 ceph-mon[30820]: pgmap v10381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:17 smithi116 ceph-mon[35411]: pgmap v10381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:19 smithi114 ceph-mon[30820]: pgmap v10382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:19 smithi116 ceph-mon[35411]: pgmap v10382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:21 smithi114 ceph-mon[30820]: pgmap v10383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:21 smithi116 ceph-mon[35411]: pgmap v10383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:23 smithi114 ceph-mon[30820]: pgmap v10384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:23 smithi116 ceph-mon[35411]: pgmap v10384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:25 smithi114 ceph-mon[30820]: pgmap v10385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:25 smithi116 ceph-mon[35411]: pgmap v10385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:27 smithi114 ceph-mon[30820]: pgmap v10386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:27 smithi116 ceph-mon[35411]: pgmap v10386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:29 smithi114 ceph-mon[30820]: pgmap v10387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:29 smithi116 ceph-mon[35411]: pgmap v10387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:31 smithi114 ceph-mon[30820]: pgmap v10388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:31 smithi116 ceph-mon[35411]: pgmap v10388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:33 smithi114 ceph-mon[30820]: pgmap v10389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:33 smithi116 ceph-mon[35411]: pgmap v10389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:35 smithi114 ceph-mon[30820]: pgmap v10390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:35 smithi116 ceph-mon[35411]: pgmap v10390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:37 smithi114 ceph-mon[30820]: pgmap v10391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:37 smithi116 ceph-mon[35411]: pgmap v10391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:11:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:39 smithi114 ceph-mon[30820]: pgmap v10392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:39 smithi116 ceph-mon[35411]: pgmap v10392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:40 smithi116 ceph-mon[35411]: pgmap v10393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:40 smithi114 ceph-mon[30820]: pgmap v10393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:11:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:42 smithi116 ceph-mon[35411]: pgmap v10394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:11:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:42 smithi114 ceph-mon[30820]: pgmap v10394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:11:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:11:45.239 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:44 smithi114 ceph-mon[30820]: pgmap v10395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:44 smithi116 ceph-mon[35411]: pgmap v10395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:11:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:11:45.236+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:11:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:11:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:11:46.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:45 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:11:46.330 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:11:46.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:11:46.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:11:46.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:11:46.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:11:46.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:11:46.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:11:46.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:11:46.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:11:46.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:11:46.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:11:46.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:11:46.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:11:46.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:45 smithi116 ceph-mon[35411]: pgmap v10396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:45 smithi116 ceph-mon[35411]: pgmap v10397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:11:46.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:11:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:11:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:11:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:11:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:11:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:11:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:11:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:11:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:11:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:11:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:11:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:11:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:11:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:11:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-mon[30820]: pgmap v10396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T12:11:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-mon[30820]: pgmap v10397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:11:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:11:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:48 smithi114 ceph-mon[30820]: pgmap v10398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:11:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:48 smithi116 ceph-mon[35411]: pgmap v10398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:11:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:50 smithi114 ceph-mon[30820]: pgmap v10399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:11:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:50 smithi116 ceph-mon[35411]: pgmap v10399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:11:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:52 smithi114 ceph-mon[30820]: pgmap v10400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:11:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:52 smithi116 ceph-mon[35411]: pgmap v10400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:11:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:54 smithi114 ceph-mon[30820]: pgmap v10401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:11:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:54 smithi116 ceph-mon[35411]: pgmap v10401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:11:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:56 smithi114 ceph-mon[30820]: pgmap v10402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:11:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:56 smithi116 ceph-mon[35411]: pgmap v10402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:11:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:11:58 smithi114 ceph-mon[30820]: pgmap v10403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:11:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:11:58 smithi116 ceph-mon[35411]: pgmap v10403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:00 smithi114 ceph-mon[30820]: pgmap v10404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:00 smithi116 ceph-mon[35411]: pgmap v10404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:02 smithi114 ceph-mon[30820]: pgmap v10405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:12:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:02 smithi116 ceph-mon[35411]: pgmap v10405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:12:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:04 smithi114 ceph-mon[30820]: pgmap v10406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:04 smithi116 ceph-mon[35411]: pgmap v10406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:06 smithi114 ceph-mon[30820]: pgmap v10407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:06 smithi116 ceph-mon[35411]: pgmap v10407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:08 smithi114 ceph-mon[30820]: pgmap v10408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:08 smithi116 ceph-mon[35411]: pgmap v10408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:10.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:10 smithi114 ceph-mon[30820]: pgmap v10409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:10 smithi116 ceph-mon[35411]: pgmap v10409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:12 smithi114 ceph-mon[30820]: pgmap v10410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:12:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:12 smithi116 ceph-mon[35411]: pgmap v10410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:12:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:14 smithi114 ceph-mon[30820]: pgmap v10411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:14.678 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:14 smithi116 ceph-mon[35411]: pgmap v10411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:16 smithi114 ceph-mon[30820]: pgmap v10412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:16 smithi116 ceph-mon[35411]: pgmap v10412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:18 smithi114 ceph-mon[30820]: pgmap v10413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:18 smithi116 ceph-mon[35411]: pgmap v10413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:20 smithi114 ceph-mon[30820]: pgmap v10414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:20 smithi116 ceph-mon[35411]: pgmap v10414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:22 smithi114 ceph-mon[30820]: pgmap v10415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:12:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:22 smithi116 ceph-mon[35411]: pgmap v10415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:12:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:24 smithi114 ceph-mon[30820]: pgmap v10416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:24 smithi116 ceph-mon[35411]: pgmap v10416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:26 smithi114 ceph-mon[30820]: pgmap v10417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:26 smithi116 ceph-mon[35411]: pgmap v10417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:28 smithi114 ceph-mon[30820]: pgmap v10418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:28 smithi116 ceph-mon[35411]: pgmap v10418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:30 smithi114 ceph-mon[30820]: pgmap v10419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:30 smithi116 ceph-mon[35411]: pgmap v10419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:32 smithi114 ceph-mon[30820]: pgmap v10420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:12:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:32 smithi116 ceph-mon[35411]: pgmap v10420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:12:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:34 smithi114 ceph-mon[30820]: pgmap v10421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:34 smithi116 ceph-mon[35411]: pgmap v10421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:36 smithi114 ceph-mon[30820]: pgmap v10422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:36 smithi116 ceph-mon[35411]: pgmap v10422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:38 smithi114 ceph-mon[30820]: pgmap v10423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:38 smithi116 ceph-mon[35411]: pgmap v10423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:40 smithi114 ceph-mon[30820]: pgmap v10424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:40 smithi116 ceph-mon[35411]: pgmap v10424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:12:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:42 smithi114 ceph-mon[30820]: pgmap v10425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:12:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:42 smithi116 ceph-mon[35411]: pgmap v10425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:12:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:44 smithi114 ceph-mon[30820]: pgmap v10426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:44 smithi116 ceph-mon[35411]: pgmap v10426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:12:45.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:12:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:12:46.666 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:46 smithi114 ceph-mon[30820]: pgmap v10427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:46 smithi116 ceph-mon[35411]: pgmap v10427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:12:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-mon[30820]: pgmap v10428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:12:48.597+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:12:48.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:12:48.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:12:48.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:12:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:12:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:48 smithi116 ceph-mon[35411]: pgmap v10428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:12:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:12:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:12:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:12:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:12:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:49 smithi116 ceph-mon[35411]: pgmap v10429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:49 smithi116 ceph-mon[35411]: pgmap v10430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:12:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:12:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:12:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:12:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:12:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:12:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:12:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:12:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:12:49.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:49 smithi114 ceph-mon[30820]: pgmap v10429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:49 smithi114 ceph-mon[30820]: pgmap v10430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:12:49.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:12:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:51 smithi114 ceph-mon[30820]: pgmap v10431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:12:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:51 smithi116 ceph-mon[35411]: pgmap v10431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:12:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:53 smithi114 ceph-mon[30820]: pgmap v10432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:12:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:53 smithi116 ceph-mon[35411]: pgmap v10432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:12:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:55 smithi114 ceph-mon[30820]: pgmap v10433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:12:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:55 smithi116 ceph-mon[35411]: pgmap v10433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:12:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:57 smithi116 ceph-mon[35411]: pgmap v10434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:12:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:57 smithi114 ceph-mon[30820]: pgmap v10434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:12:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:12:59 smithi114 ceph-mon[30820]: pgmap v10435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:12:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:12:59 smithi116 ceph-mon[35411]: pgmap v10435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:13:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:01 smithi114 ceph-mon[30820]: pgmap v10436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:01 smithi116 ceph-mon[35411]: pgmap v10436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:03 smithi114 ceph-mon[30820]: pgmap v10437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:03 smithi116 ceph-mon[35411]: pgmap v10437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:05 smithi116 ceph-mon[35411]: pgmap v10438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:05 smithi114 ceph-mon[30820]: pgmap v10438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:07 smithi116 ceph-mon[35411]: pgmap v10439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:07 smithi114 ceph-mon[30820]: pgmap v10439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:09 smithi116 ceph-mon[35411]: pgmap v10440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:09.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:09 smithi114 ceph-mon[30820]: pgmap v10440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:11 smithi114 ceph-mon[30820]: pgmap v10441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:11 smithi116 ceph-mon[35411]: pgmap v10441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:13 smithi114 ceph-mon[30820]: pgmap v10442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:13 smithi116 ceph-mon[35411]: pgmap v10442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:15 smithi114 ceph-mon[30820]: pgmap v10443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:15 smithi116 ceph-mon[35411]: pgmap v10443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:17.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:17 smithi116 ceph-mon[35411]: pgmap v10444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:18.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:17 smithi114 ceph-mon[30820]: pgmap v10444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:19 smithi114 ceph-mon[30820]: pgmap v10445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:19 smithi116 ceph-mon[35411]: pgmap v10445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:21 smithi114 ceph-mon[30820]: pgmap v10446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:21 smithi116 ceph-mon[35411]: pgmap v10446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:23 smithi114 ceph-mon[30820]: pgmap v10447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:23 smithi116 ceph-mon[35411]: pgmap v10447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:25 smithi114 ceph-mon[30820]: pgmap v10448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:25 smithi116 ceph-mon[35411]: pgmap v10448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:27 smithi114 ceph-mon[30820]: pgmap v10449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:27 smithi116 ceph-mon[35411]: pgmap v10449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:30.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:29 smithi114 ceph-mon[30820]: pgmap v10450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:29 smithi116 ceph-mon[35411]: pgmap v10450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:31 smithi114 ceph-mon[30820]: pgmap v10451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:31 smithi116 ceph-mon[35411]: pgmap v10451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:33 smithi114 ceph-mon[30820]: pgmap v10452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:34.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:33 smithi116 ceph-mon[35411]: pgmap v10452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:35 smithi114 ceph-mon[30820]: pgmap v10453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:36.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:35 smithi116 ceph-mon[35411]: pgmap v10453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:37 smithi114 ceph-mon[30820]: pgmap v10454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:37 smithi116 ceph-mon[35411]: pgmap v10454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:39 smithi114 ceph-mon[30820]: pgmap v10455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:39 smithi116 ceph-mon[35411]: pgmap v10455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:41 smithi114 ceph-mon[30820]: pgmap v10456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:41 smithi116 ceph-mon[35411]: pgmap v10456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:43 smithi114 ceph-mon[30820]: pgmap v10457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:43 smithi116 ceph-mon[35411]: pgmap v10457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:45 smithi114 ceph-mon[30820]: pgmap v10458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:45 smithi116 ceph-mon[35411]: pgmap v10458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:13:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:47 smithi114 ceph-mon[30820]: pgmap v10459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:47 smithi116 ceph-mon[35411]: pgmap v10459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:13:49.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:13:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:13:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:49 smithi114 ceph-mon[30820]: pgmap v10460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:49 smithi116 ceph-mon[35411]: pgmap v10460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:13:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-mon[30820]: pgmap v10461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:13:51.957+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:13:52.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:13:52.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:13:52.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:13:52.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:13:52.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:13:52.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:13:52.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:13:52.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:13:52.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:13:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:13:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:51 smithi116 ceph-mon[35411]: pgmap v10461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:13:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:13:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:13:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:13:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:13:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:13:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:13:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:53 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:13:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:13:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:13:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:13:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:13:54.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:13:54.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:13:54.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:13:54.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:13:54.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:13:54.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:53 smithi114 ceph-mon[30820]: pgmap v10462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:13:54.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:53 smithi114 ceph-mon[30820]: pgmap v10463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:53 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:13:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:13:54.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:13:54.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:13:54.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:13:54.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:13:54.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:53 smithi116 ceph-mon[35411]: pgmap v10462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:13:54.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:53 smithi116 ceph-mon[35411]: pgmap v10463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:13:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:55 smithi114 ceph-mon[30820]: pgmap v10464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:13:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:55 smithi116 ceph-mon[35411]: pgmap v10464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:13:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:57 smithi114 ceph-mon[30820]: pgmap v10465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:13:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:57 smithi116 ceph-mon[35411]: pgmap v10465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:14:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:13:59 smithi114 ceph-mon[30820]: pgmap v10466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:14:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:13:59 smithi116 ceph-mon[35411]: pgmap v10466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:14:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:01 smithi114 ceph-mon[30820]: pgmap v10467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:14:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:01 smithi116 ceph-mon[35411]: pgmap v10467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:14:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:03 smithi114 ceph-mon[30820]: pgmap v10468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:14:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:03 smithi116 ceph-mon[35411]: pgmap v10468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:14:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:05 smithi114 ceph-mon[30820]: pgmap v10469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:05 smithi116 ceph-mon[35411]: pgmap v10469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:07 smithi114 ceph-mon[30820]: pgmap v10470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:07 smithi116 ceph-mon[35411]: pgmap v10470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:09 smithi114 ceph-mon[30820]: pgmap v10471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:09 smithi116 ceph-mon[35411]: pgmap v10471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:11 smithi114 ceph-mon[30820]: pgmap v10472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:11 smithi116 ceph-mon[35411]: pgmap v10472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:13 smithi114 ceph-mon[30820]: pgmap v10473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:13 smithi116 ceph-mon[35411]: pgmap v10473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:15 smithi114 ceph-mon[30820]: pgmap v10474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:15 smithi116 ceph-mon[35411]: pgmap v10474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:17 smithi114 ceph-mon[30820]: pgmap v10475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:17 smithi116 ceph-mon[35411]: pgmap v10475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:19 smithi114 ceph-mon[30820]: pgmap v10476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:19 smithi116 ceph-mon[35411]: pgmap v10476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:21 smithi114 ceph-mon[30820]: pgmap v10477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:21 smithi116 ceph-mon[35411]: pgmap v10477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:23 smithi114 ceph-mon[30820]: pgmap v10478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:23 smithi116 ceph-mon[35411]: pgmap v10478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:25 smithi114 ceph-mon[30820]: pgmap v10479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:25 smithi116 ceph-mon[35411]: pgmap v10479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:27 smithi114 ceph-mon[30820]: pgmap v10480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:27 smithi116 ceph-mon[35411]: pgmap v10480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:29 smithi114 ceph-mon[30820]: pgmap v10481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:29 smithi116 ceph-mon[35411]: pgmap v10481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:31 smithi114 ceph-mon[30820]: pgmap v10482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:31 smithi116 ceph-mon[35411]: pgmap v10482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:33 smithi114 ceph-mon[30820]: pgmap v10483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:33 smithi116 ceph-mon[35411]: pgmap v10483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:35 smithi114 ceph-mon[30820]: pgmap v10484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:35 smithi116 ceph-mon[35411]: pgmap v10484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:37 smithi114 ceph-mon[30820]: pgmap v10485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:37 smithi116 ceph-mon[35411]: pgmap v10485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:39 smithi114 ceph-mon[30820]: pgmap v10486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:40.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:39 smithi116 ceph-mon[35411]: pgmap v10486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:41 smithi114 ceph-mon[30820]: pgmap v10487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:41 smithi116 ceph-mon[35411]: pgmap v10487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:43 smithi114 ceph-mon[30820]: pgmap v10488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:43 smithi116 ceph-mon[35411]: pgmap v10488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:45 smithi114 ceph-mon[30820]: pgmap v10489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:45 smithi116 ceph-mon[35411]: pgmap v10489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:47 smithi114 ceph-mon[30820]: pgmap v10490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:47 smithi116 ceph-mon[35411]: pgmap v10490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:14:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:49 smithi114 ceph-mon[30820]: pgmap v10491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:49 smithi116 ceph-mon[35411]: pgmap v10491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:51 smithi114 ceph-mon[30820]: pgmap v10492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:51 smithi116 ceph-mon[35411]: pgmap v10492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:14:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:14:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:14:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:53.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:14:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:53 smithi114 ceph-mon[30820]: pgmap v10493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:53 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:14:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:53 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:14:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:53 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:14:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:53 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:14:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:53 smithi116 ceph-mon[35411]: pgmap v10493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:14:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:53 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:14:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:53 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:14:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:53 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:14:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:53 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:14:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:14:55.317+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:14:55.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:14:55.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:14:55.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:14:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-mon[30820]: pgmap v10494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:14:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:14:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:14:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:55 smithi116 ceph-mon[35411]: pgmap v10494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:14:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:14:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:14:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:14:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:14:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:14:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:14:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:14:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:14:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:14:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:14:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:14:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:14:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:14:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:14:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:14:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:56 smithi114 ceph-mon[30820]: pgmap v10495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T12:14:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:56 smithi114 ceph-mon[30820]: pgmap v10496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:14:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:14:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:14:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:14:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:14:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:14:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:14:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:14:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:14:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:14:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:14:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:56 smithi116 ceph-mon[35411]: pgmap v10495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T12:14:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:56 smithi116 ceph-mon[35411]: pgmap v10496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:14:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:14:58 smithi114 ceph-mon[30820]: pgmap v10497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:14:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:14:58 smithi116 ceph-mon[35411]: pgmap v10497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:00 smithi114 ceph-mon[30820]: pgmap v10498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:00 smithi116 ceph-mon[35411]: pgmap v10498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:01 smithi114 ceph-mon[30820]: pgmap v10499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:01 smithi116 ceph-mon[35411]: pgmap v10499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:04 smithi114 ceph-mon[30820]: pgmap v10500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:04 smithi116 ceph-mon[35411]: pgmap v10500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:06 smithi114 ceph-mon[30820]: pgmap v10501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:15:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:06 smithi116 ceph-mon[35411]: pgmap v10501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:15:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:08 smithi114 ceph-mon[30820]: pgmap v10502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:15:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:08 smithi116 ceph-mon[35411]: pgmap v10502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:15:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:10 smithi114 ceph-mon[30820]: pgmap v10503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:15:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:10 smithi116 ceph-mon[35411]: pgmap v10503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:15:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:12 smithi114 ceph-mon[30820]: pgmap v10504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:12 smithi116 ceph-mon[35411]: pgmap v10504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:14 smithi114 ceph-mon[30820]: pgmap v10505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:15:14.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:14 smithi116 ceph-mon[35411]: pgmap v10505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:15:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:16 smithi114 ceph-mon[30820]: pgmap v10506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:15:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:16 smithi116 ceph-mon[35411]: pgmap v10506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:15:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:18 smithi114 ceph-mon[30820]: pgmap v10507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:15:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:18 smithi116 ceph-mon[35411]: pgmap v10507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:15:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:20 smithi114 ceph-mon[30820]: pgmap v10508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:15:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:20 smithi116 ceph-mon[35411]: pgmap v10508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:15:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:22 smithi116 ceph-mon[35411]: pgmap v10509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:22 smithi114 ceph-mon[30820]: pgmap v10509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:24 smithi116 ceph-mon[35411]: pgmap v10510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:15:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:24 smithi114 ceph-mon[30820]: pgmap v10510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:15:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:26 smithi116 ceph-mon[35411]: pgmap v10511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:15:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:26 smithi114 ceph-mon[30820]: pgmap v10511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:15:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:28 smithi114 ceph-mon[30820]: pgmap v10512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:15:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:28 smithi116 ceph-mon[35411]: pgmap v10512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:15:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:30 smithi114 ceph-mon[30820]: pgmap v10513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:15:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:30 smithi116 ceph-mon[35411]: pgmap v10513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:15:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:32 smithi114 ceph-mon[30820]: pgmap v10514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:32 smithi116 ceph-mon[35411]: pgmap v10514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:34 smithi114 ceph-mon[30820]: pgmap v10515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:15:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:34 smithi116 ceph-mon[35411]: pgmap v10515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:15:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:36 smithi114 ceph-mon[30820]: pgmap v10516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:36 smithi116 ceph-mon[35411]: pgmap v10516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:38 smithi114 ceph-mon[30820]: pgmap v10517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T12:15:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:38 smithi116 ceph-mon[35411]: pgmap v10517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T12:15:40.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:40 smithi114 ceph-mon[30820]: pgmap v10518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:15:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:40 smithi116 ceph-mon[35411]: pgmap v10518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:15:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:42 smithi114 ceph-mon[30820]: pgmap v10519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:42 smithi116 ceph-mon[35411]: pgmap v10519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:44 smithi114 ceph-mon[30820]: pgmap v10520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:15:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:44 smithi116 ceph-mon[35411]: pgmap v10520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:15:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:46 smithi114 ceph-mon[30820]: pgmap v10521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:46 smithi116 ceph-mon[35411]: pgmap v10521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:48.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:48 smithi116 ceph-mon[35411]: pgmap v10522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T12:15:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:48 smithi114 ceph-mon[30820]: pgmap v10522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T12:15:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:50 smithi114 ceph-mon[30820]: pgmap v10523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:15:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:50 smithi116 ceph-mon[35411]: pgmap v10523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:15:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:52 smithi114 ceph-mon[30820]: pgmap v10524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:52 smithi116 ceph-mon[35411]: pgmap v10524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:15:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:54 smithi114 ceph-mon[30820]: pgmap v10525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:15:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:54 smithi116 ceph-mon[35411]: pgmap v10525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:15:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:15:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:15:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:56 smithi114 ceph-mon[30820]: pgmap v10526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:56 smithi116 ceph-mon[35411]: pgmap v10526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:15:58.916 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-mon[30820]: pgmap v10527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:15:58.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:15:58.676+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:15:58.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:15:58.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:15:58.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:15:58.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:15:58.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:15:58.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:15:58.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:15:58.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:15:58.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:15:58 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:15:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:58 smithi116 ceph-mon[35411]: pgmap v10527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:15:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:59 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:15:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:15:59.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:15:59.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:15:59.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:15:59.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:15:59.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:15:59.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:15:59.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:15:59.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:15:59.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:15:59.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:15:59.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:15:59.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:15:59.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:15:59.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:15:59.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:15:59.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:15:59.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:15:59.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:15:59.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:15:59.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:15:59.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:15:59.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:15:59.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:15:59.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:59 smithi114 ceph-mon[30820]: pgmap v10528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:15:59.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:59 smithi114 ceph-mon[30820]: pgmap v10529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:15:59.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:15:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:15:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:59 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:15:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:15:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:15:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:15:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:15:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:15:59.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:15:59.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:15:59.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:15:59.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:15:59.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:15:59.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:15:59.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:15:59.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:15:59.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:59 smithi116 ceph-mon[35411]: pgmap v10528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:15:59.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:59 smithi116 ceph-mon[35411]: pgmap v10529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:15:59.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:15:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:16:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:01 smithi114 ceph-mon[30820]: pgmap v10530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:16:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:01 smithi116 ceph-mon[35411]: pgmap v10530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:16:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:03 smithi114 ceph-mon[30820]: pgmap v10531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:16:04.222 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:03 smithi116 ceph-mon[35411]: pgmap v10531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:16:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:05 smithi114 ceph-mon[30820]: pgmap v10532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:16:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:05 smithi116 ceph-mon[35411]: pgmap v10532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:16:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:07 smithi114 ceph-mon[30820]: pgmap v10533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:16:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:07 smithi116 ceph-mon[35411]: pgmap v10533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:16:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:09 smithi114 ceph-mon[30820]: pgmap v10534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:16:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:09 smithi116 ceph-mon[35411]: pgmap v10534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:16:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:11 smithi114 ceph-mon[30820]: pgmap v10535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:12.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:11 smithi116 ceph-mon[35411]: pgmap v10535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:13 smithi114 ceph-mon[30820]: pgmap v10536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:13 smithi116 ceph-mon[35411]: pgmap v10536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:15 smithi114 ceph-mon[30820]: pgmap v10537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T12:16:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:15 smithi116 ceph-mon[35411]: pgmap v10537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T12:16:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:17 smithi114 ceph-mon[30820]: pgmap v10538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:17 smithi116 ceph-mon[35411]: pgmap v10538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:19 smithi114 ceph-mon[30820]: pgmap v10539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:19 smithi116 ceph-mon[35411]: pgmap v10539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:21 smithi114 ceph-mon[30820]: pgmap v10540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:21 smithi116 ceph-mon[35411]: pgmap v10540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:23 smithi114 ceph-mon[30820]: pgmap v10541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:23 smithi116 ceph-mon[35411]: pgmap v10541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:25 smithi114 ceph-mon[30820]: pgmap v10542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:16:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:25 smithi116 ceph-mon[35411]: pgmap v10542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:16:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:27 smithi114 ceph-mon[30820]: pgmap v10543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:27 smithi116 ceph-mon[35411]: pgmap v10543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:29 smithi114 ceph-mon[30820]: pgmap v10544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:29 smithi116 ceph-mon[35411]: pgmap v10544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:31 smithi114 ceph-mon[30820]: pgmap v10545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:31 smithi116 ceph-mon[35411]: pgmap v10545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:33 smithi114 ceph-mon[30820]: pgmap v10546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:33 smithi116 ceph-mon[35411]: pgmap v10546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:35 smithi114 ceph-mon[30820]: pgmap v10547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:16:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:35 smithi116 ceph-mon[35411]: pgmap v10547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:16:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:37 smithi114 ceph-mon[30820]: pgmap v10548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:37 smithi116 ceph-mon[35411]: pgmap v10548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:39 smithi114 ceph-mon[30820]: pgmap v10549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:39 smithi116 ceph-mon[35411]: pgmap v10549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:41 smithi114 ceph-mon[30820]: pgmap v10550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:41 smithi116 ceph-mon[35411]: pgmap v10550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:43 smithi114 ceph-mon[30820]: pgmap v10551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:44.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:43 smithi116 ceph-mon[35411]: pgmap v10551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:45 smithi114 ceph-mon[30820]: pgmap v10552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:16:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:45 smithi116 ceph-mon[35411]: pgmap v10552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:16:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:47 smithi114 ceph-mon[30820]: pgmap v10553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:47 smithi116 ceph-mon[35411]: pgmap v10553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:49 smithi114 ceph-mon[30820]: pgmap v10554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:50.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:49 smithi116 ceph-mon[35411]: pgmap v10554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:51 smithi114 ceph-mon[30820]: pgmap v10555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:51 smithi116 ceph-mon[35411]: pgmap v10555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:16:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:53 smithi114 ceph-mon[30820]: pgmap v10556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:53 smithi116 ceph-mon[35411]: pgmap v10556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:16:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:55 smithi114 ceph-mon[30820]: pgmap v10557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:16:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:55 smithi116 ceph-mon[35411]: pgmap v10557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:16:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:57 smithi114 ceph-mon[30820]: pgmap v10558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:57 smithi116 ceph-mon[35411]: pgmap v10558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:16:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:16:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:17:00.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:16:59 smithi114 ceph-mon[30820]: pgmap v10559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:16:59 smithi116 ceph-mon[35411]: pgmap v10559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:01 smithi114 ceph-mon[30820]: pgmap v10560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:02.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:17:02.037+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:17:02.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:17:02.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:17:02.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:17:02.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:17:02.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:17:02.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:17:02.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:17:02.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:17:02.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:17:02.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:17:02.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:17:02.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:17:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:01 smithi116 ceph-mon[35411]: pgmap v10560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:17:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:17:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:17:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:17:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:17:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:17:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:17:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:17:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:17:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:17:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:17:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:17:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:17:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:17:04.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:17:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:17:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:17:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:17:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:03 smithi114 ceph-mon[30820]: pgmap v10561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:17:04.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:03 smithi114 ceph-mon[30820]: pgmap v10562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:17:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:17:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:17:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:17:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:17:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:17:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:17:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:17:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:17:04.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:17:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:17:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:17:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:03 smithi116 ceph-mon[35411]: pgmap v10561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:17:04.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:03 smithi116 ceph-mon[35411]: pgmap v10562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:17:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:05 smithi114 ceph-mon[30820]: pgmap v10563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:17:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:05 smithi116 ceph-mon[35411]: pgmap v10563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:17:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:07 smithi114 ceph-mon[30820]: pgmap v10564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:17:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:07 smithi116 ceph-mon[35411]: pgmap v10564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:17:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:09 smithi114 ceph-mon[30820]: pgmap v10565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:17:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:09 smithi116 ceph-mon[35411]: pgmap v10565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:17:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:11 smithi114 ceph-mon[30820]: pgmap v10566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:17:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:11 smithi116 ceph-mon[35411]: pgmap v10566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:17:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:13 smithi114 ceph-mon[30820]: pgmap v10567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:17:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:13 smithi116 ceph-mon[35411]: pgmap v10567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:17:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:15 smithi114 ceph-mon[30820]: pgmap v10568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:15 smithi116 ceph-mon[35411]: pgmap v10568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:17 smithi114 ceph-mon[30820]: pgmap v10569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:17 smithi116 ceph-mon[35411]: pgmap v10569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:19 smithi114 ceph-mon[30820]: pgmap v10570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:19 smithi116 ceph-mon[35411]: pgmap v10570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:21 smithi114 ceph-mon[30820]: pgmap v10571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:17:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:21 smithi116 ceph-mon[35411]: pgmap v10571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:17:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:23 smithi114 ceph-mon[30820]: pgmap v10572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:17:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:23 smithi116 ceph-mon[35411]: pgmap v10572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:17:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:25 smithi114 ceph-mon[30820]: pgmap v10573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:26.198 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:25 smithi116 ceph-mon[35411]: pgmap v10573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:27 smithi114 ceph-mon[30820]: pgmap v10574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:27 smithi116 ceph-mon[35411]: pgmap v10574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:29 smithi114 ceph-mon[30820]: pgmap v10575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:17:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:29 smithi116 ceph-mon[35411]: pgmap v10575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:17:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:31 smithi114 ceph-mon[30820]: pgmap v10576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:17:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:31 smithi116 ceph-mon[35411]: pgmap v10576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:17:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:33 smithi114 ceph-mon[30820]: pgmap v10577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:17:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:33 smithi116 ceph-mon[35411]: pgmap v10577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:17:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:35 smithi116 ceph-mon[35411]: pgmap v10578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:17:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:35 smithi114 ceph-mon[30820]: pgmap v10578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:17:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:37 smithi116 ceph-mon[35411]: pgmap v10579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:37 smithi114 ceph-mon[30820]: pgmap v10579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:39 smithi116 ceph-mon[35411]: pgmap v10580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:39 smithi114 ceph-mon[30820]: pgmap v10580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:42.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:41 smithi114 ceph-mon[30820]: pgmap v10581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:17:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:41 smithi116 ceph-mon[35411]: pgmap v10581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:17:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:43 smithi114 ceph-mon[30820]: pgmap v10582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:17:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:43 smithi116 ceph-mon[35411]: pgmap v10582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:17:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:44 smithi114 ceph-mon[30820]: pgmap v10583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:44 smithi116 ceph-mon[35411]: pgmap v10583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:47 smithi114 ceph-mon[30820]: pgmap v10584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:47 smithi116 ceph-mon[35411]: pgmap v10584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:49 smithi114 ceph-mon[30820]: pgmap v10585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:49 smithi116 ceph-mon[35411]: pgmap v10585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:51 smithi114 ceph-mon[30820]: pgmap v10586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:17:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:51 smithi116 ceph-mon[35411]: pgmap v10586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:17:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:53 smithi114 ceph-mon[30820]: pgmap v10587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:17:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:53 smithi116 ceph-mon[35411]: pgmap v10587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:17:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:55 smithi114 ceph-mon[30820]: pgmap v10588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:55 smithi116 ceph-mon[35411]: pgmap v10588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:57 smithi114 ceph-mon[30820]: pgmap v10589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:57.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:57 smithi116 ceph-mon[35411]: pgmap v10589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:17:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:17:59 smithi114 ceph-mon[30820]: pgmap v10590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:17:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:17:59 smithi116 ceph-mon[35411]: pgmap v10590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:01 smithi114 ceph-mon[30820]: pgmap v10591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:01 smithi116 ceph-mon[35411]: pgmap v10591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:03 smithi116 ceph-mon[35411]: pgmap v10592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:18:03.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:03 smithi114 ceph-mon[30820]: pgmap v10592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:03.343 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:18:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:05.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:18:05.397+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:18:05.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:18:05.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:18:05.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:18:05.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:18:05.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:18:05.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:18:05.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:18:05.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:18:05.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:18:05.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:18:05.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:18:05.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:18:05.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:18:05.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:18:05.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:18:05.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:18:05.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:18:05.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:18:05.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:18:05.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:18:05.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:18:05.672 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-mon[30820]: pgmap v10593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:18:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:05 smithi116 ceph-mon[35411]: pgmap v10593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:18:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:18:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:18:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:06 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:18:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:18:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:18:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:18:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:18:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:18:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:18:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:18:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:18:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:18:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:18:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:06 smithi114 ceph-mon[30820]: pgmap v10594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T12:18:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:06 smithi114 ceph-mon[30820]: pgmap v10595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:06 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:18:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:18:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:06 smithi116 ceph-mon[35411]: pgmap v10594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T12:18:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:06 smithi116 ceph-mon[35411]: pgmap v10595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:08 smithi114 ceph-mon[30820]: pgmap v10596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:08 smithi116 ceph-mon[35411]: pgmap v10596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:10 smithi114 ceph-mon[30820]: pgmap v10597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:10 smithi116 ceph-mon[35411]: pgmap v10597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:12 smithi114 ceph-mon[30820]: pgmap v10598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:12 smithi116 ceph-mon[35411]: pgmap v10598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:14 smithi114 ceph-mon[30820]: pgmap v10599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:14 smithi116 ceph-mon[35411]: pgmap v10599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:18:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:16 smithi114 ceph-mon[30820]: pgmap v10600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:18:17.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:16 smithi116 ceph-mon[35411]: pgmap v10600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:18:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:18 smithi114 ceph-mon[30820]: pgmap v10601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:18 smithi116 ceph-mon[35411]: pgmap v10601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:20 smithi114 ceph-mon[30820]: pgmap v10602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:20 smithi116 ceph-mon[35411]: pgmap v10602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:22 smithi114 ceph-mon[30820]: pgmap v10603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:22 smithi116 ceph-mon[35411]: pgmap v10603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:24 smithi114 ceph-mon[30820]: pgmap v10604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:24 smithi116 ceph-mon[35411]: pgmap v10604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:26 smithi114 ceph-mon[30820]: pgmap v10605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:18:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:26 smithi116 ceph-mon[35411]: pgmap v10605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:18:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:28 smithi114 ceph-mon[30820]: pgmap v10606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:28 smithi116 ceph-mon[35411]: pgmap v10606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:30 smithi114 ceph-mon[30820]: pgmap v10607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:30 smithi116 ceph-mon[35411]: pgmap v10607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:32 smithi114 ceph-mon[30820]: pgmap v10608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:32 smithi116 ceph-mon[35411]: pgmap v10608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:34 smithi114 ceph-mon[30820]: pgmap v10609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:34 smithi116 ceph-mon[35411]: pgmap v10609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:36 smithi114 ceph-mon[30820]: pgmap v10610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:18:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:36 smithi116 ceph-mon[35411]: pgmap v10610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:18:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:38 smithi114 ceph-mon[30820]: pgmap v10611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:38 smithi116 ceph-mon[35411]: pgmap v10611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:40 smithi114 ceph-mon[30820]: pgmap v10612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:40 smithi116 ceph-mon[35411]: pgmap v10612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:42 smithi114 ceph-mon[30820]: pgmap v10613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:42 smithi116 ceph-mon[35411]: pgmap v10613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:44 smithi114 ceph-mon[30820]: pgmap v10614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:44 smithi116 ceph-mon[35411]: pgmap v10614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:46 smithi114 ceph-mon[30820]: pgmap v10615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:18:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:46 smithi116 ceph-mon[35411]: pgmap v10615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:18:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:48 smithi114 ceph-mon[30820]: pgmap v10616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:48 smithi116 ceph-mon[35411]: pgmap v10616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:50 smithi114 ceph-mon[30820]: pgmap v10617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:50 smithi116 ceph-mon[35411]: pgmap v10617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:18:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:52 smithi114 ceph-mon[30820]: pgmap v10618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:52 smithi116 ceph-mon[35411]: pgmap v10618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:18:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:54 smithi114 ceph-mon[30820]: pgmap v10619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:54 smithi116 ceph-mon[35411]: pgmap v10619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:56 smithi114 ceph-mon[30820]: pgmap v10620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:18:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:56 smithi116 ceph-mon[35411]: pgmap v10620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:18:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:18:58 smithi114 ceph-mon[30820]: pgmap v10621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:18:59.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:18:58 smithi116 ceph-mon[35411]: pgmap v10621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:00 smithi114 ceph-mon[30820]: pgmap v10622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:00 smithi116 ceph-mon[35411]: pgmap v10622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:02 smithi114 ceph-mon[30820]: pgmap v10623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:02 smithi116 ceph-mon[35411]: pgmap v10623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:04 smithi114 ceph-mon[30820]: pgmap v10624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:04 smithi116 ceph-mon[35411]: pgmap v10624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:19:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:06.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:19:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:06 smithi114 ceph-mon[30820]: pgmap v10625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:06 smithi116 ceph-mon[35411]: pgmap v10625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-mon[30820]: pgmap v10626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:19:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:19:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:19:08.693+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:19:09.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:19:09.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:19:08 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:19:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:08 smithi116 ceph-mon[35411]: pgmap v10626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:19:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:19:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:19:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:09 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:19:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:19:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:19:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:19:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:19:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:19:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:19:10.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:19:10.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:19:10.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:19:10.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:19:10.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:09 smithi114 ceph-mon[30820]: pgmap v10627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:19:10.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:09 smithi114 ceph-mon[30820]: pgmap v10628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:09 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:19:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:19:10.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:19:10.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:09 smithi116 ceph-mon[35411]: pgmap v10627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:19:10.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:09 smithi116 ceph-mon[35411]: pgmap v10628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:19:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:11 smithi114 ceph-mon[30820]: pgmap v10629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:19:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:11 smithi116 ceph-mon[35411]: pgmap v10629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:19:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:13 smithi114 ceph-mon[30820]: pgmap v10630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:19:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:13 smithi116 ceph-mon[35411]: pgmap v10630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:19:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:15 smithi114 ceph-mon[30820]: pgmap v10631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:19:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:15 smithi116 ceph-mon[35411]: pgmap v10631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:19:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:17 smithi114 ceph-mon[30820]: pgmap v10632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:19:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:17 smithi116 ceph-mon[35411]: pgmap v10632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:19:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:19 smithi114 ceph-mon[30820]: pgmap v10633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:19:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:19 smithi116 ceph-mon[35411]: pgmap v10633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:19:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:21 smithi114 ceph-mon[30820]: pgmap v10634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:21 smithi116 ceph-mon[35411]: pgmap v10634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:23 smithi114 ceph-mon[30820]: pgmap v10635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:23 smithi116 ceph-mon[35411]: pgmap v10635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:25 smithi114 ceph-mon[30820]: pgmap v10636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:25 smithi116 ceph-mon[35411]: pgmap v10636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:27 smithi114 ceph-mon[30820]: pgmap v10637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:27 smithi116 ceph-mon[35411]: pgmap v10637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:29 smithi114 ceph-mon[30820]: pgmap v10638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:30.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:29 smithi116 ceph-mon[35411]: pgmap v10638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:31 smithi116 ceph-mon[35411]: pgmap v10639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:31 smithi114 ceph-mon[30820]: pgmap v10639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:33 smithi116 ceph-mon[35411]: pgmap v10640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:33 smithi114 ceph-mon[30820]: pgmap v10640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:35 smithi116 ceph-mon[35411]: pgmap v10641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:35 smithi114 ceph-mon[30820]: pgmap v10641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:37 smithi114 ceph-mon[30820]: pgmap v10642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:37 smithi116 ceph-mon[35411]: pgmap v10642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:39 smithi114 ceph-mon[30820]: pgmap v10643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:39 smithi116 ceph-mon[35411]: pgmap v10643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:41 smithi114 ceph-mon[30820]: pgmap v10644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:41 smithi116 ceph-mon[35411]: pgmap v10644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:44.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:43 smithi114 ceph-mon[30820]: pgmap v10645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:43 smithi116 ceph-mon[35411]: pgmap v10645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:45 smithi114 ceph-mon[30820]: pgmap v10646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:45 smithi116 ceph-mon[35411]: pgmap v10646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:47 smithi114 ceph-mon[30820]: pgmap v10647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:47 smithi116 ceph-mon[35411]: pgmap v10647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:49 smithi114 ceph-mon[30820]: pgmap v10648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:49 smithi116 ceph-mon[35411]: pgmap v10648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:52.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:51 smithi116 ceph-mon[35411]: pgmap v10649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:51 smithi114 ceph-mon[30820]: pgmap v10649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:19:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:53 smithi114 ceph-mon[30820]: pgmap v10650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:53 smithi116 ceph-mon[35411]: pgmap v10650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:19:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:55 smithi114 ceph-mon[30820]: pgmap v10651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:55 smithi116 ceph-mon[35411]: pgmap v10651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:19:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:57 smithi114 ceph-mon[30820]: pgmap v10652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:19:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:57 smithi116 ceph-mon[35411]: pgmap v10652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:19:59 smithi114 ceph-mon[30820]: pgmap v10653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:19:59 smithi116 ceph-mon[35411]: pgmap v10653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:20:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:20:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:01 smithi114 ceph-mon[30820]: pgmap v10654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:01 smithi116 ceph-mon[35411]: pgmap v10654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:03 smithi114 ceph-mon[30820]: pgmap v10655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:03 smithi116 ceph-mon[35411]: pgmap v10655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:05 smithi114 ceph-mon[30820]: pgmap v10656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:20:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:05 smithi116 ceph-mon[35411]: pgmap v10656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:20:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:07 smithi114 ceph-mon[30820]: pgmap v10657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:07 smithi116 ceph-mon[35411]: pgmap v10657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:09 smithi114 ceph-mon[30820]: pgmap v10658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:20:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:09 smithi116 ceph-mon[35411]: pgmap v10658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:20:12.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:11 smithi114 ceph-mon[30820]: pgmap v10659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:12.057 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:20:12.052+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:20:12.057 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:20:12.057 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:20:12.057 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:20:12.057 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:20:12.057 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:20:12.057 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:20:12.057 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:20:12.058 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:20:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:11 smithi116 ceph-mon[35411]: pgmap v10659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:20:12.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:20:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:20:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:20:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:20:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:20:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:20:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:20:14.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:13 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:20:14.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:20:14.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:20:14.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:20:14.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:20:14.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:20:14.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:20:14.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:20:14.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:20:14.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:20:14.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:13 smithi114 ceph-mon[30820]: pgmap v10660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:20:14.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:13 smithi114 ceph-mon[30820]: pgmap v10661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:20:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:13 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:20:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:20:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:20:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:20:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:20:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:20:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:20:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:20:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:20:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:20:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:20:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:13 smithi116 ceph-mon[35411]: pgmap v10660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:20:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:13 smithi116 ceph-mon[35411]: pgmap v10661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:20:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:15 smithi114 ceph-mon[30820]: pgmap v10662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:20:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:15 smithi116 ceph-mon[35411]: pgmap v10662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:20:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:17 smithi114 ceph-mon[30820]: pgmap v10663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:20:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:17 smithi116 ceph-mon[35411]: pgmap v10663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:20:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:19 smithi114 ceph-mon[30820]: pgmap v10664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:20:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:19 smithi116 ceph-mon[35411]: pgmap v10664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:20:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:21 smithi114 ceph-mon[30820]: pgmap v10665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:20:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:21 smithi116 ceph-mon[35411]: pgmap v10665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:20:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:23 smithi114 ceph-mon[30820]: pgmap v10666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:20:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:23 smithi116 ceph-mon[35411]: pgmap v10666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:20:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:25 smithi114 ceph-mon[30820]: pgmap v10667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:25 smithi116 ceph-mon[35411]: pgmap v10667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:27 smithi114 ceph-mon[30820]: pgmap v10668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:27 smithi116 ceph-mon[35411]: pgmap v10668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:29 smithi114 ceph-mon[30820]: pgmap v10669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:29 smithi116 ceph-mon[35411]: pgmap v10669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:31 smithi114 ceph-mon[30820]: pgmap v10670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:20:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:31 smithi116 ceph-mon[35411]: pgmap v10670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:20:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:33 smithi114 ceph-mon[30820]: pgmap v10671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:33 smithi116 ceph-mon[35411]: pgmap v10671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:35 smithi114 ceph-mon[30820]: pgmap v10672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:35 smithi116 ceph-mon[35411]: pgmap v10672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:37 smithi114 ceph-mon[30820]: pgmap v10673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:37 smithi116 ceph-mon[35411]: pgmap v10673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:39 smithi114 ceph-mon[30820]: pgmap v10674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:39 smithi116 ceph-mon[35411]: pgmap v10674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:41 smithi114 ceph-mon[30820]: pgmap v10675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:20:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:41 smithi116 ceph-mon[35411]: pgmap v10675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:20:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:43 smithi114 ceph-mon[30820]: pgmap v10676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:43 smithi116 ceph-mon[35411]: pgmap v10676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:45 smithi114 ceph-mon[30820]: pgmap v10677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:45 smithi116 ceph-mon[35411]: pgmap v10677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:47 smithi114 ceph-mon[30820]: pgmap v10678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:47 smithi116 ceph-mon[35411]: pgmap v10678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:49 smithi114 ceph-mon[30820]: pgmap v10679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:49 smithi116 ceph-mon[35411]: pgmap v10679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:51 smithi114 ceph-mon[30820]: pgmap v10680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:20:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:51 smithi116 ceph-mon[35411]: pgmap v10680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:20:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:53 smithi114 ceph-mon[30820]: pgmap v10681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:53 smithi116 ceph-mon[35411]: pgmap v10681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:20:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:55 smithi114 ceph-mon[30820]: pgmap v10682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:55 smithi116 ceph-mon[35411]: pgmap v10682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:20:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:57 smithi114 ceph-mon[30820]: pgmap v10683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:20:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:57 smithi116 ceph-mon[35411]: pgmap v10683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:20:59 smithi114 ceph-mon[30820]: pgmap v10684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:20:59 smithi116 ceph-mon[35411]: pgmap v10684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:01 smithi114 ceph-mon[30820]: pgmap v10685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:01 smithi116 ceph-mon[35411]: pgmap v10685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:03 smithi114 ceph-mon[30820]: pgmap v10686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:03 smithi116 ceph-mon[35411]: pgmap v10686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:05 smithi114 ceph-mon[30820]: pgmap v10687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:05 smithi116 ceph-mon[35411]: pgmap v10687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:07 smithi116 ceph-mon[35411]: pgmap v10688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:07 smithi114 ceph-mon[30820]: pgmap v10688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:09 smithi116 ceph-mon[35411]: pgmap v10689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:09 smithi114 ceph-mon[30820]: pgmap v10689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:11 smithi116 ceph-mon[35411]: pgmap v10690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:11 smithi114 ceph-mon[30820]: pgmap v10690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:21:13.428 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:21:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:13 smithi114 ceph-mon[30820]: pgmap v10691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:13 smithi116 ceph-mon[35411]: pgmap v10691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:21:15.412+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:21:15.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:21:15.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:21:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:21:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:21:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:21:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:21:15.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-mon[30820]: pgmap v10692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:21:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:15 smithi116 ceph-mon[35411]: pgmap v10692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:21:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:21:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:16 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:21:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:21:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:16 smithi114 ceph-mon[30820]: pgmap v10693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T12:21:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:16 smithi114 ceph-mon[30820]: pgmap v10694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:16 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:21:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:21:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:21:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:16 smithi116 ceph-mon[35411]: pgmap v10693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T12:21:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:16 smithi116 ceph-mon[35411]: pgmap v10694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:18 smithi114 ceph-mon[30820]: pgmap v10695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:18 smithi116 ceph-mon[35411]: pgmap v10695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:20 smithi114 ceph-mon[30820]: pgmap v10696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:20 smithi116 ceph-mon[35411]: pgmap v10696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:23.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:22 smithi116 ceph-mon[35411]: pgmap v10697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:22 smithi114 ceph-mon[30820]: pgmap v10697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:24 smithi114 ceph-mon[30820]: pgmap v10698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:24 smithi116 ceph-mon[35411]: pgmap v10698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:21:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:26 smithi114 ceph-mon[30820]: pgmap v10699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:26 smithi116 ceph-mon[35411]: pgmap v10699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:21:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:28 smithi114 ceph-mon[30820]: pgmap v10700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:28 smithi116 ceph-mon[35411]: pgmap v10700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:30 smithi114 ceph-mon[30820]: pgmap v10701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:30 smithi116 ceph-mon[35411]: pgmap v10701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:32 smithi114 ceph-mon[30820]: pgmap v10702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:32 smithi116 ceph-mon[35411]: pgmap v10702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:34 smithi114 ceph-mon[30820]: pgmap v10703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:34 smithi116 ceph-mon[35411]: pgmap v10703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:36 smithi114 ceph-mon[30820]: pgmap v10704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:36 smithi116 ceph-mon[35411]: pgmap v10704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:38 smithi114 ceph-mon[30820]: pgmap v10705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:38 smithi116 ceph-mon[35411]: pgmap v10705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:40 smithi114 ceph-mon[30820]: pgmap v10706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:40 smithi116 ceph-mon[35411]: pgmap v10706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:42 smithi114 ceph-mon[30820]: pgmap v10707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:42 smithi116 ceph-mon[35411]: pgmap v10707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:44 smithi114 ceph-mon[30820]: pgmap v10708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:44 smithi116 ceph-mon[35411]: pgmap v10708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:46 smithi114 ceph-mon[30820]: pgmap v10709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:46 smithi116 ceph-mon[35411]: pgmap v10709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:48 smithi114 ceph-mon[30820]: pgmap v10710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:48 smithi116 ceph-mon[35411]: pgmap v10710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:50 smithi114 ceph-mon[30820]: pgmap v10711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:50 smithi116 ceph-mon[35411]: pgmap v10711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:21:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:52 smithi114 ceph-mon[30820]: pgmap v10712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:52 smithi116 ceph-mon[35411]: pgmap v10712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:21:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:54 smithi114 ceph-mon[30820]: pgmap v10713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:54 smithi116 ceph-mon[35411]: pgmap v10713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:56 smithi114 ceph-mon[30820]: pgmap v10714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:56 smithi116 ceph-mon[35411]: pgmap v10714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:21:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:21:58 smithi114 ceph-mon[30820]: pgmap v10715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:21:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:21:58 smithi116 ceph-mon[35411]: pgmap v10715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:00 smithi114 ceph-mon[30820]: pgmap v10716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:00 smithi116 ceph-mon[35411]: pgmap v10716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:02 smithi114 ceph-mon[30820]: pgmap v10717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:02 smithi116 ceph-mon[35411]: pgmap v10717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:04 smithi114 ceph-mon[30820]: pgmap v10718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:04 smithi116 ceph-mon[35411]: pgmap v10718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:06 smithi114 ceph-mon[30820]: pgmap v10719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:06 smithi116 ceph-mon[35411]: pgmap v10719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:08 smithi114 ceph-mon[30820]: pgmap v10720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:08 smithi116 ceph-mon[35411]: pgmap v10720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:10 smithi114 ceph-mon[30820]: pgmap v10721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:10 smithi116 ceph-mon[35411]: pgmap v10721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:12 smithi114 ceph-mon[30820]: pgmap v10722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:12 smithi116 ceph-mon[35411]: pgmap v10722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:14 smithi114 ceph-mon[30820]: pgmap v10723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:14 smithi116 ceph-mon[35411]: pgmap v10723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:16.068 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:22:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:22:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:16 smithi114 ceph-mon[30820]: pgmap v10724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:16 smithi116 ceph-mon[35411]: pgmap v10724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-mon[30820]: pgmap v10725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:19.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:22:18.773+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:22:19.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:22:19.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:22:18 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:22:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:18 smithi116 ceph-mon[35411]: pgmap v10725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:19 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:22:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:22:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:22:20.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:22:20.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:22:20.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:19 smithi114 ceph-mon[30820]: pgmap v10726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:22:20.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:19 smithi114 ceph-mon[30820]: pgmap v10727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:22:20.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:22:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:19 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:22:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:22:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:22:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:22:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:22:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:22:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:22:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:22:20.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:19 smithi116 ceph-mon[35411]: pgmap v10726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:19 smithi116 ceph-mon[35411]: pgmap v10727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:22:20.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:22:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:21 smithi114 ceph-mon[30820]: pgmap v10728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:22:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:21 smithi116 ceph-mon[35411]: pgmap v10728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:22:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:23 smithi114 ceph-mon[30820]: pgmap v10729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:22:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:23 smithi116 ceph-mon[35411]: pgmap v10729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:22:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:25 smithi114 ceph-mon[30820]: pgmap v10730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:22:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:25 smithi116 ceph-mon[35411]: pgmap v10730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:22:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:27 smithi114 ceph-mon[30820]: pgmap v10731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:22:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:27 smithi116 ceph-mon[35411]: pgmap v10731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:22:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:29 smithi114 ceph-mon[30820]: pgmap v10732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:22:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:29 smithi116 ceph-mon[35411]: pgmap v10732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:22:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:31 smithi114 ceph-mon[30820]: pgmap v10733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:32.178 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:31 smithi116 ceph-mon[35411]: pgmap v10733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:33 smithi114 ceph-mon[30820]: pgmap v10734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:33 smithi116 ceph-mon[35411]: pgmap v10734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:35 smithi114 ceph-mon[30820]: pgmap v10735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:35 smithi116 ceph-mon[35411]: pgmap v10735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:37 smithi114 ceph-mon[30820]: pgmap v10736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:37 smithi116 ceph-mon[35411]: pgmap v10736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:39 smithi114 ceph-mon[30820]: pgmap v10737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:39 smithi116 ceph-mon[35411]: pgmap v10737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:41 smithi114 ceph-mon[30820]: pgmap v10738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:41 smithi116 ceph-mon[35411]: pgmap v10738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:43 smithi114 ceph-mon[30820]: pgmap v10739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:43 smithi116 ceph-mon[35411]: pgmap v10739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:45 smithi114 ceph-mon[30820]: pgmap v10740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:45 smithi116 ceph-mon[35411]: pgmap v10740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:47 smithi114 ceph-mon[30820]: pgmap v10741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:47 smithi116 ceph-mon[35411]: pgmap v10741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:49 smithi114 ceph-mon[30820]: pgmap v10742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:49 smithi116 ceph-mon[35411]: pgmap v10742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:51 smithi114 ceph-mon[30820]: pgmap v10743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:51 smithi116 ceph-mon[35411]: pgmap v10743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:22:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:53 smithi114 ceph-mon[30820]: pgmap v10744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:53 smithi116 ceph-mon[35411]: pgmap v10744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:22:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:55 smithi114 ceph-mon[30820]: pgmap v10745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:55 smithi116 ceph-mon[35411]: pgmap v10745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:22:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:57 smithi114 ceph-mon[30820]: pgmap v10746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:22:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:57 smithi116 ceph-mon[35411]: pgmap v10746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:22:59 smithi114 ceph-mon[30820]: pgmap v10747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:22:59 smithi116 ceph-mon[35411]: pgmap v10747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:01 smithi114 ceph-mon[30820]: pgmap v10748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:01 smithi116 ceph-mon[35411]: pgmap v10748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:03 smithi116 ceph-mon[35411]: pgmap v10749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:03 smithi114 ceph-mon[30820]: pgmap v10749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:05 smithi116 ceph-mon[35411]: pgmap v10750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:23:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:05 smithi114 ceph-mon[30820]: pgmap v10750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:23:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:07 smithi116 ceph-mon[35411]: pgmap v10751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:07 smithi114 ceph-mon[30820]: pgmap v10751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:09 smithi114 ceph-mon[30820]: pgmap v10752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:09 smithi116 ceph-mon[35411]: pgmap v10752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:11 smithi114 ceph-mon[30820]: pgmap v10753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:11 smithi116 ceph-mon[35411]: pgmap v10753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:13 smithi114 ceph-mon[30820]: pgmap v10754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:13 smithi116 ceph-mon[35411]: pgmap v10754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:15 smithi114 ceph-mon[30820]: pgmap v10755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:23:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:15 smithi116 ceph-mon[35411]: pgmap v10755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:23:18.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:17 smithi114 ceph-mon[30820]: pgmap v10756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:17 smithi116 ceph-mon[35411]: pgmap v10756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:19.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:23:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:23:20.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:19 smithi114 ceph-mon[30820]: pgmap v10757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:19 smithi116 ceph-mon[35411]: pgmap v10757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:21 smithi114 ceph-mon[30820]: pgmap v10758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:22.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:23:22.132+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:23:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:23:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:23:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:23:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:23:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:23:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:23:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:23:22.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:23:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:23:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:23:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:23:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:23:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:23:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:23:22.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:23:22.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:23:22.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:23:22.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:23:22.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:23:22.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:23:22.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:23:22.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:23:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:21 smithi116 ceph-mon[35411]: pgmap v10758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:23:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:23:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:23:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:23:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:23:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:23:24.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:23 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:23:24.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:23:24.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:23:24.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:23:24.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:23:24.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:23 smithi114 ceph-mon[30820]: pgmap v10759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:23:24.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:23 smithi114 ceph-mon[30820]: pgmap v10760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:23 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:23:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:23:24.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:23:24.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:23 smithi116 ceph-mon[35411]: pgmap v10759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:23:24.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:23 smithi116 ceph-mon[35411]: pgmap v10760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:23:26.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:25 smithi116 ceph-mon[35411]: pgmap v10761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:23:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:25 smithi114 ceph-mon[30820]: pgmap v10761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:23:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:27 smithi114 ceph-mon[30820]: pgmap v10762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:23:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:27 smithi116 ceph-mon[35411]: pgmap v10762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:23:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:29 smithi114 ceph-mon[30820]: pgmap v10763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:23:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:29 smithi116 ceph-mon[35411]: pgmap v10763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:23:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:31 smithi114 ceph-mon[30820]: pgmap v10764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:23:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:31 smithi116 ceph-mon[35411]: pgmap v10764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:23:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:33 smithi114 ceph-mon[30820]: pgmap v10765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:23:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:33 smithi116 ceph-mon[35411]: pgmap v10765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:23:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:35 smithi114 ceph-mon[30820]: pgmap v10766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:35 smithi116 ceph-mon[35411]: pgmap v10766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:37 smithi114 ceph-mon[30820]: pgmap v10767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:37 smithi116 ceph-mon[35411]: pgmap v10767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:39 smithi114 ceph-mon[30820]: pgmap v10768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:39 smithi116 ceph-mon[35411]: pgmap v10768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:41 smithi114 ceph-mon[30820]: pgmap v10769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:23:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:41 smithi116 ceph-mon[35411]: pgmap v10769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:23:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:43 smithi114 ceph-mon[30820]: pgmap v10770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:43 smithi116 ceph-mon[35411]: pgmap v10770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:45 smithi114 ceph-mon[30820]: pgmap v10771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:45 smithi116 ceph-mon[35411]: pgmap v10771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:48 smithi114 ceph-mon[30820]: pgmap v10772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:48 smithi116 ceph-mon[35411]: pgmap v10772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:49 smithi114 ceph-mon[30820]: pgmap v10773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:49 smithi116 ceph-mon[35411]: pgmap v10773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:51 smithi114 ceph-mon[30820]: pgmap v10774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:23:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:51 smithi116 ceph-mon[35411]: pgmap v10774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:23:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:53 smithi114 ceph-mon[30820]: pgmap v10775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:53 smithi116 ceph-mon[35411]: pgmap v10775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:23:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:55 smithi114 ceph-mon[30820]: pgmap v10776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:55 smithi116 ceph-mon[35411]: pgmap v10776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:57.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:57 smithi116 ceph-mon[35411]: pgmap v10777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:57 smithi114 ceph-mon[30820]: pgmap v10777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:23:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:23:59 smithi114 ceph-mon[30820]: pgmap v10778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:23:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:23:59 smithi116 ceph-mon[35411]: pgmap v10778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:01 smithi114 ceph-mon[30820]: pgmap v10779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:01 smithi116 ceph-mon[35411]: pgmap v10779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:03 smithi114 ceph-mon[30820]: pgmap v10780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:03 smithi116 ceph-mon[35411]: pgmap v10780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:05 smithi114 ceph-mon[30820]: pgmap v10781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:05 smithi116 ceph-mon[35411]: pgmap v10781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:07 smithi114 ceph-mon[30820]: pgmap v10782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:07 smithi116 ceph-mon[35411]: pgmap v10782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:09 smithi114 ceph-mon[30820]: pgmap v10783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:09 smithi116 ceph-mon[35411]: pgmap v10783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:11 smithi114 ceph-mon[30820]: pgmap v10784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:11 smithi116 ceph-mon[35411]: pgmap v10784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:13 smithi114 ceph-mon[30820]: pgmap v10785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:13 smithi116 ceph-mon[35411]: pgmap v10785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:15 smithi114 ceph-mon[30820]: pgmap v10786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:15 smithi116 ceph-mon[35411]: pgmap v10786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:17 smithi114 ceph-mon[30820]: pgmap v10787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:17 smithi116 ceph-mon[35411]: pgmap v10787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:19 smithi114 ceph-mon[30820]: pgmap v10788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:19 smithi116 ceph-mon[35411]: pgmap v10788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:21 smithi114 ceph-mon[30820]: pgmap v10789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:21 smithi116 ceph-mon[35411]: pgmap v10789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:23.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:23 smithi116 ceph-mon[35411]: pgmap v10790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:23.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:24:23.604 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:23 smithi114 ceph-mon[30820]: pgmap v10790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:23.604 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:24:25.576 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:25 smithi116 ceph-mon[35411]: pgmap v10791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:25.596 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-mon[30820]: pgmap v10791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:25.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:24:25.493+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:24:25.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:24:25.596 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:24:25.597 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:24:25.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:24:25.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:24:25.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:24:25.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:24:25.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:24:25.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:24:25.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:24:25.598 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:24:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:24:26.534 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:24:26.534 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:24:26.534 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:24:26.534 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:24:26.534 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:24:26.534 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:24:26.535 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:24:26.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:24:26.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:24:26.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:24:26.536 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:24:26.536 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:26.536 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:24:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:24:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:24:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:24:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:24:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:24:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:24:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:24:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:24:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:24:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:24:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:27.533 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:27 smithi114 ceph-mon[30820]: pgmap v10792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:27 smithi116 ceph-mon[35411]: pgmap v10792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:24:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:24:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:28 smithi114 ceph-mon[30820]: pgmap v10793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T12:24:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:28 smithi114 ceph-mon[30820]: pgmap v10794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:24:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:24:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:28 smithi116 ceph-mon[35411]: pgmap v10793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T12:24:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:28 smithi116 ceph-mon[35411]: pgmap v10794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:24:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:30 smithi114 ceph-mon[30820]: pgmap v10795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:30 smithi116 ceph-mon[35411]: pgmap v10795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:32 smithi114 ceph-mon[30820]: pgmap v10796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:32 smithi116 ceph-mon[35411]: pgmap v10796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:34 smithi114 ceph-mon[30820]: pgmap v10797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:34 smithi116 ceph-mon[35411]: pgmap v10797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:36 smithi114 ceph-mon[30820]: pgmap v10798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:36 smithi116 ceph-mon[35411]: pgmap v10798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:24:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:38 smithi114 ceph-mon[30820]: pgmap v10799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:24:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:38 smithi116 ceph-mon[35411]: pgmap v10799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:24:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:40 smithi114 ceph-mon[30820]: pgmap v10800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:40 smithi116 ceph-mon[35411]: pgmap v10800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:42 smithi114 ceph-mon[30820]: pgmap v10801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:42 smithi116 ceph-mon[35411]: pgmap v10801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:44 smithi114 ceph-mon[30820]: pgmap v10802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:44 smithi116 ceph-mon[35411]: pgmap v10802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:46 smithi114 ceph-mon[30820]: pgmap v10803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:46.465 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:46 smithi116 ceph-mon[35411]: pgmap v10803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:48 smithi114 ceph-mon[30820]: pgmap v10804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:48 smithi116 ceph-mon[35411]: pgmap v10804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:50 smithi114 ceph-mon[30820]: pgmap v10805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:50 smithi116 ceph-mon[35411]: pgmap v10805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:24:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:52 smithi114 ceph-mon[30820]: pgmap v10806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:52 smithi116 ceph-mon[35411]: pgmap v10806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:24:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:54 smithi114 ceph-mon[30820]: pgmap v10807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:54 smithi116 ceph-mon[35411]: pgmap v10807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:56 smithi114 ceph-mon[30820]: pgmap v10808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:56 smithi116 ceph-mon[35411]: pgmap v10808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:24:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:24:58 smithi114 ceph-mon[30820]: pgmap v10809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:24:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:24:58 smithi116 ceph-mon[35411]: pgmap v10809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:00 smithi114 ceph-mon[30820]: pgmap v10810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:00 smithi116 ceph-mon[35411]: pgmap v10810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:02 smithi114 ceph-mon[30820]: pgmap v10811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:25:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:02 smithi116 ceph-mon[35411]: pgmap v10811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:25:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:04 smithi114 ceph-mon[30820]: pgmap v10812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:04 smithi116 ceph-mon[35411]: pgmap v10812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:06 smithi114 ceph-mon[30820]: pgmap v10813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:25:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:06 smithi116 ceph-mon[35411]: pgmap v10813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:25:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:08 smithi114 ceph-mon[30820]: pgmap v10814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:08 smithi116 ceph-mon[35411]: pgmap v10814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:10 smithi114 ceph-mon[30820]: pgmap v10815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:10 smithi116 ceph-mon[35411]: pgmap v10815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:12 smithi114 ceph-mon[30820]: pgmap v10816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:25:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:12 smithi116 ceph-mon[35411]: pgmap v10816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:25:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:14 smithi114 ceph-mon[30820]: pgmap v10817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:14 smithi116 ceph-mon[35411]: pgmap v10817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:16 smithi114 ceph-mon[30820]: pgmap v10818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:25:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:16 smithi116 ceph-mon[35411]: pgmap v10818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:25:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:18 smithi114 ceph-mon[30820]: pgmap v10819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:18 smithi116 ceph-mon[35411]: pgmap v10819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:20 smithi114 ceph-mon[30820]: pgmap v10820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:20 smithi116 ceph-mon[35411]: pgmap v10820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:22 smithi114 ceph-mon[30820]: pgmap v10821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:25:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:22 smithi116 ceph-mon[35411]: pgmap v10821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:25:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:24 smithi114 ceph-mon[30820]: pgmap v10822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:24 smithi116 ceph-mon[35411]: pgmap v10822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:26 smithi114 ceph-mon[30820]: pgmap v10823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:25:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:26 smithi116 ceph-mon[35411]: pgmap v10823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:25:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:28 smithi114 ceph-mon[30820]: pgmap v10824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:28 smithi116 ceph-mon[35411]: pgmap v10824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:29 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:25:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:25:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:25:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:29 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:25:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:30 smithi114 ceph-mon[30820]: pgmap v10825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:30 smithi116 ceph-mon[35411]: pgmap v10825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:25:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:25:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:25:31.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:25:31.284+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:25:31.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:25:31.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:25:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:25:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:25:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:25:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:25:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:32 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:25:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:25:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:25:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:25:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:25:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:25:32.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:25:32.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:25:32.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:25:32.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:25:32.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:25:32.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:25:32.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:32 smithi114 ceph-mon[30820]: pgmap v10826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:25:32.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:32 smithi114 ceph-mon[30820]: pgmap v10827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:25:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:32 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:25:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:25:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:25:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:25:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:25:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:25:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:25:32.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:25:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:25:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:25:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:25:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:32 smithi116 ceph-mon[35411]: pgmap v10826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:25:32.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:32 smithi116 ceph-mon[35411]: pgmap v10827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:25:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:34 smithi114 ceph-mon[30820]: pgmap v10828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:25:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:34 smithi116 ceph-mon[35411]: pgmap v10828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:25:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:36 smithi114 ceph-mon[30820]: pgmap v10829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:25:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:36 smithi116 ceph-mon[35411]: pgmap v10829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:25:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:38 smithi114 ceph-mon[30820]: pgmap v10830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:25:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:38 smithi116 ceph-mon[35411]: pgmap v10830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:25:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:40 smithi114 ceph-mon[30820]: pgmap v10831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:25:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:40 smithi116 ceph-mon[35411]: pgmap v10831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:25:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:42 smithi114 ceph-mon[30820]: pgmap v10832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:25:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:42 smithi116 ceph-mon[35411]: pgmap v10832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:25:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:44 smithi114 ceph-mon[30820]: pgmap v10833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:44 smithi116 ceph-mon[35411]: pgmap v10833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:46 smithi114 ceph-mon[30820]: pgmap v10834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:25:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:46 smithi116 ceph-mon[35411]: pgmap v10834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:25:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:48 smithi114 ceph-mon[30820]: pgmap v10835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:25:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:48 smithi116 ceph-mon[35411]: pgmap v10835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:25:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:50 smithi114 ceph-mon[30820]: pgmap v10836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:50 smithi116 ceph-mon[35411]: pgmap v10836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:25:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:52 smithi114 ceph-mon[30820]: pgmap v10837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:25:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:52 smithi116 ceph-mon[35411]: pgmap v10837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:25:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:54 smithi114 ceph-mon[30820]: pgmap v10838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:54 smithi116 ceph-mon[35411]: pgmap v10838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:25:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:56 smithi114 ceph-mon[30820]: pgmap v10839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:25:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:56 smithi116 ceph-mon[35411]: pgmap v10839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:25:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:25:58 smithi114 ceph-mon[30820]: pgmap v10840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:25:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:25:58 smithi116 ceph-mon[35411]: pgmap v10840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:26:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:00 smithi114 ceph-mon[30820]: pgmap v10841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:00 smithi116 ceph-mon[35411]: pgmap v10841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:02 smithi114 ceph-mon[30820]: pgmap v10842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:02 smithi116 ceph-mon[35411]: pgmap v10842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:04 smithi114 ceph-mon[30820]: pgmap v10843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:26:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:04 smithi116 ceph-mon[35411]: pgmap v10843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:26:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:06 smithi114 ceph-mon[30820]: pgmap v10844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:26:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:06 smithi116 ceph-mon[35411]: pgmap v10844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:26:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:08 smithi114 ceph-mon[30820]: pgmap v10845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:26:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:08 smithi116 ceph-mon[35411]: pgmap v10845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:26:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:10 smithi116 ceph-mon[35411]: pgmap v10846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:10 smithi114 ceph-mon[30820]: pgmap v10846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:12 smithi116 ceph-mon[35411]: pgmap v10847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:12 smithi114 ceph-mon[30820]: pgmap v10847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:14 smithi116 ceph-mon[35411]: pgmap v10848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:26:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:14 smithi114 ceph-mon[30820]: pgmap v10848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:26:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:16 smithi114 ceph-mon[30820]: pgmap v10849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:26:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:16 smithi116 ceph-mon[35411]: pgmap v10849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:26:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:18 smithi114 ceph-mon[30820]: pgmap v10850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T12:26:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:18 smithi116 ceph-mon[35411]: pgmap v10850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T12:26:20.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:20 smithi114 ceph-mon[30820]: pgmap v10851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:26:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:20 smithi116 ceph-mon[35411]: pgmap v10851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:26:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:22 smithi114 ceph-mon[30820]: pgmap v10852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:22 smithi116 ceph-mon[35411]: pgmap v10852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:24 smithi114 ceph-mon[30820]: pgmap v10853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:26:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:24 smithi116 ceph-mon[35411]: pgmap v10853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:26:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:26 smithi114 ceph-mon[30820]: pgmap v10854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:26:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:26 smithi116 ceph-mon[35411]: pgmap v10854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:26:28.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:28 smithi116 ceph-mon[35411]: pgmap v10855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:26:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:28 smithi114 ceph-mon[30820]: pgmap v10855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:26:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:30 smithi114 ceph-mon[30820]: pgmap v10856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:30 smithi116 ceph-mon[35411]: pgmap v10856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:26:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-mon[30820]: pgmap v10857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:26:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:26:32.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:26:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:26:32.276+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:26:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:26:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:26:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:26:32.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:26:32.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:26:32.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:26:32.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:26:32.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:26:32.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:26:32.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:26:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:26:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:32 smithi116 ceph-mon[35411]: pgmap v10857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:26:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:26:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:26:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:33 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:26:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:26:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:26:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:26:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:26:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:26:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:26:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:26:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:26:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:26:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:26:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:33 smithi114 ceph-mon[30820]: pgmap v10858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:26:33.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:33 smithi114 ceph-mon[30820]: pgmap v10859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:26:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:33 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:26:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:26:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:26:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:26:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:26:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:26:33.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:26:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:26:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:26:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:26:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:26:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:33 smithi116 ceph-mon[35411]: pgmap v10858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:26:33.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:33 smithi116 ceph-mon[35411]: pgmap v10859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:26:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:35 smithi114 ceph-mon[30820]: pgmap v10860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T12:26:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:35 smithi116 ceph-mon[35411]: pgmap v10860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T12:26:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:37 smithi114 ceph-mon[30820]: pgmap v10861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:26:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:37 smithi116 ceph-mon[35411]: pgmap v10861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:26:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:39 smithi114 ceph-mon[30820]: pgmap v10862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T12:26:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:39 smithi116 ceph-mon[35411]: pgmap v10862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T12:26:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:41 smithi114 ceph-mon[30820]: pgmap v10863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T12:26:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:41 smithi116 ceph-mon[35411]: pgmap v10863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T12:26:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:43 smithi114 ceph-mon[30820]: pgmap v10864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:26:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:43 smithi116 ceph-mon[35411]: pgmap v10864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:26:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:45 smithi114 ceph-mon[30820]: pgmap v10865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:45 smithi116 ceph-mon[35411]: pgmap v10865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:47 smithi114 ceph-mon[30820]: pgmap v10866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:47 smithi116 ceph-mon[35411]: pgmap v10866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:49 smithi114 ceph-mon[30820]: pgmap v10867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:26:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:49 smithi116 ceph-mon[35411]: pgmap v10867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:26:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:51 smithi114 ceph-mon[30820]: pgmap v10868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:26:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:51 smithi116 ceph-mon[35411]: pgmap v10868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:26:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:53 smithi114 ceph-mon[30820]: pgmap v10869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:26:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:53 smithi116 ceph-mon[35411]: pgmap v10869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:26:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:55 smithi114 ceph-mon[30820]: pgmap v10870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:55 smithi116 ceph-mon[35411]: pgmap v10870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:26:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:57 smithi114 ceph-mon[30820]: pgmap v10871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:57 smithi116 ceph-mon[35411]: pgmap v10871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:26:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:26:59 smithi114 ceph-mon[30820]: pgmap v10872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:26:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:26:59 smithi116 ceph-mon[35411]: pgmap v10872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:01 smithi114 ceph-mon[30820]: pgmap v10873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:27:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:01 smithi116 ceph-mon[35411]: pgmap v10873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:27:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:03 smithi114 ceph-mon[30820]: pgmap v10874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:27:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:03 smithi116 ceph-mon[35411]: pgmap v10874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:27:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:05 smithi114 ceph-mon[30820]: pgmap v10875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:05 smithi116 ceph-mon[35411]: pgmap v10875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:07 smithi114 ceph-mon[30820]: pgmap v10876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:27:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:07 smithi116 ceph-mon[35411]: pgmap v10876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:27:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:09 smithi114 ceph-mon[30820]: pgmap v10877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:09 smithi116 ceph-mon[35411]: pgmap v10877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:11 smithi114 ceph-mon[30820]: pgmap v10878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:27:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:11 smithi116 ceph-mon[35411]: pgmap v10878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:27:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:13 smithi114 ceph-mon[30820]: pgmap v10879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:27:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:13 smithi116 ceph-mon[35411]: pgmap v10879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:27:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:15 smithi114 ceph-mon[30820]: pgmap v10880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:15 smithi116 ceph-mon[35411]: pgmap v10880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:17 smithi114 ceph-mon[30820]: pgmap v10881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:27:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:17 smithi116 ceph-mon[35411]: pgmap v10881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:27:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:19 smithi114 ceph-mon[30820]: pgmap v10882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:19 smithi116 ceph-mon[35411]: pgmap v10882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:21 smithi114 ceph-mon[30820]: pgmap v10883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:27:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:21 smithi116 ceph-mon[35411]: pgmap v10883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:27:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:23 smithi114 ceph-mon[30820]: pgmap v10884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:27:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:23 smithi116 ceph-mon[35411]: pgmap v10884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:27:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:25 smithi114 ceph-mon[30820]: pgmap v10885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:25 smithi116 ceph-mon[35411]: pgmap v10885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:27 smithi114 ceph-mon[30820]: pgmap v10886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:27:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:27 smithi116 ceph-mon[35411]: pgmap v10886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:27:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:29 smithi114 ceph-mon[30820]: pgmap v10887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:29 smithi116 ceph-mon[35411]: pgmap v10887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:31 smithi114 ceph-mon[30820]: pgmap v10888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:27:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:31 smithi116 ceph-mon[35411]: pgmap v10888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:27:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:27:32.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:27:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:33 smithi114 ceph-mon[30820]: pgmap v10889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:27:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:33 smithi116 ceph-mon[35411]: pgmap v10889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:27:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-mon[30820]: pgmap v10890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:27:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:27:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:27:35.572+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:27:35.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:27:35.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:27:35.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:27:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:27:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:35 smithi116 ceph-mon[35411]: pgmap v10890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:27:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:27:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:27:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:27:36.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:36 smithi114 ceph-mon[30820]: pgmap v10891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:27:36.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:36 smithi114 ceph-mon[30820]: pgmap v10892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:27:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:27:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:27:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:27:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:27:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:27:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:27:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:27:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:27:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:27:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:27:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:27:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:27:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:36 smithi116 ceph-mon[35411]: pgmap v10891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:27:36.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:36 smithi116 ceph-mon[35411]: pgmap v10892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:27:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:38 smithi114 ceph-mon[30820]: pgmap v10893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:27:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:38 smithi116 ceph-mon[35411]: pgmap v10893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:27:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:40 smithi114 ceph-mon[30820]: pgmap v10894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T12:27:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:40 smithi116 ceph-mon[35411]: pgmap v10894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T12:27:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:42 smithi114 ceph-mon[30820]: pgmap v10895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:27:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:42 smithi116 ceph-mon[35411]: pgmap v10895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:27:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:44 smithi114 ceph-mon[30820]: pgmap v10896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:27:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:44 smithi116 ceph-mon[35411]: pgmap v10896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:27:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:46 smithi114 ceph-mon[30820]: pgmap v10897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:27:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:46 smithi116 ceph-mon[35411]: pgmap v10897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:27:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:48 smithi114 ceph-mon[30820]: pgmap v10898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:48 smithi116 ceph-mon[35411]: pgmap v10898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:50 smithi114 ceph-mon[30820]: pgmap v10899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:50 smithi116 ceph-mon[35411]: pgmap v10899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:27:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:52 smithi114 ceph-mon[30820]: pgmap v10900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:27:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:52 smithi116 ceph-mon[35411]: pgmap v10900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:27:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:54 smithi114 ceph-mon[30820]: pgmap v10901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:54 smithi116 ceph-mon[35411]: pgmap v10901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:56 smithi114 ceph-mon[30820]: pgmap v10902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:27:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:56 smithi116 ceph-mon[35411]: pgmap v10902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:27:58.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:27:58 smithi114 ceph-mon[30820]: pgmap v10903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:27:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:27:58 smithi116 ceph-mon[35411]: pgmap v10903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:00 smithi114 ceph-mon[30820]: pgmap v10904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:28:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:00 smithi116 ceph-mon[35411]: pgmap v10904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:28:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:02 smithi114 ceph-mon[30820]: pgmap v10905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:02 smithi116 ceph-mon[35411]: pgmap v10905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:04 smithi114 ceph-mon[30820]: pgmap v10906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:04 smithi116 ceph-mon[35411]: pgmap v10906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:06 smithi114 ceph-mon[30820]: pgmap v10907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:06 smithi116 ceph-mon[35411]: pgmap v10907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:08 smithi114 ceph-mon[30820]: pgmap v10908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:08 smithi116 ceph-mon[35411]: pgmap v10908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:10 smithi116 ceph-mon[35411]: pgmap v10909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:28:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:10 smithi114 ceph-mon[30820]: pgmap v10909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:28:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:12 smithi116 ceph-mon[35411]: pgmap v10910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:12 smithi114 ceph-mon[30820]: pgmap v10910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:14 smithi116 ceph-mon[35411]: pgmap v10911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:14 smithi114 ceph-mon[30820]: pgmap v10911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:16 smithi114 ceph-mon[30820]: pgmap v10912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:16 smithi116 ceph-mon[35411]: pgmap v10912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:18 smithi114 ceph-mon[30820]: pgmap v10913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:18 smithi116 ceph-mon[35411]: pgmap v10913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:20 smithi114 ceph-mon[30820]: pgmap v10914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:28:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:20 smithi116 ceph-mon[35411]: pgmap v10914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:28:22.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:22 smithi114 ceph-mon[30820]: pgmap v10915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:22 smithi116 ceph-mon[35411]: pgmap v10915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:24 smithi114 ceph-mon[30820]: pgmap v10916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:24 smithi116 ceph-mon[35411]: pgmap v10916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:26 smithi114 ceph-mon[30820]: pgmap v10917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:26 smithi116 ceph-mon[35411]: pgmap v10917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:28 smithi114 ceph-mon[30820]: pgmap v10918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:28:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:28 smithi116 ceph-mon[35411]: pgmap v10918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:28:30.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:30 smithi116 ceph-mon[35411]: pgmap v10919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:28:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:30 smithi114 ceph-mon[30820]: pgmap v10919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:28:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:32 smithi114 ceph-mon[30820]: pgmap v10920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:32 smithi116 ceph-mon[35411]: pgmap v10920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:34 smithi114 ceph-mon[30820]: pgmap v10921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:28:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:34 smithi116 ceph-mon[35411]: pgmap v10921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:28:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:28:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:28:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:36 smithi114 ceph-mon[30820]: pgmap v10922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:36 smithi116 ceph-mon[35411]: pgmap v10922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-mon[30820]: pgmap v10923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:38 smithi116 ceph-mon[35411]: pgmap v10923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:39 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:28:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:28:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:28:39.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:39 smithi114 ceph-mon[30820]: pgmap v10924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:39 smithi114 ceph-mon[30820]: pgmap v10925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:28:39.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:28:38.932+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:28:39.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:28:39.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:28:39.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:28:38 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:28:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:39 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:28:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:28:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:28:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:28:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:28:39.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:39 smithi116 ceph-mon[35411]: pgmap v10924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:39 smithi116 ceph-mon[35411]: pgmap v10925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:28:39.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:28:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:41 smithi114 ceph-mon[30820]: pgmap v10926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:28:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:41 smithi116 ceph-mon[35411]: pgmap v10926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:28:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:43 smithi114 ceph-mon[30820]: pgmap v10927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:28:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:43 smithi116 ceph-mon[35411]: pgmap v10927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:28:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:46 smithi114 ceph-mon[30820]: pgmap v10928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:28:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:46 smithi116 ceph-mon[35411]: pgmap v10928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:28:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:48 smithi114 ceph-mon[30820]: pgmap v10929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:28:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:48 smithi116 ceph-mon[35411]: pgmap v10929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:28:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:50 smithi114 ceph-mon[30820]: pgmap v10930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:28:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:50 smithi116 ceph-mon[35411]: pgmap v10930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:28:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:52 smithi114 ceph-mon[30820]: pgmap v10931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:52 smithi116 ceph-mon[35411]: pgmap v10931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:28:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:54 smithi114 ceph-mon[30820]: pgmap v10932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:54 smithi116 ceph-mon[35411]: pgmap v10932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:28:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:56 smithi114 ceph-mon[30820]: pgmap v10933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:28:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:56 smithi116 ceph-mon[35411]: pgmap v10933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:28:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:28:58 smithi114 ceph-mon[30820]: pgmap v10934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:28:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:28:58 smithi116 ceph-mon[35411]: pgmap v10934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:00 smithi114 ceph-mon[30820]: pgmap v10935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:00 smithi116 ceph-mon[35411]: pgmap v10935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:02 smithi114 ceph-mon[30820]: pgmap v10936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:02 smithi116 ceph-mon[35411]: pgmap v10936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:04 smithi114 ceph-mon[30820]: pgmap v10937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:04 smithi116 ceph-mon[35411]: pgmap v10937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:06 smithi114 ceph-mon[30820]: pgmap v10938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:29:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:06 smithi116 ceph-mon[35411]: pgmap v10938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:29:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:08 smithi114 ceph-mon[30820]: pgmap v10939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:08 smithi116 ceph-mon[35411]: pgmap v10939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:10 smithi114 ceph-mon[30820]: pgmap v10940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:10 smithi116 ceph-mon[35411]: pgmap v10940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:12 smithi114 ceph-mon[30820]: pgmap v10941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:12 smithi116 ceph-mon[35411]: pgmap v10941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:14 smithi114 ceph-mon[30820]: pgmap v10942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:14 smithi116 ceph-mon[35411]: pgmap v10942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:16 smithi114 ceph-mon[30820]: pgmap v10943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:29:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:16 smithi116 ceph-mon[35411]: pgmap v10943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:29:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:17 smithi114 ceph-mon[30820]: pgmap v10944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:17 smithi116 ceph-mon[35411]: pgmap v10944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:20 smithi114 ceph-mon[30820]: pgmap v10945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:20 smithi116 ceph-mon[35411]: pgmap v10945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:22 smithi114 ceph-mon[30820]: pgmap v10946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:22 smithi116 ceph-mon[35411]: pgmap v10946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:24 smithi114 ceph-mon[30820]: pgmap v10947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:24 smithi116 ceph-mon[35411]: pgmap v10947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:26 smithi114 ceph-mon[30820]: pgmap v10948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:29:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:26 smithi116 ceph-mon[35411]: pgmap v10948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:29:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:28 smithi114 ceph-mon[30820]: pgmap v10949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:28 smithi116 ceph-mon[35411]: pgmap v10949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:29 smithi114 ceph-mon[30820]: pgmap v10950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:29:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:29 smithi116 ceph-mon[35411]: pgmap v10950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:29:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:32 smithi114 ceph-mon[30820]: pgmap v10951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:32 smithi116 ceph-mon[35411]: pgmap v10951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:34 smithi114 ceph-mon[30820]: pgmap v10952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:29:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:34 smithi116 ceph-mon[35411]: pgmap v10952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:29:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:36 smithi114 ceph-mon[30820]: pgmap v10953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:29:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:36 smithi116 ceph-mon[35411]: pgmap v10953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:29:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:38 smithi114 ceph-mon[30820]: pgmap v10954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:38 smithi116 ceph-mon[35411]: pgmap v10954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:29:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:40 smithi114 ceph-mon[30820]: pgmap v10955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:29:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:40.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:40.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:40 smithi116 ceph-mon[35411]: pgmap v10955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:29:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-mon[30820]: pgmap v10956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:42.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:29:42.229+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:29:42.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:29:42.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:29:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:29:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:42 smithi116 ceph-mon[35411]: pgmap v10956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:29:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:29:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:29:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:29:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:29:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:44 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:29:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:29:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:29:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:29:44.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:44 smithi114 ceph-mon[30820]: pgmap v10957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:29:44.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:44 smithi114 ceph-mon[30820]: pgmap v10958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:44 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:29:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:29:44.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:29:44.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:44 smithi116 ceph-mon[35411]: pgmap v10957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:29:44.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:44 smithi116 ceph-mon[35411]: pgmap v10958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:29:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:46 smithi114 ceph-mon[30820]: pgmap v10959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:29:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:46 smithi116 ceph-mon[35411]: pgmap v10959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:29:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:48 smithi114 ceph-mon[30820]: pgmap v10960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:29:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:48 smithi116 ceph-mon[35411]: pgmap v10960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:29:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:50 smithi114 ceph-mon[30820]: pgmap v10961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:29:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:50 smithi116 ceph-mon[35411]: pgmap v10961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:29:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:52 smithi114 ceph-mon[30820]: pgmap v10962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:29:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:52 smithi116 ceph-mon[35411]: pgmap v10962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:29:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:54 smithi114 ceph-mon[30820]: pgmap v10963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:29:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:54 smithi116 ceph-mon[35411]: pgmap v10963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:29:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:56 smithi114 ceph-mon[30820]: pgmap v10964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:56 smithi116 ceph-mon[35411]: pgmap v10964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:29:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:29:58 smithi114 ceph-mon[30820]: pgmap v10965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:29:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:29:58 smithi116 ceph-mon[35411]: pgmap v10965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:00 smithi114 ceph-mon[30820]: pgmap v10966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:30:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:00 smithi116 ceph-mon[35411]: pgmap v10966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:30:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:02 smithi114 ceph-mon[30820]: pgmap v10967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:30:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:02 smithi116 ceph-mon[35411]: pgmap v10967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:30:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:03 smithi114 ceph-mon[30820]: pgmap v10968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:30:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:03 smithi116 ceph-mon[35411]: pgmap v10968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:30:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:05 smithi114 ceph-mon[30820]: pgmap v10969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:05 smithi116 ceph-mon[35411]: pgmap v10969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:07 smithi114 ceph-mon[30820]: pgmap v10970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:07 smithi116 ceph-mon[35411]: pgmap v10970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:09 smithi114 ceph-mon[30820]: pgmap v10971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:09 smithi116 ceph-mon[35411]: pgmap v10971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:11 smithi114 ceph-mon[30820]: pgmap v10972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:30:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:11 smithi116 ceph-mon[35411]: pgmap v10972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:30:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:13 smithi114 ceph-mon[30820]: pgmap v10973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:30:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:13 smithi116 ceph-mon[35411]: pgmap v10973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:30:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:15 smithi114 ceph-mon[30820]: pgmap v10974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:15 smithi116 ceph-mon[35411]: pgmap v10974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:17 smithi114 ceph-mon[30820]: pgmap v10975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:17 smithi116 ceph-mon[35411]: pgmap v10975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:19 smithi114 ceph-mon[30820]: pgmap v10976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:19 smithi116 ceph-mon[35411]: pgmap v10976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:21 smithi114 ceph-mon[30820]: pgmap v10977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:30:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:21 smithi116 ceph-mon[35411]: pgmap v10977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:30:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:23 smithi114 ceph-mon[30820]: pgmap v10978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:30:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:23 smithi116 ceph-mon[35411]: pgmap v10978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:30:25.597 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:25 smithi114 ceph-mon[30820]: pgmap v10979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:25 smithi116 ceph-mon[35411]: pgmap v10979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:27 smithi114 ceph-mon[30820]: pgmap v10980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:27 smithi116 ceph-mon[35411]: pgmap v10980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:29 smithi114 ceph-mon[30820]: pgmap v10981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:29 smithi116 ceph-mon[35411]: pgmap v10981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:31 smithi114 ceph-mon[30820]: pgmap v10982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:30:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:31 smithi116 ceph-mon[35411]: pgmap v10982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:30:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:33 smithi114 ceph-mon[30820]: pgmap v10983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:30:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:33 smithi116 ceph-mon[35411]: pgmap v10983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:30:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:35 smithi114 ceph-mon[30820]: pgmap v10984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:35 smithi116 ceph-mon[35411]: pgmap v10984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:37 smithi114 ceph-mon[30820]: pgmap v10985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:37 smithi116 ceph-mon[35411]: pgmap v10985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:30:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:39 smithi114 ceph-mon[30820]: pgmap v10986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:39 smithi116 ceph-mon[35411]: pgmap v10986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:41 smithi114 ceph-mon[30820]: pgmap v10987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:30:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:41 smithi116 ceph-mon[35411]: pgmap v10987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:30:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:30:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:30:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:43 smithi114 ceph-mon[30820]: pgmap v10988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:30:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:43 smithi116 ceph-mon[35411]: pgmap v10988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:30:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-mon[30820]: pgmap v10989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:30:45.588+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:30:45.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:30:45.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:30:45.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:30:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:30:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:45 smithi116 ceph-mon[35411]: pgmap v10989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:30:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:30:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:30:46.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:46 smithi114 ceph-mon[30820]: pgmap v10990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T12:30:46.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:46 smithi114 ceph-mon[30820]: pgmap v10991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:30:46.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:30:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:30:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:30:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:30:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:30:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:30:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:46 smithi116 ceph-mon[35411]: pgmap v10990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T12:30:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:46 smithi116 ceph-mon[35411]: pgmap v10991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:30:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:30:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:48 smithi114 ceph-mon[30820]: pgmap v10992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:30:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:48 smithi116 ceph-mon[35411]: pgmap v10992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:30:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:50 smithi114 ceph-mon[30820]: pgmap v10993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:30:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:50 smithi116 ceph-mon[35411]: pgmap v10993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:30:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:52 smithi114 ceph-mon[30820]: pgmap v10994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:30:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:52 smithi116 ceph-mon[35411]: pgmap v10994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:30:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:54 smithi114 ceph-mon[30820]: pgmap v10995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:30:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:54 smithi116 ceph-mon[35411]: pgmap v10995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:30:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:56 smithi114 ceph-mon[30820]: pgmap v10996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:30:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:56 smithi116 ceph-mon[35411]: pgmap v10996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:30:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:30:58 smithi116 ceph-mon[35411]: pgmap v10997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:30:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:30:58 smithi114 ceph-mon[30820]: pgmap v10997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:00 smithi116 ceph-mon[35411]: pgmap v10998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:00 smithi114 ceph-mon[30820]: pgmap v10998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:02 smithi116 ceph-mon[35411]: pgmap v10999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:02 smithi114 ceph-mon[30820]: pgmap v10999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:04 smithi114 ceph-mon[30820]: pgmap v11000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:04 smithi116 ceph-mon[35411]: pgmap v11000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:06 smithi114 ceph-mon[30820]: pgmap v11001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:06 smithi116 ceph-mon[35411]: pgmap v11001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:08 smithi114 ceph-mon[30820]: pgmap v11002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:08 smithi116 ceph-mon[35411]: pgmap v11002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:10 smithi114 ceph-mon[30820]: pgmap v11003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:10 smithi116 ceph-mon[35411]: pgmap v11003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:12 smithi114 ceph-mon[30820]: pgmap v11004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:12 smithi116 ceph-mon[35411]: pgmap v11004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:14 smithi114 ceph-mon[30820]: pgmap v11005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:14 smithi116 ceph-mon[35411]: pgmap v11005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:16 smithi114 ceph-mon[30820]: pgmap v11006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:16 smithi116 ceph-mon[35411]: pgmap v11006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:18 smithi114 ceph-mon[30820]: pgmap v11007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:18 smithi116 ceph-mon[35411]: pgmap v11007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:20 smithi114 ceph-mon[30820]: pgmap v11008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:20 smithi116 ceph-mon[35411]: pgmap v11008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:22 smithi114 ceph-mon[30820]: pgmap v11009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:22 smithi116 ceph-mon[35411]: pgmap v11009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:24.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:24 smithi114 ceph-mon[30820]: pgmap v11010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:24 smithi116 ceph-mon[35411]: pgmap v11010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:26 smithi114 ceph-mon[30820]: pgmap v11011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:26 smithi116 ceph-mon[35411]: pgmap v11011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:28 smithi114 ceph-mon[30820]: pgmap v11012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:28 smithi116 ceph-mon[35411]: pgmap v11012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:30 smithi114 ceph-mon[30820]: pgmap v11013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:31.242 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:30 smithi116 ceph-mon[35411]: pgmap v11013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:32.982 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:32 smithi116 ceph-mon[35411]: pgmap v11014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:32 smithi114 ceph-mon[30820]: pgmap v11014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:34 smithi114 ceph-mon[30820]: pgmap v11015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:34 smithi116 ceph-mon[35411]: pgmap v11015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:36 smithi114 ceph-mon[30820]: pgmap v11016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:36 smithi116 ceph-mon[35411]: pgmap v11016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:38 smithi114 ceph-mon[30820]: pgmap v11017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:38 smithi116 ceph-mon[35411]: pgmap v11017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:40 smithi114 ceph-mon[30820]: pgmap v11018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:40 smithi116 ceph-mon[35411]: pgmap v11018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:31:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:42 smithi114 ceph-mon[30820]: pgmap v11019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:42 smithi116 ceph-mon[35411]: pgmap v11019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:31:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:44 smithi114 ceph-mon[30820]: pgmap v11020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:44 smithi116 ceph-mon[35411]: pgmap v11020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:31:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:31:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:46 smithi114 ceph-mon[30820]: pgmap v11021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:46 smithi116 ceph-mon[35411]: pgmap v11021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:31:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-mon[30820]: pgmap v11022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:31:48.949+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:31:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:31:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:31:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:31:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:31:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:31:49.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:31:49.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:31:49.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:31:49.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:31:49.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:31:49.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:31:49.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:31:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:31:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:48 smithi116 ceph-mon[35411]: pgmap v11022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:31:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:31:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:31:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:31:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:31:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:31:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:31:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:31:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:31:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:31:50.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:31:50.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:31:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:31:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:31:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:31:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:31:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:49 smithi114 ceph-mon[30820]: pgmap v11023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:31:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:49 smithi114 ceph-mon[30820]: pgmap v11024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:31:50.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:31:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:31:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:31:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:31:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:31:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:31:50.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:49 smithi116 ceph-mon[35411]: pgmap v11023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:49 smithi116 ceph-mon[35411]: pgmap v11024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:31:50.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:31:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:52 smithi114 ceph-mon[30820]: pgmap v11025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:31:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:52 smithi116 ceph-mon[35411]: pgmap v11025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:31:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:54 smithi114 ceph-mon[30820]: pgmap v11026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:31:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:54 smithi116 ceph-mon[35411]: pgmap v11026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:31:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:56 smithi114 ceph-mon[30820]: pgmap v11027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:31:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:56 smithi116 ceph-mon[35411]: pgmap v11027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:31:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:31:58 smithi114 ceph-mon[30820]: pgmap v11028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:31:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:31:58 smithi116 ceph-mon[35411]: pgmap v11028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:32:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:00 smithi114 ceph-mon[30820]: pgmap v11029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:32:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:00 smithi116 ceph-mon[35411]: pgmap v11029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:32:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:02 smithi114 ceph-mon[30820]: pgmap v11030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:02 smithi116 ceph-mon[35411]: pgmap v11030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:04 smithi114 ceph-mon[30820]: pgmap v11031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:04 smithi116 ceph-mon[35411]: pgmap v11031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:06 smithi114 ceph-mon[30820]: pgmap v11032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:06 smithi116 ceph-mon[35411]: pgmap v11032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:08 smithi114 ceph-mon[30820]: pgmap v11033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:08 smithi116 ceph-mon[35411]: pgmap v11033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:10 smithi114 ceph-mon[30820]: pgmap v11034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:10 smithi116 ceph-mon[35411]: pgmap v11034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:12 smithi114 ceph-mon[30820]: pgmap v11035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:12 smithi116 ceph-mon[35411]: pgmap v11035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:14 smithi114 ceph-mon[30820]: pgmap v11036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:14 smithi116 ceph-mon[35411]: pgmap v11036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:16 smithi114 ceph-mon[30820]: pgmap v11037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:16 smithi116 ceph-mon[35411]: pgmap v11037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:18 smithi114 ceph-mon[30820]: pgmap v11038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:18 smithi116 ceph-mon[35411]: pgmap v11038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:20 smithi114 ceph-mon[30820]: pgmap v11039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:20 smithi116 ceph-mon[35411]: pgmap v11039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:22 smithi114 ceph-mon[30820]: pgmap v11040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:22 smithi116 ceph-mon[35411]: pgmap v11040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:24 smithi114 ceph-mon[30820]: pgmap v11041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:24 smithi116 ceph-mon[35411]: pgmap v11041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:26 smithi114 ceph-mon[30820]: pgmap v11042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:26 smithi116 ceph-mon[35411]: pgmap v11042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:28 smithi114 ceph-mon[30820]: pgmap v11043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:28.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:28 smithi116 ceph-mon[35411]: pgmap v11043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:30 smithi114 ceph-mon[30820]: pgmap v11044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:30 smithi116 ceph-mon[35411]: pgmap v11044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:32 smithi114 ceph-mon[30820]: pgmap v11045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:32 smithi116 ceph-mon[35411]: pgmap v11045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:34 smithi114 ceph-mon[30820]: pgmap v11046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:34 smithi116 ceph-mon[35411]: pgmap v11046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:36 smithi114 ceph-mon[30820]: pgmap v11047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:36 smithi116 ceph-mon[35411]: pgmap v11047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:38 smithi114 ceph-mon[30820]: pgmap v11048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:38 smithi116 ceph-mon[35411]: pgmap v11048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:39 smithi114 ceph-mon[30820]: pgmap v11049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:39 smithi116 ceph-mon[35411]: pgmap v11049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:42 smithi114 ceph-mon[30820]: pgmap v11050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:42 smithi116 ceph-mon[35411]: pgmap v11050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:44 smithi114 ceph-mon[30820]: pgmap v11051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:44 smithi116 ceph-mon[35411]: pgmap v11051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:46 smithi114 ceph-mon[30820]: pgmap v11052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:46 smithi116 ceph-mon[35411]: pgmap v11052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:32:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:48 smithi114 ceph-mon[30820]: pgmap v11053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:48 smithi116 ceph-mon[35411]: pgmap v11053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:32:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:49 smithi114 ceph-mon[30820]: pgmap v11054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:32:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:49 smithi116 ceph-mon[35411]: pgmap v11054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:32:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:32:52.311 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-mon[30820]: pgmap v11055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:52 smithi116 ceph-mon[35411]: pgmap v11055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:32:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:32:52.308+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:32:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:32:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:32:52.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:32:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:32:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:32:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:32:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:32:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:32:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:32:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:32:52.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:32:52.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:32:52.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:32:52.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:32:52.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:32:52.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:32:52.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:32:52.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:32:52.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:32:52.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:32:52.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:32:52.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:32:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:32:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:32:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:32:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:32:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:32:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:32:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:32:54.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:32:54.319 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:32:54.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:54 smithi114 ceph-mon[30820]: pgmap v11056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:32:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:54 smithi114 ceph-mon[30820]: pgmap v11057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:32:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:32:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:32:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:32:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:32:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:32:54.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:32:54.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:32:54.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:32:54.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:32:54.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:32:54.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:32:54.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:54 smithi116 ceph-mon[35411]: pgmap v11056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:32:54.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:54 smithi116 ceph-mon[35411]: pgmap v11057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:32:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:56 smithi114 ceph-mon[30820]: pgmap v11058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:32:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:56 smithi116 ceph-mon[35411]: pgmap v11058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:32:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:32:58 smithi114 ceph-mon[30820]: pgmap v11059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:32:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:32:58 smithi116 ceph-mon[35411]: pgmap v11059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:33:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:00 smithi114 ceph-mon[30820]: pgmap v11060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:33:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:00 smithi116 ceph-mon[35411]: pgmap v11060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:33:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:02 smithi114 ceph-mon[30820]: pgmap v11061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:33:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:02 smithi116 ceph-mon[35411]: pgmap v11061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:33:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:04 smithi114 ceph-mon[30820]: pgmap v11062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:33:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:04 smithi116 ceph-mon[35411]: pgmap v11062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:33:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:06 smithi114 ceph-mon[30820]: pgmap v11063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:06 smithi116 ceph-mon[35411]: pgmap v11063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:08 smithi114 ceph-mon[30820]: pgmap v11064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:08 smithi116 ceph-mon[35411]: pgmap v11064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:10 smithi114 ceph-mon[30820]: pgmap v11065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:10 smithi116 ceph-mon[35411]: pgmap v11065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:12 smithi114 ceph-mon[30820]: pgmap v11066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:12 smithi116 ceph-mon[35411]: pgmap v11066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:14 smithi114 ceph-mon[30820]: pgmap v11067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:14 smithi116 ceph-mon[35411]: pgmap v11067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:16 smithi114 ceph-mon[30820]: pgmap v11068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:16 smithi116 ceph-mon[35411]: pgmap v11068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:18 smithi114 ceph-mon[30820]: pgmap v11069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:18 smithi116 ceph-mon[35411]: pgmap v11069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:20 smithi114 ceph-mon[30820]: pgmap v11070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:20 smithi116 ceph-mon[35411]: pgmap v11070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:22 smithi114 ceph-mon[30820]: pgmap v11071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:22 smithi116 ceph-mon[35411]: pgmap v11071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:24 smithi114 ceph-mon[30820]: pgmap v11072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:24 smithi116 ceph-mon[35411]: pgmap v11072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:26 smithi114 ceph-mon[30820]: pgmap v11073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:26 smithi116 ceph-mon[35411]: pgmap v11073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:28 smithi114 ceph-mon[30820]: pgmap v11074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:28 smithi116 ceph-mon[35411]: pgmap v11074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:29.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:29 smithi114 ceph-mon[30820]: pgmap v11075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:29 smithi116 ceph-mon[35411]: pgmap v11075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:31 smithi114 ceph-mon[30820]: pgmap v11076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:31 smithi116 ceph-mon[35411]: pgmap v11076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:33 smithi114 ceph-mon[30820]: pgmap v11077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:33 smithi116 ceph-mon[35411]: pgmap v11077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:35 smithi114 ceph-mon[30820]: pgmap v11078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:35 smithi116 ceph-mon[35411]: pgmap v11078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:37 smithi114 ceph-mon[30820]: pgmap v11079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:37 smithi116 ceph-mon[35411]: pgmap v11079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:39 smithi114 ceph-mon[30820]: pgmap v11080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:39 smithi116 ceph-mon[35411]: pgmap v11080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:41 smithi114 ceph-mon[30820]: pgmap v11081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:41 smithi116 ceph-mon[35411]: pgmap v11081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:43 smithi114 ceph-mon[30820]: pgmap v11082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:43 smithi116 ceph-mon[35411]: pgmap v11082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:45 smithi114 ceph-mon[30820]: pgmap v11083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:45 smithi116 ceph-mon[35411]: pgmap v11083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:47 smithi114 ceph-mon[30820]: pgmap v11084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:47 smithi116 ceph-mon[35411]: pgmap v11084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:33:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:49 smithi116 ceph-mon[35411]: pgmap v11085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:49 smithi114 ceph-mon[30820]: pgmap v11085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:33:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:51 smithi116 ceph-mon[35411]: pgmap v11086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:51 smithi114 ceph-mon[30820]: pgmap v11086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:33:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:53 smithi116 ceph-mon[35411]: pgmap v11087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:33:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:53 smithi114 ceph-mon[30820]: pgmap v11087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:33:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:33:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:55 smithi116 ceph-mon[35411]: pgmap v11088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:55.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-mon[30820]: pgmap v11088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:33:55.719 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:33:55.668+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:33:55.720 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:33:55.721 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:33:55 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:33:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:33:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:33:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:33:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:33:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:56 smithi114 ceph-mon[30820]: pgmap v11089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:56 smithi114 ceph-mon[30820]: pgmap v11090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:33:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:33:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:33:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:33:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:33:56.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:56 smithi116 ceph-mon[35411]: pgmap v11089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:56 smithi116 ceph-mon[35411]: pgmap v11090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:33:56.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:33:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:33:58 smithi114 ceph-mon[30820]: pgmap v11091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:33:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:33:58 smithi116 ceph-mon[35411]: pgmap v11091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:34:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:00 smithi114 ceph-mon[30820]: pgmap v11092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:34:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:00 smithi116 ceph-mon[35411]: pgmap v11092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:34:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:02 smithi114 ceph-mon[30820]: pgmap v11093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:34:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:02 smithi116 ceph-mon[35411]: pgmap v11093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:34:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:04 smithi114 ceph-mon[30820]: pgmap v11094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:34:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:04 smithi116 ceph-mon[35411]: pgmap v11094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:34:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:06 smithi114 ceph-mon[30820]: pgmap v11095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:06 smithi116 ceph-mon[35411]: pgmap v11095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:34:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:08 smithi114 ceph-mon[30820]: pgmap v11096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:08 smithi116 ceph-mon[35411]: pgmap v11096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:10 smithi114 ceph-mon[30820]: pgmap v11097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:10 smithi116 ceph-mon[35411]: pgmap v11097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:12 smithi114 ceph-mon[30820]: pgmap v11098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:12 smithi116 ceph-mon[35411]: pgmap v11098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:14 smithi114 ceph-mon[30820]: pgmap v11099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:14 smithi116 ceph-mon[35411]: pgmap v11099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:16 smithi114 ceph-mon[30820]: pgmap v11100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:16 smithi116 ceph-mon[35411]: pgmap v11100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:18 smithi114 ceph-mon[30820]: pgmap v11101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:18 smithi116 ceph-mon[35411]: pgmap v11101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:20 smithi114 ceph-mon[30820]: pgmap v11102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:20 smithi116 ceph-mon[35411]: pgmap v11102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:22 smithi114 ceph-mon[30820]: pgmap v11103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:22 smithi116 ceph-mon[35411]: pgmap v11103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:24 smithi114 ceph-mon[30820]: pgmap v11104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:24 smithi116 ceph-mon[35411]: pgmap v11104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:26 smithi114 ceph-mon[30820]: pgmap v11105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:26 smithi116 ceph-mon[35411]: pgmap v11105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:28 smithi114 ceph-mon[30820]: pgmap v11106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:28 smithi116 ceph-mon[35411]: pgmap v11106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:30 smithi114 ceph-mon[30820]: pgmap v11107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:30 smithi116 ceph-mon[35411]: pgmap v11107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:32 smithi114 ceph-mon[30820]: pgmap v11108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:32 smithi116 ceph-mon[35411]: pgmap v11108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:34 smithi114 ceph-mon[30820]: pgmap v11109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:34 smithi116 ceph-mon[35411]: pgmap v11109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:36 smithi114 ceph-mon[30820]: pgmap v11110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:36 smithi116 ceph-mon[35411]: pgmap v11110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:38 smithi114 ceph-mon[30820]: pgmap v11111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:38 smithi116 ceph-mon[35411]: pgmap v11111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:40 smithi114 ceph-mon[30820]: pgmap v11112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:40 smithi116 ceph-mon[35411]: pgmap v11112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:42 smithi114 ceph-mon[30820]: pgmap v11113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:42 smithi116 ceph-mon[35411]: pgmap v11113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:44 smithi114 ceph-mon[30820]: pgmap v11114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:44 smithi116 ceph-mon[35411]: pgmap v11114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:46 smithi114 ceph-mon[30820]: pgmap v11115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:46 smithi116 ceph-mon[35411]: pgmap v11115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:48 smithi114 ceph-mon[30820]: pgmap v11116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:48 smithi116 ceph-mon[35411]: pgmap v11116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:50 smithi114 ceph-mon[30820]: pgmap v11117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:50 smithi116 ceph-mon[35411]: pgmap v11117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:34:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:52 smithi114 ceph-mon[30820]: pgmap v11118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:52 smithi116 ceph-mon[35411]: pgmap v11118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:34:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:54 smithi114 ceph-mon[30820]: pgmap v11119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:54 smithi116 ceph-mon[35411]: pgmap v11119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:34:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:34:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:56 smithi114 ceph-mon[30820]: pgmap v11120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:56 smithi116 ceph-mon[35411]: pgmap v11120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:34:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:58 smithi114 ceph-mon[30820]: pgmap v11121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:34:59.028+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:34:59.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:34:59.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:34:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:58 smithi116 ceph-mon[35411]: pgmap v11121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:35:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:35:00.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:35:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:35:00.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-mon[30820]: pgmap v11122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:35:00.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-mon[30820]: pgmap v11123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:35:00.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:34:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:35:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:59 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:35:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:35:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:35:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:35:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:35:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:59 smithi116 ceph-mon[35411]: pgmap v11122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:59 smithi116 ceph-mon[35411]: pgmap v11123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:35:00.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:34:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:35:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:02 smithi114 ceph-mon[30820]: pgmap v11124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:35:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:02 smithi116 ceph-mon[35411]: pgmap v11124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:35:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:04 smithi114 ceph-mon[30820]: pgmap v11125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:35:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:04 smithi116 ceph-mon[35411]: pgmap v11125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:35:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:06 smithi114 ceph-mon[30820]: pgmap v11126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:35:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:06 smithi116 ceph-mon[35411]: pgmap v11126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:35:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:08 smithi114 ceph-mon[30820]: pgmap v11127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:35:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:08 smithi116 ceph-mon[35411]: pgmap v11127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:35:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:10 smithi114 ceph-mon[30820]: pgmap v11128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:10 smithi116 ceph-mon[35411]: pgmap v11128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:35:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:12 smithi114 ceph-mon[30820]: pgmap v11129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:12 smithi116 ceph-mon[35411]: pgmap v11129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:14 smithi114 ceph-mon[30820]: pgmap v11130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:14 smithi116 ceph-mon[35411]: pgmap v11130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:16 smithi114 ceph-mon[30820]: pgmap v11131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:16 smithi116 ceph-mon[35411]: pgmap v11131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:18 smithi114 ceph-mon[30820]: pgmap v11132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:18 smithi116 ceph-mon[35411]: pgmap v11132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:20 smithi114 ceph-mon[30820]: pgmap v11133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:20 smithi116 ceph-mon[35411]: pgmap v11133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:22 smithi114 ceph-mon[30820]: pgmap v11134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:22 smithi116 ceph-mon[35411]: pgmap v11134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:23 smithi114 ceph-mon[30820]: pgmap v11135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:23 smithi116 ceph-mon[35411]: pgmap v11135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:26 smithi114 ceph-mon[30820]: pgmap v11136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:26 smithi116 ceph-mon[35411]: pgmap v11136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:28 smithi114 ceph-mon[30820]: pgmap v11137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:28 smithi116 ceph-mon[35411]: pgmap v11137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:30 smithi114 ceph-mon[30820]: pgmap v11138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:30 smithi116 ceph-mon[35411]: pgmap v11138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:32 smithi114 ceph-mon[30820]: pgmap v11139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:32 smithi116 ceph-mon[35411]: pgmap v11139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:34 smithi114 ceph-mon[30820]: pgmap v11140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:34 smithi116 ceph-mon[35411]: pgmap v11140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:36 smithi114 ceph-mon[30820]: pgmap v11141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:36 smithi116 ceph-mon[35411]: pgmap v11141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:38 smithi114 ceph-mon[30820]: pgmap v11142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:38 smithi116 ceph-mon[35411]: pgmap v11142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:40 smithi114 ceph-mon[30820]: pgmap v11143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:40 smithi116 ceph-mon[35411]: pgmap v11143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:42 smithi114 ceph-mon[30820]: pgmap v11144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:42 smithi116 ceph-mon[35411]: pgmap v11144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:44 smithi114 ceph-mon[30820]: pgmap v11145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:44 smithi116 ceph-mon[35411]: pgmap v11145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:45 smithi114 ceph-mon[30820]: pgmap v11146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:45 smithi116 ceph-mon[35411]: pgmap v11146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:48 smithi114 ceph-mon[30820]: pgmap v11147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:48 smithi116 ceph-mon[35411]: pgmap v11147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:50 smithi114 ceph-mon[30820]: pgmap v11148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:50 smithi116 ceph-mon[35411]: pgmap v11148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:52 smithi114 ceph-mon[30820]: pgmap v11149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:52 smithi116 ceph-mon[35411]: pgmap v11149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:35:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:54 smithi114 ceph-mon[30820]: pgmap v11150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:54 smithi116 ceph-mon[35411]: pgmap v11150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:35:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:56 smithi114 ceph-mon[30820]: pgmap v11151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:56 smithi116 ceph-mon[35411]: pgmap v11151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:35:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:35:58 smithi114 ceph-mon[30820]: pgmap v11152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:35:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:35:58 smithi116 ceph-mon[35411]: pgmap v11152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: pgmap v11153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:36:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: pgmap v11153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:36:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:00.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:36:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:01 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:36:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:01 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:36:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:01 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:36:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:01 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:36:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:01 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:36:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:01 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:36:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:01 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:36:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:01 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:36:02.391 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-mon[30820]: pgmap v11154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:02 smithi116 ceph-mon[35411]: pgmap v11154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:36:02.388+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:36:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:36:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:36:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:36:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:36:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:36:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:36:02.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:36:02.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:36:02.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:36:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:36:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:36:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:36:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:36:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:36:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:36:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:36:03.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:36:03.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:36:03.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:36:03.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:36:03.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:36:03.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:36:03.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:36:03.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:36:03.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:36:03.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:36:03.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:36:03.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:36:03.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:36:03.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:36:03.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:36:03.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:36:03.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:36:03.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:36:03.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:36:03.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:03 smithi114 ceph-mon[30820]: pgmap v11155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:36:03.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:03 smithi114 ceph-mon[30820]: pgmap v11156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:03.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:36:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:36:03.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:36:03.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:36:03.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:36:03.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:36:03.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:36:03.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:36:03.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:36:03.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:36:03.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:36:03.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:36:03.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:36:03.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:36:03.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:36:03.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:36:03.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:36:03.432 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:36:03.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:36:03.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:36:03.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:36:03.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:36:03.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:36:03.433 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:36:03.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:36:03.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:36:03.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:03 smithi116 ceph-mon[35411]: pgmap v11155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:36:03.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:03 smithi116 ceph-mon[35411]: pgmap v11156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:03.434 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:36:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:05 smithi114 ceph-mon[30820]: pgmap v11157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:05 smithi116 ceph-mon[35411]: pgmap v11157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:07 smithi114 ceph-mon[30820]: pgmap v11158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:07 smithi116 ceph-mon[35411]: pgmap v11158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:09 smithi114 ceph-mon[30820]: pgmap v11159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:09 smithi116 ceph-mon[35411]: pgmap v11159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:11 smithi114 ceph-mon[30820]: pgmap v11160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:11 smithi116 ceph-mon[35411]: pgmap v11160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:36:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:13 smithi114 ceph-mon[30820]: pgmap v11161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:36:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:13 smithi116 ceph-mon[35411]: pgmap v11161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:36:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:15 smithi114 ceph-mon[30820]: pgmap v11162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:15 smithi116 ceph-mon[35411]: pgmap v11162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:17 smithi114 ceph-mon[30820]: pgmap v11163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:17 smithi116 ceph-mon[35411]: pgmap v11163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:19 smithi114 ceph-mon[30820]: pgmap v11164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:19 smithi116 ceph-mon[35411]: pgmap v11164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:21 smithi114 ceph-mon[30820]: pgmap v11165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:21 smithi116 ceph-mon[35411]: pgmap v11165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:23 smithi114 ceph-mon[30820]: pgmap v11166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:23 smithi116 ceph-mon[35411]: pgmap v11166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:25 smithi114 ceph-mon[30820]: pgmap v11167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:25 smithi116 ceph-mon[35411]: pgmap v11167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:27 smithi114 ceph-mon[30820]: pgmap v11168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:27 smithi116 ceph-mon[35411]: pgmap v11168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:29 smithi114 ceph-mon[30820]: pgmap v11169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:29 smithi116 ceph-mon[35411]: pgmap v11169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:31 smithi114 ceph-mon[30820]: pgmap v11170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:31 smithi116 ceph-mon[35411]: pgmap v11170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:33 smithi114 ceph-mon[30820]: pgmap v11171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:33 smithi116 ceph-mon[35411]: pgmap v11171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:35 smithi114 ceph-mon[30820]: pgmap v11172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:35 smithi116 ceph-mon[35411]: pgmap v11172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:37 smithi114 ceph-mon[30820]: pgmap v11173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:37 smithi116 ceph-mon[35411]: pgmap v11173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:39 smithi114 ceph-mon[30820]: pgmap v11174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:39 smithi116 ceph-mon[35411]: pgmap v11174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:41 smithi114 ceph-mon[30820]: pgmap v11175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:41 smithi116 ceph-mon[35411]: pgmap v11175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:43 smithi114 ceph-mon[30820]: pgmap v11176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:43 smithi116 ceph-mon[35411]: pgmap v11176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:45 smithi114 ceph-mon[30820]: pgmap v11177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:45 smithi116 ceph-mon[35411]: pgmap v11177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:47 smithi114 ceph-mon[30820]: pgmap v11178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:47 smithi116 ceph-mon[35411]: pgmap v11178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:49 smithi114 ceph-mon[30820]: pgmap v11179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:49 smithi116 ceph-mon[35411]: pgmap v11179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:51 smithi114 ceph-mon[30820]: pgmap v11180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:51 smithi116 ceph-mon[35411]: pgmap v11180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:36:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:53 smithi114 ceph-mon[30820]: pgmap v11181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:53 smithi116 ceph-mon[35411]: pgmap v11181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:55 smithi114 ceph-mon[30820]: pgmap v11182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:55 smithi116 ceph-mon[35411]: pgmap v11182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:36:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:57 smithi114 ceph-mon[30820]: pgmap v11183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:57 smithi116 ceph-mon[35411]: pgmap v11183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:36:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:36:59 smithi114 ceph-mon[30820]: pgmap v11184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:36:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:36:59 smithi116 ceph-mon[35411]: pgmap v11184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:01 smithi114 ceph-mon[30820]: pgmap v11185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:01 smithi116 ceph-mon[35411]: pgmap v11185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:37:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:37:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:03 smithi114 ceph-mon[30820]: pgmap v11186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:03 smithi116 ceph-mon[35411]: pgmap v11186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-mon[30820]: pgmap v11187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:37:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:37:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:37:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:37:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:37:05.748+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:37:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:37:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:37:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:37:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:37:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:37:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:37:06.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:37:06.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:37:06.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:37:05 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:37:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:05 smithi116 ceph-mon[35411]: pgmap v11187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:37:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:37:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:37:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:37:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:06 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:37:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:37:07.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:37:07.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:37:07.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:06 smithi114 ceph-mon[30820]: pgmap v11188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:37:07.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:06 smithi114 ceph-mon[30820]: pgmap v11189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:37:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:06 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:37:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:37:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:37:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:37:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:37:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:37:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:37:07.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:37:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:37:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:37:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:37:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:37:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:37:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:06 smithi116 ceph-mon[35411]: pgmap v11188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:37:07.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:06 smithi116 ceph-mon[35411]: pgmap v11189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:37:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:08 smithi114 ceph-mon[30820]: pgmap v11190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:37:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:08 smithi116 ceph-mon[35411]: pgmap v11190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:37:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:10 smithi116 ceph-mon[35411]: pgmap v11191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:37:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:10 smithi114 ceph-mon[30820]: pgmap v11191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:37:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:12 smithi116 ceph-mon[35411]: pgmap v11192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:37:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:12 smithi114 ceph-mon[30820]: pgmap v11192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:37:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:14 smithi116 ceph-mon[35411]: pgmap v11193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:37:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:14 smithi114 ceph-mon[30820]: pgmap v11193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:37:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:16 smithi114 ceph-mon[30820]: pgmap v11194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:37:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:16 smithi116 ceph-mon[35411]: pgmap v11194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:37:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:18 smithi114 ceph-mon[30820]: pgmap v11195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:18 smithi116 ceph-mon[35411]: pgmap v11195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:20 smithi114 ceph-mon[30820]: pgmap v11196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:37:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:20 smithi116 ceph-mon[35411]: pgmap v11196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:37:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:22 smithi114 ceph-mon[30820]: pgmap v11197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:37:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:22 smithi116 ceph-mon[35411]: pgmap v11197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:37:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:24 smithi114 ceph-mon[30820]: pgmap v11198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:24 smithi116 ceph-mon[35411]: pgmap v11198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:27.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:26 smithi114 ceph-mon[30820]: pgmap v11199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:26 smithi116 ceph-mon[35411]: pgmap v11199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:28 smithi114 ceph-mon[30820]: pgmap v11200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:28 smithi116 ceph-mon[35411]: pgmap v11200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:30 smithi114 ceph-mon[30820]: pgmap v11201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:37:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:30 smithi116 ceph-mon[35411]: pgmap v11201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:37:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:32 smithi114 ceph-mon[30820]: pgmap v11202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:37:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:32 smithi116 ceph-mon[35411]: pgmap v11202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:37:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:34 smithi114 ceph-mon[30820]: pgmap v11203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:34 smithi116 ceph-mon[35411]: pgmap v11203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:37.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:36 smithi116 ceph-mon[35411]: pgmap v11204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:36 smithi114 ceph-mon[30820]: pgmap v11204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:38 smithi114 ceph-mon[30820]: pgmap v11205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:37:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:38 smithi116 ceph-mon[35411]: pgmap v11205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:37:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:40 smithi114 ceph-mon[30820]: pgmap v11206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:37:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:40 smithi116 ceph-mon[35411]: pgmap v11206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:37:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:42 smithi114 ceph-mon[30820]: pgmap v11207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:37:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:42 smithi116 ceph-mon[35411]: pgmap v11207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:37:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:44 smithi114 ceph-mon[30820]: pgmap v11208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:37:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:44 smithi116 ceph-mon[35411]: pgmap v11208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:37:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:46 smithi114 ceph-mon[30820]: pgmap v11209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:46 smithi116 ceph-mon[35411]: pgmap v11209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:48 smithi114 ceph-mon[30820]: pgmap v11210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:48 smithi116 ceph-mon[35411]: pgmap v11210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:50 smithi114 ceph-mon[30820]: pgmap v11211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:37:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:50 smithi116 ceph-mon[35411]: pgmap v11211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:37:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:52 smithi114 ceph-mon[30820]: pgmap v11212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:37:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:52 smithi116 ceph-mon[35411]: pgmap v11212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:37:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:54 smithi114 ceph-mon[30820]: pgmap v11213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:55 smithi116 ceph-mon[35411]: pgmap v11213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:57 smithi114 ceph-mon[30820]: pgmap v11214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:57 smithi116 ceph-mon[35411]: pgmap v11214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:37:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:37:59 smithi114 ceph-mon[30820]: pgmap v11215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:37:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:37:59 smithi116 ceph-mon[35411]: pgmap v11215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:01 smithi114 ceph-mon[30820]: pgmap v11216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:38:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:01 smithi116 ceph-mon[35411]: pgmap v11216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:38:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:03 smithi114 ceph-mon[30820]: pgmap v11217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:03 smithi116 ceph-mon[35411]: pgmap v11217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:05 smithi114 ceph-mon[30820]: pgmap v11218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:05 smithi116 ceph-mon[35411]: pgmap v11218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:38:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:38:07.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:07 smithi116 ceph-mon[35411]: pgmap v11219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:07 smithi114 ceph-mon[30820]: pgmap v11219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-mon[30820]: pgmap v11220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:38:09.109+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:38:09.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:38:09.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:38:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:38:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:09 smithi116 ceph-mon[35411]: pgmap v11220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:10 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:38:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:10 smithi114 ceph-mon[30820]: pgmap v11221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:10 smithi114 ceph-mon[30820]: pgmap v11222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:38:10.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:38:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:10 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:38:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:38:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:38:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:38:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:38:10.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:10 smithi116 ceph-mon[35411]: pgmap v11221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:10 smithi116 ceph-mon[35411]: pgmap v11222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:38:10.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:38:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:11 smithi114 ceph-mon[30820]: pgmap v11223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:38:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:11 smithi116 ceph-mon[35411]: pgmap v11223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:38:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:13 smithi114 ceph-mon[30820]: pgmap v11224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:38:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:13 smithi116 ceph-mon[35411]: pgmap v11224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:38:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:15 smithi114 ceph-mon[30820]: pgmap v11225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:38:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:15 smithi116 ceph-mon[35411]: pgmap v11225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:38:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:17 smithi114 ceph-mon[30820]: pgmap v11226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:38:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:17 smithi116 ceph-mon[35411]: pgmap v11226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:38:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:19 smithi114 ceph-mon[30820]: pgmap v11227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:38:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:19 smithi116 ceph-mon[35411]: pgmap v11227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:38:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:21 smithi114 ceph-mon[30820]: pgmap v11228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:21 smithi116 ceph-mon[35411]: pgmap v11228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:23 smithi114 ceph-mon[30820]: pgmap v11229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:23 smithi116 ceph-mon[35411]: pgmap v11229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:25 smithi114 ceph-mon[30820]: pgmap v11230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:38:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:25 smithi116 ceph-mon[35411]: pgmap v11230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:38:27.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:27 smithi114 ceph-mon[30820]: pgmap v11231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:27 smithi116 ceph-mon[35411]: pgmap v11231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:29 smithi114 ceph-mon[30820]: pgmap v11232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:29 smithi116 ceph-mon[35411]: pgmap v11232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:31 smithi114 ceph-mon[30820]: pgmap v11233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:31 smithi116 ceph-mon[35411]: pgmap v11233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:33 smithi114 ceph-mon[30820]: pgmap v11234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:33 smithi116 ceph-mon[35411]: pgmap v11234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:35 smithi114 ceph-mon[30820]: pgmap v11235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:38:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:35 smithi116 ceph-mon[35411]: pgmap v11235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:38:37.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:37 smithi116 ceph-mon[35411]: pgmap v11236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:37 smithi114 ceph-mon[30820]: pgmap v11236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:39 smithi114 ceph-mon[30820]: pgmap v11237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:39 smithi116 ceph-mon[35411]: pgmap v11237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:41 smithi114 ceph-mon[30820]: pgmap v11238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:41 smithi116 ceph-mon[35411]: pgmap v11238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:43 smithi114 ceph-mon[30820]: pgmap v11239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:38:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:43 smithi116 ceph-mon[35411]: pgmap v11239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:38:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:45 smithi114 ceph-mon[30820]: pgmap v11240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:38:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:45 smithi116 ceph-mon[35411]: pgmap v11240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:38:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:47 smithi114 ceph-mon[30820]: pgmap v11241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:47 smithi116 ceph-mon[35411]: pgmap v11241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:49 smithi114 ceph-mon[30820]: pgmap v11242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:38:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:49 smithi116 ceph-mon[35411]: pgmap v11242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:38:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:51 smithi114 ceph-mon[30820]: pgmap v11243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:51 smithi116 ceph-mon[35411]: pgmap v11243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:38:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:53 smithi114 ceph-mon[30820]: pgmap v11244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:53 smithi116 ceph-mon[35411]: pgmap v11244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:55 smithi114 ceph-mon[30820]: pgmap v11245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:38:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:55 smithi116 ceph-mon[35411]: pgmap v11245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:38:57.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:57 smithi114 ceph-mon[30820]: pgmap v11246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:57 smithi116 ceph-mon[35411]: pgmap v11246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:38:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:38:59 smithi114 ceph-mon[30820]: pgmap v11247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:38:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:38:59 smithi116 ceph-mon[35411]: pgmap v11247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:01 smithi114 ceph-mon[30820]: pgmap v11248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:01 smithi116 ceph-mon[35411]: pgmap v11248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:03 smithi114 ceph-mon[30820]: pgmap v11249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:03 smithi116 ceph-mon[35411]: pgmap v11249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:05 smithi114 ceph-mon[30820]: pgmap v11250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:05 smithi116 ceph-mon[35411]: pgmap v11250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:07.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:07 smithi116 ceph-mon[35411]: pgmap v11251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:07 smithi114 ceph-mon[30820]: pgmap v11251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:09 smithi114 ceph-mon[30820]: pgmap v11252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:09 smithi116 ceph-mon[35411]: pgmap v11252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:39:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:39:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:11 smithi114 ceph-mon[30820]: pgmap v11253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:11 smithi116 ceph-mon[35411]: pgmap v11253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:12.755 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:39:12.468+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:39:12.755 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:39:12.755 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:39:12.755 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:39:12.755 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:39:12.756 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:39:12.757 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:39:12.757 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:39:12.757 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:39:12.757 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:39:12.757 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:39:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:39:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:39:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:39:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:39:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:39:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:13 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:39:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:39:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:39:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:13 smithi114 ceph-mon[30820]: pgmap v11254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:39:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:13 smithi114 ceph-mon[30820]: pgmap v11255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:13 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:39:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:39:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:39:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:39:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:39:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:39:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:39:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:39:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:39:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:39:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:39:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:13 smithi116 ceph-mon[35411]: pgmap v11254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:39:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:13 smithi116 ceph-mon[35411]: pgmap v11255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:15 smithi114 ceph-mon[30820]: pgmap v11256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:15 smithi116 ceph-mon[35411]: pgmap v11256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:17 smithi114 ceph-mon[30820]: pgmap v11257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:17 smithi116 ceph-mon[35411]: pgmap v11257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:19 smithi114 ceph-mon[30820]: pgmap v11258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:19 smithi116 ceph-mon[35411]: pgmap v11258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:21 smithi114 ceph-mon[30820]: pgmap v11259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:21 smithi116 ceph-mon[35411]: pgmap v11259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:39:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:23 smithi114 ceph-mon[30820]: pgmap v11260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:39:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:23 smithi116 ceph-mon[35411]: pgmap v11260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:39:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:25 smithi114 ceph-mon[30820]: pgmap v11261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:25 smithi116 ceph-mon[35411]: pgmap v11261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:27 smithi114 ceph-mon[30820]: pgmap v11262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:27 smithi116 ceph-mon[35411]: pgmap v11262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:29 smithi114 ceph-mon[30820]: pgmap v11263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:29 smithi116 ceph-mon[35411]: pgmap v11263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:31 smithi114 ceph-mon[30820]: pgmap v11264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:31 smithi116 ceph-mon[35411]: pgmap v11264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:33 smithi114 ceph-mon[30820]: pgmap v11265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:33 smithi116 ceph-mon[35411]: pgmap v11265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:35 smithi114 ceph-mon[30820]: pgmap v11266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:35 smithi116 ceph-mon[35411]: pgmap v11266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:37 smithi114 ceph-mon[30820]: pgmap v11267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:37 smithi116 ceph-mon[35411]: pgmap v11267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:39 smithi114 ceph-mon[30820]: pgmap v11268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:39 smithi116 ceph-mon[35411]: pgmap v11268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:41 smithi114 ceph-mon[30820]: pgmap v11269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:41 smithi116 ceph-mon[35411]: pgmap v11269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:43 smithi114 ceph-mon[30820]: pgmap v11270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:43 smithi116 ceph-mon[35411]: pgmap v11270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:45 smithi114 ceph-mon[30820]: pgmap v11271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:45 smithi116 ceph-mon[35411]: pgmap v11271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:47 smithi114 ceph-mon[30820]: pgmap v11272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:47 smithi116 ceph-mon[35411]: pgmap v11272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:49 smithi114 ceph-mon[30820]: pgmap v11273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:49 smithi116 ceph-mon[35411]: pgmap v11273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:51 smithi114 ceph-mon[30820]: pgmap v11274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:51 smithi116 ceph-mon[35411]: pgmap v11274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:39:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:53 smithi114 ceph-mon[30820]: pgmap v11275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:53 smithi116 ceph-mon[35411]: pgmap v11275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:39:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:55 smithi114 ceph-mon[30820]: pgmap v11276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:55 smithi116 ceph-mon[35411]: pgmap v11276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:39:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:57 smithi114 ceph-mon[30820]: pgmap v11277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:39:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:57 smithi116 ceph-mon[35411]: pgmap v11277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:39:59 smithi114 ceph-mon[30820]: pgmap v11278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:39:59 smithi116 ceph-mon[35411]: pgmap v11278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:40:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:40:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:01 smithi114 ceph-mon[30820]: pgmap v11279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:01 smithi116 ceph-mon[35411]: pgmap v11279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:03 smithi114 ceph-mon[30820]: pgmap v11280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:03 smithi116 ceph-mon[35411]: pgmap v11280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:05 smithi114 ceph-mon[30820]: pgmap v11281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:05 smithi116 ceph-mon[35411]: pgmap v11281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:07 smithi114 ceph-mon[30820]: pgmap v11282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:07 smithi116 ceph-mon[35411]: pgmap v11282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:09 smithi114 ceph-mon[30820]: pgmap v11283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:09 smithi116 ceph-mon[35411]: pgmap v11283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:11 smithi114 ceph-mon[30820]: pgmap v11284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:11 smithi116 ceph-mon[35411]: pgmap v11284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:40:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:40:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:13 smithi114 ceph-mon[30820]: pgmap v11285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:13 smithi116 ceph-mon[35411]: pgmap v11285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:14.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-mon[30820]: pgmap v11286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:40:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:40:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:40:15.828+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:40:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:40:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:40:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:40:16.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:40:16.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:40:16.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:40:16.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:40:16.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:40:16.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:40:16.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:40:15 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:40:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:15 smithi116 ceph-mon[35411]: pgmap v11286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:40:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:40:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:40:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:16 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:40:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:40:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:16 smithi114 ceph-mon[30820]: pgmap v11287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:40:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:16 smithi114 ceph-mon[30820]: pgmap v11288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:40:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:16 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:40:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:40:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:40:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:16 smithi116 ceph-mon[35411]: pgmap v11287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:40:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:16 smithi116 ceph-mon[35411]: pgmap v11288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:40:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:18 smithi114 ceph-mon[30820]: pgmap v11289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:40:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:18 smithi116 ceph-mon[35411]: pgmap v11289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:40:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:20 smithi114 ceph-mon[30820]: pgmap v11290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:40:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:20 smithi116 ceph-mon[35411]: pgmap v11290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:40:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:22 smithi114 ceph-mon[30820]: pgmap v11291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:40:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:22 smithi116 ceph-mon[35411]: pgmap v11291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:40:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:24 smithi114 ceph-mon[30820]: pgmap v11292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:40:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:24 smithi116 ceph-mon[35411]: pgmap v11292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:40:27.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:26 smithi114 ceph-mon[30820]: pgmap v11293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:40:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:26 smithi116 ceph-mon[35411]: pgmap v11293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:40:29.358 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:28 smithi114 ceph-mon[30820]: pgmap v11294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:28 smithi116 ceph-mon[35411]: pgmap v11294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:30 smithi114 ceph-mon[30820]: pgmap v11295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:30 smithi116 ceph-mon[35411]: pgmap v11295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:32 smithi114 ceph-mon[30820]: pgmap v11296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:32 smithi116 ceph-mon[35411]: pgmap v11296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:34 smithi114 ceph-mon[30820]: pgmap v11297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:34 smithi116 ceph-mon[35411]: pgmap v11297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:37.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:36 smithi116 ceph-mon[35411]: pgmap v11298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:36 smithi114 ceph-mon[30820]: pgmap v11298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:38 smithi114 ceph-mon[30820]: pgmap v11299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:38 smithi116 ceph-mon[35411]: pgmap v11299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:40 smithi114 ceph-mon[30820]: pgmap v11300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:40 smithi116 ceph-mon[35411]: pgmap v11300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:42 smithi114 ceph-mon[30820]: pgmap v11301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:42 smithi116 ceph-mon[35411]: pgmap v11301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:44 smithi114 ceph-mon[30820]: pgmap v11302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:44 smithi116 ceph-mon[35411]: pgmap v11302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:46 smithi114 ceph-mon[30820]: pgmap v11303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:46 smithi116 ceph-mon[35411]: pgmap v11303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:48 smithi114 ceph-mon[30820]: pgmap v11304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:48 smithi116 ceph-mon[35411]: pgmap v11304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:50 smithi114 ceph-mon[30820]: pgmap v11305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:50 smithi116 ceph-mon[35411]: pgmap v11305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:40:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:52 smithi114 ceph-mon[30820]: pgmap v11306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:52 smithi116 ceph-mon[35411]: pgmap v11306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:40:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:54 smithi114 ceph-mon[30820]: pgmap v11307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:54 smithi116 ceph-mon[35411]: pgmap v11307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:57 smithi114 ceph-mon[30820]: pgmap v11308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:57 smithi116 ceph-mon[35411]: pgmap v11308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:40:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:40:59 smithi114 ceph-mon[30820]: pgmap v11309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:40:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:40:59 smithi116 ceph-mon[35411]: pgmap v11309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:01 smithi114 ceph-mon[30820]: pgmap v11310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:01 smithi116 ceph-mon[35411]: pgmap v11310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:03 smithi114 ceph-mon[30820]: pgmap v11311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:03 smithi116 ceph-mon[35411]: pgmap v11311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:05 smithi114 ceph-mon[30820]: pgmap v11312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:05 smithi116 ceph-mon[35411]: pgmap v11312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:07 smithi114 ceph-mon[30820]: pgmap v11313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:07 smithi116 ceph-mon[35411]: pgmap v11313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:09 smithi114 ceph-mon[30820]: pgmap v11314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:09 smithi116 ceph-mon[35411]: pgmap v11314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:11 smithi114 ceph-mon[30820]: pgmap v11315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:11 smithi116 ceph-mon[35411]: pgmap v11315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:13 smithi114 ceph-mon[30820]: pgmap v11316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:13 smithi116 ceph-mon[35411]: pgmap v11316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:15 smithi114 ceph-mon[30820]: pgmap v11317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:15 smithi116 ceph-mon[35411]: pgmap v11317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:16 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:41:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:16 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:41:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:17 smithi114 ceph-mon[30820]: pgmap v11318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:17 smithi116 ceph-mon[35411]: pgmap v11318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-mon[30820]: pgmap v11319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:19.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:41:19.188+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:41:19.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:41:19.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:41:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:41:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:19 smithi116 ceph-mon[35411]: pgmap v11319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:41:20.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:41:20.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:41:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:41:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:41:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:41:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:41:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:41:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:21 smithi114 ceph-mon[30820]: pgmap v11320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:41:21.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:21 smithi114 ceph-mon[30820]: pgmap v11321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:41:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:41:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:41:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:41:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:41:21.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:41:21.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:41:21.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:41:21.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:41:21.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:41:21.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:41:21.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:41:21.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:21 smithi116 ceph-mon[35411]: pgmap v11320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:41:21.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:21 smithi116 ceph-mon[35411]: pgmap v11321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:41:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:23 smithi114 ceph-mon[30820]: pgmap v11322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:41:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:23 smithi116 ceph-mon[35411]: pgmap v11322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:41:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:25 smithi114 ceph-mon[30820]: pgmap v11323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:41:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:25 smithi116 ceph-mon[35411]: pgmap v11323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:41:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:27 smithi114 ceph-mon[30820]: pgmap v11324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:41:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:27 smithi116 ceph-mon[35411]: pgmap v11324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:41:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:29 smithi114 ceph-mon[30820]: pgmap v11325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:41:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:29 smithi116 ceph-mon[35411]: pgmap v11325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:41:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:31 smithi114 ceph-mon[30820]: pgmap v11326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:41:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:31 smithi116 ceph-mon[35411]: pgmap v11326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:41:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:33 smithi114 ceph-mon[30820]: pgmap v11327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:33 smithi116 ceph-mon[35411]: pgmap v11327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:35 smithi114 ceph-mon[30820]: pgmap v11328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:35 smithi116 ceph-mon[35411]: pgmap v11328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:37 smithi114 ceph-mon[30820]: pgmap v11329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:37 smithi116 ceph-mon[35411]: pgmap v11329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:39 smithi114 ceph-mon[30820]: pgmap v11330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:39 smithi116 ceph-mon[35411]: pgmap v11330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:41 smithi114 ceph-mon[30820]: pgmap v11331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:41 smithi116 ceph-mon[35411]: pgmap v11331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:43 smithi114 ceph-mon[30820]: pgmap v11332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:43 smithi116 ceph-mon[35411]: pgmap v11332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:45 smithi114 ceph-mon[30820]: pgmap v11333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:45 smithi116 ceph-mon[35411]: pgmap v11333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:47 smithi114 ceph-mon[30820]: pgmap v11334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:47 smithi116 ceph-mon[35411]: pgmap v11334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:49 smithi114 ceph-mon[30820]: pgmap v11335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:49 smithi116 ceph-mon[35411]: pgmap v11335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:51 smithi114 ceph-mon[30820]: pgmap v11336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:51 smithi116 ceph-mon[35411]: pgmap v11336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:53 smithi114 ceph-mon[30820]: pgmap v11337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:53 smithi116 ceph-mon[35411]: pgmap v11337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:41:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:55 smithi114 ceph-mon[30820]: pgmap v11338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:55 smithi116 ceph-mon[35411]: pgmap v11338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:41:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:57 smithi114 ceph-mon[30820]: pgmap v11339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:57 smithi116 ceph-mon[35411]: pgmap v11339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:41:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:41:59 smithi114 ceph-mon[30820]: pgmap v11340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:41:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:41:59 smithi116 ceph-mon[35411]: pgmap v11340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:01 smithi114 ceph-mon[30820]: pgmap v11341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:01 smithi116 ceph-mon[35411]: pgmap v11341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:03 smithi114 ceph-mon[30820]: pgmap v11342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:03 smithi116 ceph-mon[35411]: pgmap v11342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:05 smithi114 ceph-mon[30820]: pgmap v11343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:05 smithi116 ceph-mon[35411]: pgmap v11343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:07 smithi114 ceph-mon[30820]: pgmap v11344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:07 smithi116 ceph-mon[35411]: pgmap v11344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:09 smithi114 ceph-mon[30820]: pgmap v11345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:09 smithi116 ceph-mon[35411]: pgmap v11345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:11 smithi114 ceph-mon[30820]: pgmap v11346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:11 smithi116 ceph-mon[35411]: pgmap v11346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:13 smithi114 ceph-mon[30820]: pgmap v11347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:13 smithi116 ceph-mon[35411]: pgmap v11347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:15.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:15 smithi114 ceph-mon[30820]: pgmap v11348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:15 smithi116 ceph-mon[35411]: pgmap v11348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:17 smithi114 ceph-mon[30820]: pgmap v11349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:17 smithi116 ceph-mon[35411]: pgmap v11349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:19 smithi114 ceph-mon[30820]: pgmap v11350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:19 smithi116 ceph-mon[35411]: pgmap v11350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:42:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:42:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:21 smithi116 ceph-mon[35411]: pgmap v11351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:21 smithi114 ceph-mon[30820]: pgmap v11351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:42:22.548+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:42:22.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:42:22.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:42:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:42:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:23 smithi114 ceph-mon[30820]: pgmap v11352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:42:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:42:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:42:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:23 smithi116 ceph-mon[35411]: pgmap v11352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:42:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:42:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:42:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:42:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:42:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:42:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:24 smithi114 ceph-mon[30820]: pgmap v11353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:42:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:24 smithi114 ceph-mon[30820]: pgmap v11354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:42:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:42:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:42:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:42:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:42:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:42:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:42:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:42:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:42:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:42:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:24 smithi116 ceph-mon[35411]: pgmap v11353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:42:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:24 smithi116 ceph-mon[35411]: pgmap v11354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:26 smithi114 ceph-mon[30820]: pgmap v11355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:26 smithi116 ceph-mon[35411]: pgmap v11355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:28.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:28 smithi114 ceph-mon[30820]: pgmap v11356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:28 smithi116 ceph-mon[35411]: pgmap v11356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:30 smithi114 ceph-mon[30820]: pgmap v11357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:30 smithi116 ceph-mon[35411]: pgmap v11357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:31 smithi114 ceph-mon[30820]: pgmap v11358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:31 smithi116 ceph-mon[35411]: pgmap v11358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:42:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:33 smithi114 ceph-mon[30820]: pgmap v11359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:42:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:33 smithi116 ceph-mon[35411]: pgmap v11359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:42:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:35 smithi114 ceph-mon[30820]: pgmap v11360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:35 smithi116 ceph-mon[35411]: pgmap v11360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:37 smithi114 ceph-mon[30820]: pgmap v11361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:37 smithi116 ceph-mon[35411]: pgmap v11361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:39 smithi114 ceph-mon[30820]: pgmap v11362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:39 smithi116 ceph-mon[35411]: pgmap v11362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:41 smithi114 ceph-mon[30820]: pgmap v11363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:41 smithi116 ceph-mon[35411]: pgmap v11363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:43 smithi114 ceph-mon[30820]: pgmap v11364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:43 smithi116 ceph-mon[35411]: pgmap v11364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:45 smithi114 ceph-mon[30820]: pgmap v11365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:45 smithi116 ceph-mon[35411]: pgmap v11365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:47 smithi114 ceph-mon[30820]: pgmap v11366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:47 smithi116 ceph-mon[35411]: pgmap v11366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:49 smithi114 ceph-mon[30820]: pgmap v11367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:49 smithi116 ceph-mon[35411]: pgmap v11367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:51 smithi114 ceph-mon[30820]: pgmap v11368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:51 smithi116 ceph-mon[35411]: pgmap v11368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:42:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:53 smithi114 ceph-mon[30820]: pgmap v11369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:53 smithi116 ceph-mon[35411]: pgmap v11369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:55 smithi114 ceph-mon[30820]: pgmap v11370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:55 smithi116 ceph-mon[35411]: pgmap v11370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:42:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:57 smithi114 ceph-mon[30820]: pgmap v11371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:57 smithi116 ceph-mon[35411]: pgmap v11371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:42:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:42:59 smithi114 ceph-mon[30820]: pgmap v11372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:42:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:42:59 smithi116 ceph-mon[35411]: pgmap v11372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:01 smithi116 ceph-mon[35411]: pgmap v11373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:01 smithi114 ceph-mon[30820]: pgmap v11373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:03 smithi116 ceph-mon[35411]: pgmap v11374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:03 smithi114 ceph-mon[30820]: pgmap v11374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:05 smithi116 ceph-mon[35411]: pgmap v11375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:05 smithi114 ceph-mon[30820]: pgmap v11375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:07 smithi114 ceph-mon[30820]: pgmap v11376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:43:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:07 smithi116 ceph-mon[35411]: pgmap v11376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:43:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:09 smithi114 ceph-mon[30820]: pgmap v11377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:09 smithi116 ceph-mon[35411]: pgmap v11377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:11 smithi114 ceph-mon[30820]: pgmap v11378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:11 smithi116 ceph-mon[35411]: pgmap v11378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:13 smithi114 ceph-mon[30820]: pgmap v11379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:13 smithi116 ceph-mon[35411]: pgmap v11379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:15 smithi114 ceph-mon[30820]: pgmap v11380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:15 smithi116 ceph-mon[35411]: pgmap v11380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:17 smithi114 ceph-mon[30820]: pgmap v11381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:43:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:17 smithi116 ceph-mon[35411]: pgmap v11381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:43:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:19 smithi114 ceph-mon[30820]: pgmap v11382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:19 smithi116 ceph-mon[35411]: pgmap v11382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:21 smithi114 ceph-mon[30820]: pgmap v11383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:21 smithi116 ceph-mon[35411]: pgmap v11383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:22 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:43:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:22 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:43:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:23 smithi114 ceph-mon[30820]: pgmap v11384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:23 smithi116 ceph-mon[35411]: pgmap v11384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-mon[30820]: pgmap v11385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:43:25.908+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:43:26.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:43:26.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:43:26.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:43:25 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:43:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:25 smithi116 ceph-mon[35411]: pgmap v11385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:43:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:43:27.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:26 smithi114 ceph-mon[30820]: pgmap v11386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:43:27.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:26 smithi114 ceph-mon[30820]: pgmap v11387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:43:27.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:43:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:43:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:43:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:43:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:43:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:43:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:43:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:43:27.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:26 smithi116 ceph-mon[35411]: pgmap v11386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:26 smithi116 ceph-mon[35411]: pgmap v11387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:43:27.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:43:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:28 smithi114 ceph-mon[30820]: pgmap v11388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:43:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:28 smithi116 ceph-mon[35411]: pgmap v11388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:43:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:30 smithi114 ceph-mon[30820]: pgmap v11389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:43:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:30 smithi116 ceph-mon[35411]: pgmap v11389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:43:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:32 smithi114 ceph-mon[30820]: pgmap v11390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:43:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:32 smithi116 ceph-mon[35411]: pgmap v11390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:43:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:34 smithi114 ceph-mon[30820]: pgmap v11391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:43:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:34 smithi116 ceph-mon[35411]: pgmap v11391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:43:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:36 smithi114 ceph-mon[30820]: pgmap v11392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:43:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:36 smithi116 ceph-mon[35411]: pgmap v11392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:43:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:38 smithi114 ceph-mon[30820]: pgmap v11393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:38 smithi116 ceph-mon[35411]: pgmap v11393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:40 smithi114 ceph-mon[30820]: pgmap v11394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:40 smithi116 ceph-mon[35411]: pgmap v11394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:42 smithi114 ceph-mon[30820]: pgmap v11395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:43:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:42 smithi116 ceph-mon[35411]: pgmap v11395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:43:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:44 smithi114 ceph-mon[30820]: pgmap v11396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:44 smithi116 ceph-mon[35411]: pgmap v11396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:47 smithi114 ceph-mon[30820]: pgmap v11397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:47 smithi116 ceph-mon[35411]: pgmap v11397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:49 smithi114 ceph-mon[30820]: pgmap v11398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:49 smithi116 ceph-mon[35411]: pgmap v11398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:51 smithi114 ceph-mon[30820]: pgmap v11399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:51 smithi116 ceph-mon[35411]: pgmap v11399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:43:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:53 smithi114 ceph-mon[30820]: pgmap v11400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:43:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:53 smithi116 ceph-mon[35411]: pgmap v11400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:43:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:55 smithi114 ceph-mon[30820]: pgmap v11401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:55 smithi116 ceph-mon[35411]: pgmap v11401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:57 smithi114 ceph-mon[30820]: pgmap v11402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:57.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:57 smithi116 ceph-mon[35411]: pgmap v11402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:43:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:43:59 smithi114 ceph-mon[30820]: pgmap v11403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:43:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:43:59 smithi116 ceph-mon[35411]: pgmap v11403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:01 smithi114 ceph-mon[30820]: pgmap v11404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:01 smithi116 ceph-mon[35411]: pgmap v11404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:03 smithi114 ceph-mon[30820]: pgmap v11405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:03 smithi116 ceph-mon[35411]: pgmap v11405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:05 smithi114 ceph-mon[30820]: pgmap v11406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:05 smithi116 ceph-mon[35411]: pgmap v11406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:07 smithi114 ceph-mon[30820]: pgmap v11407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:44:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:07 smithi116 ceph-mon[35411]: pgmap v11407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:44:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:09 smithi114 ceph-mon[30820]: pgmap v11408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:09 smithi116 ceph-mon[35411]: pgmap v11408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:11 smithi114 ceph-mon[30820]: pgmap v11409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:11 smithi116 ceph-mon[35411]: pgmap v11409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:13 smithi114 ceph-mon[30820]: pgmap v11410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:13 smithi116 ceph-mon[35411]: pgmap v11410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:15 smithi114 ceph-mon[30820]: pgmap v11411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:15.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:15 smithi116 ceph-mon[35411]: pgmap v11411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:17 smithi114 ceph-mon[30820]: pgmap v11412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:44:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:17 smithi116 ceph-mon[35411]: pgmap v11412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:44:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:19 smithi114 ceph-mon[30820]: pgmap v11413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:19 smithi116 ceph-mon[35411]: pgmap v11413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:21 smithi114 ceph-mon[30820]: pgmap v11414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:21 smithi116 ceph-mon[35411]: pgmap v11414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:23 smithi114 ceph-mon[30820]: pgmap v11415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:23 smithi116 ceph-mon[35411]: pgmap v11415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:25 smithi114 ceph-mon[30820]: pgmap v11416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:25 smithi116 ceph-mon[35411]: pgmap v11416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:44:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:44:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:27 smithi114 ceph-mon[30820]: pgmap v11417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:44:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:27 smithi116 ceph-mon[35411]: pgmap v11417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:44:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-mon[30820]: pgmap v11418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:44:29.268+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:44:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:44:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:44:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:44:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:44:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:44:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:44:29.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:44:29.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:44:29.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:44:29.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:44:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:44:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:29 smithi116 ceph-mon[35411]: pgmap v11418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:44:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:44:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:44:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:44:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:44:31.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:31 smithi114 ceph-mon[30820]: pgmap v11419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:44:31.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:31 smithi114 ceph-mon[30820]: pgmap v11420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:44:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:44:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:44:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:44:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:44:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:44:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:44:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:44:31.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:44:31.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:44:31.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:44:31.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:31 smithi116 ceph-mon[35411]: pgmap v11419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:44:31.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:31 smithi116 ceph-mon[35411]: pgmap v11420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:44:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:33 smithi114 ceph-mon[30820]: pgmap v11421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:44:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:33 smithi116 ceph-mon[35411]: pgmap v11421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:44:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:35 smithi114 ceph-mon[30820]: pgmap v11422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:44:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:35 smithi116 ceph-mon[35411]: pgmap v11422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:44:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:37 smithi114 ceph-mon[30820]: pgmap v11423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:44:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:37 smithi116 ceph-mon[35411]: pgmap v11423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:44:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:39 smithi114 ceph-mon[30820]: pgmap v11424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:44:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:39 smithi116 ceph-mon[35411]: pgmap v11424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:44:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:41 smithi114 ceph-mon[30820]: pgmap v11425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:44:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:41 smithi116 ceph-mon[35411]: pgmap v11425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:44:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:43 smithi114 ceph-mon[30820]: pgmap v11426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:43 smithi116 ceph-mon[35411]: pgmap v11426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:45 smithi114 ceph-mon[30820]: pgmap v11427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:45 smithi116 ceph-mon[35411]: pgmap v11427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:47 smithi114 ceph-mon[30820]: pgmap v11428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:47 smithi116 ceph-mon[35411]: pgmap v11428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:49 smithi114 ceph-mon[30820]: pgmap v11429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:49 smithi116 ceph-mon[35411]: pgmap v11429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:51 smithi114 ceph-mon[30820]: pgmap v11430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:51 smithi116 ceph-mon[35411]: pgmap v11430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:53 smithi114 ceph-mon[30820]: pgmap v11431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:53 smithi116 ceph-mon[35411]: pgmap v11431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:55 smithi114 ceph-mon[30820]: pgmap v11432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:55 smithi116 ceph-mon[35411]: pgmap v11432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:44:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:57 smithi114 ceph-mon[30820]: pgmap v11433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:57 smithi116 ceph-mon[35411]: pgmap v11433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:44:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:44:59 smithi114 ceph-mon[30820]: pgmap v11434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:44:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:44:59 smithi116 ceph-mon[35411]: pgmap v11434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:01 smithi114 ceph-mon[30820]: pgmap v11435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:45:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:01 smithi116 ceph-mon[35411]: pgmap v11435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:45:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:03 smithi114 ceph-mon[30820]: pgmap v11436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:03 smithi116 ceph-mon[35411]: pgmap v11436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:05 smithi114 ceph-mon[30820]: pgmap v11437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:45:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:05 smithi116 ceph-mon[35411]: pgmap v11437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:45:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:07 smithi114 ceph-mon[30820]: pgmap v11438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:45:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:07 smithi116 ceph-mon[35411]: pgmap v11438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:45:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:09 smithi114 ceph-mon[30820]: pgmap v11439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:09 smithi116 ceph-mon[35411]: pgmap v11439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:11 smithi114 ceph-mon[30820]: pgmap v11440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:11 smithi116 ceph-mon[35411]: pgmap v11440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:13 smithi116 ceph-mon[35411]: pgmap v11441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:13 smithi114 ceph-mon[30820]: pgmap v11441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:15 smithi116 ceph-mon[35411]: pgmap v11442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:45:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:15 smithi114 ceph-mon[30820]: pgmap v11442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:45:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:17 smithi116 ceph-mon[35411]: pgmap v11443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:17 smithi114 ceph-mon[30820]: pgmap v11443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:19 smithi114 ceph-mon[30820]: pgmap v11444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:19 smithi116 ceph-mon[35411]: pgmap v11444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:21 smithi114 ceph-mon[30820]: pgmap v11445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:21 smithi116 ceph-mon[35411]: pgmap v11445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:23 smithi114 ceph-mon[30820]: pgmap v11446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:23 smithi116 ceph-mon[35411]: pgmap v11446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:25 smithi114 ceph-mon[30820]: pgmap v11447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:45:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:25 smithi116 ceph-mon[35411]: pgmap v11447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:45:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:27 smithi114 ceph-mon[30820]: pgmap v11448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:27 smithi116 ceph-mon[35411]: pgmap v11448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:29.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:29 smithi114 ceph-mon[30820]: pgmap v11449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:29 smithi116 ceph-mon[35411]: pgmap v11449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:45:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:45:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:31 smithi114 ceph-mon[30820]: pgmap v11450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:31 smithi116 ceph-mon[35411]: pgmap v11450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:32.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:45:32.628+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:45:32.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:45:32.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:45:32.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:45:32.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:45:32.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:45:32.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:45:32.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:45:32.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:45:32.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:45:32.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:45:32.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:45:32.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:45:32 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:45:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:33 smithi114 ceph-mon[30820]: pgmap v11451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:45:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:45:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:45:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:33 smithi116 ceph-mon[35411]: pgmap v11451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:45:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:45:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:45:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:45:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:45:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:34 smithi114 ceph-mon[30820]: pgmap v11452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:45:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:34 smithi114 ceph-mon[30820]: pgmap v11453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:45:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:45:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:45:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:45:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:45:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:45:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:45:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:45:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:45:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:45:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:45:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:45:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:34 smithi116 ceph-mon[35411]: pgmap v11452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:45:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:34 smithi116 ceph-mon[35411]: pgmap v11453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:45:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:36 smithi114 ceph-mon[30820]: pgmap v11454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:45:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:36 smithi116 ceph-mon[35411]: pgmap v11454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:45:38.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:38 smithi116 ceph-mon[35411]: pgmap v11455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:45:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:38 smithi114 ceph-mon[30820]: pgmap v11455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:45:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:40 smithi114 ceph-mon[30820]: pgmap v11456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:45:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:40 smithi116 ceph-mon[35411]: pgmap v11456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:45:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:42 smithi114 ceph-mon[30820]: pgmap v11457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:45:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:42 smithi116 ceph-mon[35411]: pgmap v11457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:45:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:44 smithi114 ceph-mon[30820]: pgmap v11458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:45:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:44 smithi116 ceph-mon[35411]: pgmap v11458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:45:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:45 smithi114 ceph-mon[30820]: pgmap v11459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:45 smithi116 ceph-mon[35411]: pgmap v11459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:47 smithi114 ceph-mon[30820]: pgmap v11460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:47 smithi116 ceph-mon[35411]: pgmap v11460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:49 smithi114 ceph-mon[30820]: pgmap v11461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:45:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:49 smithi116 ceph-mon[35411]: pgmap v11461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:45:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:51 smithi114 ceph-mon[30820]: pgmap v11462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:45:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:51 smithi116 ceph-mon[35411]: pgmap v11462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:45:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:53 smithi114 ceph-mon[30820]: pgmap v11463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:45:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:53 smithi116 ceph-mon[35411]: pgmap v11463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:45:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:55 smithi114 ceph-mon[30820]: pgmap v11464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:55 smithi116 ceph-mon[35411]: pgmap v11464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:45:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:45:57 smithi114 ceph-mon[30820]: pgmap v11465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:45:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:45:57 smithi116 ceph-mon[35411]: pgmap v11465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:00 smithi114 ceph-mon[30820]: pgmap v11466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:00 smithi116 ceph-mon[35411]: pgmap v11466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:02 smithi114 ceph-mon[30820]: pgmap v11467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:02 smithi116 ceph-mon[35411]: pgmap v11467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:04 smithi114 ceph-mon[30820]: pgmap v11468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:04 smithi116 ceph-mon[35411]: pgmap v11468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:06 smithi114 ceph-mon[30820]: pgmap v11469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:06 smithi116 ceph-mon[35411]: pgmap v11469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:08 smithi114 ceph-mon[30820]: pgmap v11470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:08 smithi116 ceph-mon[35411]: pgmap v11470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:10 smithi114 ceph-mon[30820]: pgmap v11471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:10 smithi116 ceph-mon[35411]: pgmap v11471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:12 smithi114 ceph-mon[30820]: pgmap v11472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:12 smithi116 ceph-mon[35411]: pgmap v11472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:14 smithi114 ceph-mon[30820]: pgmap v11473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:14 smithi116 ceph-mon[35411]: pgmap v11473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:16 smithi114 ceph-mon[30820]: pgmap v11474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:16 smithi116 ceph-mon[35411]: pgmap v11474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:18 smithi114 ceph-mon[30820]: pgmap v11475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:18 smithi116 ceph-mon[35411]: pgmap v11475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:20 smithi114 ceph-mon[30820]: pgmap v11476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:20 smithi116 ceph-mon[35411]: pgmap v11476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:22 smithi114 ceph-mon[30820]: pgmap v11477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:22 smithi116 ceph-mon[35411]: pgmap v11477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:24 smithi114 ceph-mon[30820]: pgmap v11478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:24 smithi116 ceph-mon[35411]: pgmap v11478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:26 smithi114 ceph-mon[30820]: pgmap v11479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:26 smithi116 ceph-mon[35411]: pgmap v11479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:28 smithi114 ceph-mon[30820]: pgmap v11480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:28 smithi116 ceph-mon[35411]: pgmap v11480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:30 smithi114 ceph-mon[30820]: pgmap v11481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:30 smithi116 ceph-mon[35411]: pgmap v11481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:32 smithi114 ceph-mon[30820]: pgmap v11482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:32 smithi116 ceph-mon[35411]: pgmap v11482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:46:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:46:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: pgmap v11483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:46:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:46:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: pgmap v11483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:46:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:46:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:46:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:46:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:46:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:35 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:46:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:35 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:46:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:36 smithi114 ceph-mon[30820]: pgmap v11484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:46:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:46:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:46:36.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:46:35.989+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:46:36.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:46:36.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:46:36.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:46:36.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:46:36.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:46:35 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:46:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:36 smithi116 ceph-mon[35411]: pgmap v11484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:46:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:46:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:46:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:37 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:46:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:46:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:46:37.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:46:37.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:46:37.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:46:37.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:46:37.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:46:37.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:46:37.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:46:37.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:37 smithi114 ceph-mon[30820]: pgmap v11485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:46:37.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:37 smithi114 ceph-mon[30820]: pgmap v11486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:46:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:37 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:46:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:46:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:46:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:46:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:46:37.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:46:37.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:46:37.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:46:37.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:46:37.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:46:37.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:37 smithi116 ceph-mon[35411]: pgmap v11485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:46:37.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:37 smithi116 ceph-mon[35411]: pgmap v11486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:46:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:39 smithi114 ceph-mon[30820]: pgmap v11487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:46:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:39 smithi116 ceph-mon[35411]: pgmap v11487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:46:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:41 smithi114 ceph-mon[30820]: pgmap v11488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:46:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:41 smithi116 ceph-mon[35411]: pgmap v11488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:46:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:43 smithi114 ceph-mon[30820]: pgmap v11489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:46:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:43 smithi116 ceph-mon[35411]: pgmap v11489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:46:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:45 smithi114 ceph-mon[30820]: pgmap v11490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:46:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:45 smithi116 ceph-mon[35411]: pgmap v11490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:46:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:47 smithi114 ceph-mon[30820]: pgmap v11491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:46:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:47 smithi116 ceph-mon[35411]: pgmap v11491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:46:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:49 smithi114 ceph-mon[30820]: pgmap v11492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:49 smithi116 ceph-mon[35411]: pgmap v11492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:51 smithi114 ceph-mon[30820]: pgmap v11493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:51 smithi116 ceph-mon[35411]: pgmap v11493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:46:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:53 smithi114 ceph-mon[30820]: pgmap v11494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:53 smithi116 ceph-mon[35411]: pgmap v11494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:46:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:55 smithi114 ceph-mon[30820]: pgmap v11495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:55 smithi116 ceph-mon[35411]: pgmap v11495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:57 smithi114 ceph-mon[30820]: pgmap v11496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:57 smithi116 ceph-mon[35411]: pgmap v11496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:46:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:46:59 smithi114 ceph-mon[30820]: pgmap v11497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:46:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:46:59 smithi116 ceph-mon[35411]: pgmap v11497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:01 smithi114 ceph-mon[30820]: pgmap v11498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:47:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:01 smithi116 ceph-mon[35411]: pgmap v11498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:47:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:03 smithi114 ceph-mon[30820]: pgmap v11499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:03 smithi116 ceph-mon[35411]: pgmap v11499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:05 smithi114 ceph-mon[30820]: pgmap v11500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:05 smithi116 ceph-mon[35411]: pgmap v11500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:07 smithi114 ceph-mon[30820]: pgmap v11501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:07 smithi116 ceph-mon[35411]: pgmap v11501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:09 smithi114 ceph-mon[30820]: pgmap v11502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:09 smithi116 ceph-mon[35411]: pgmap v11502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:11 smithi114 ceph-mon[30820]: pgmap v11503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:47:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:11 smithi116 ceph-mon[35411]: pgmap v11503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:47:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:13 smithi116 ceph-mon[35411]: pgmap v11504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:13 smithi114 ceph-mon[30820]: pgmap v11504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:15 smithi116 ceph-mon[35411]: pgmap v11505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:15 smithi114 ceph-mon[30820]: pgmap v11505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:17 smithi116 ceph-mon[35411]: pgmap v11506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:17 smithi114 ceph-mon[30820]: pgmap v11506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:19 smithi114 ceph-mon[30820]: pgmap v11507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:19 smithi116 ceph-mon[35411]: pgmap v11507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:21 smithi114 ceph-mon[30820]: pgmap v11508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:47:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:21 smithi116 ceph-mon[35411]: pgmap v11508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:47:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:23 smithi114 ceph-mon[30820]: pgmap v11509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:23 smithi116 ceph-mon[35411]: pgmap v11509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:25 smithi114 ceph-mon[30820]: pgmap v11510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:25 smithi116 ceph-mon[35411]: pgmap v11510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:27 smithi114 ceph-mon[30820]: pgmap v11511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:27 smithi116 ceph-mon[35411]: pgmap v11511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:29 smithi114 ceph-mon[30820]: pgmap v11512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:29 smithi116 ceph-mon[35411]: pgmap v11512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:31.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:31 smithi114 ceph-mon[30820]: pgmap v11513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:47:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:31 smithi116 ceph-mon[35411]: pgmap v11513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:47:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:33 smithi114 ceph-mon[30820]: pgmap v11514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:33 smithi116 ceph-mon[35411]: pgmap v11514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:35 smithi114 ceph-mon[30820]: pgmap v11515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:35 smithi116 ceph-mon[35411]: pgmap v11515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:47:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:47:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:37 smithi114 ceph-mon[30820]: pgmap v11516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:37 smithi116 ceph-mon[35411]: pgmap v11516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:39.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:39 smithi116 ceph-mon[35411]: pgmap v11517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-mon[30820]: pgmap v11517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:47:39.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:47:39.349+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:47:39.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:47:39.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:47:39.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:47:39.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:47:39.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:47:39.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:47:39.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:47:39.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:47:39.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:47:39.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:47:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:47:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:47:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:47:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:47:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:47:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:47:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:47:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:47:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:47:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:47:41.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:41 smithi114 ceph-mon[30820]: pgmap v11518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:47:41.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:41 smithi114 ceph-mon[30820]: pgmap v11519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:47:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:47:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:47:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:47:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:47:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:47:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:47:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:47:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:47:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:47:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:47:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:41 smithi116 ceph-mon[35411]: pgmap v11518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:47:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:41 smithi116 ceph-mon[35411]: pgmap v11519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:47:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:43 smithi114 ceph-mon[30820]: pgmap v11520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:47:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:43 smithi116 ceph-mon[35411]: pgmap v11520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:47:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:45 smithi114 ceph-mon[30820]: pgmap v11521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:47:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:45 smithi116 ceph-mon[35411]: pgmap v11521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:47:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:47 smithi114 ceph-mon[30820]: pgmap v11522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:47:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:47 smithi116 ceph-mon[35411]: pgmap v11522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T12:47:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:49 smithi114 ceph-mon[30820]: pgmap v11523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:47:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:49 smithi116 ceph-mon[35411]: pgmap v11523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:47:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:51 smithi114 ceph-mon[30820]: pgmap v11524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:47:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:51 smithi116 ceph-mon[35411]: pgmap v11524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:47:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:53 smithi114 ceph-mon[30820]: pgmap v11525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:53 smithi116 ceph-mon[35411]: pgmap v11525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:47:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:55 smithi114 ceph-mon[30820]: pgmap v11526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:47:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:55 smithi116 ceph-mon[35411]: pgmap v11526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:47:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:57 smithi114 ceph-mon[30820]: pgmap v11527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:57 smithi116 ceph-mon[35411]: pgmap v11527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:47:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:47:59 smithi114 ceph-mon[30820]: pgmap v11528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 5 op/s 2024-09-19T12:47:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:47:59 smithi116 ceph-mon[35411]: pgmap v11528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 5 op/s 2024-09-19T12:48:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:00 smithi114 ceph-mon[30820]: pgmap v11529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:48:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:00 smithi116 ceph-mon[35411]: pgmap v11529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:48:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:02 smithi116 ceph-mon[35411]: pgmap v11530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:02 smithi114 ceph-mon[30820]: pgmap v11530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:04 smithi116 ceph-mon[35411]: pgmap v11531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:04 smithi114 ceph-mon[30820]: pgmap v11531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:06 smithi116 ceph-mon[35411]: pgmap v11532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:06 smithi114 ceph-mon[30820]: pgmap v11532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:08 smithi114 ceph-mon[30820]: pgmap v11533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:08 smithi116 ceph-mon[35411]: pgmap v11533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:10.732 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:10 smithi116 ceph-mon[35411]: pgmap v11534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:48:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:10 smithi114 ceph-mon[30820]: pgmap v11534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:48:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:12 smithi114 ceph-mon[30820]: pgmap v11535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:12 smithi116 ceph-mon[35411]: pgmap v11535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:14 smithi114 ceph-mon[30820]: pgmap v11536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:14 smithi116 ceph-mon[35411]: pgmap v11536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:16 smithi114 ceph-mon[30820]: pgmap v11537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:48:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:16 smithi116 ceph-mon[35411]: pgmap v11537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:48:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:18 smithi114 ceph-mon[30820]: pgmap v11538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:18 smithi116 ceph-mon[35411]: pgmap v11538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:20 smithi114 ceph-mon[30820]: pgmap v11539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:48:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:20 smithi116 ceph-mon[35411]: pgmap v11539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:48:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:22 smithi114 ceph-mon[30820]: pgmap v11540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:22 smithi116 ceph-mon[35411]: pgmap v11540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:24 smithi114 ceph-mon[30820]: pgmap v11541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:24 smithi116 ceph-mon[35411]: pgmap v11541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:26 smithi114 ceph-mon[30820]: pgmap v11542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:48:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:26 smithi116 ceph-mon[35411]: pgmap v11542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:48:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:28 smithi114 ceph-mon[30820]: pgmap v11543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:28 smithi116 ceph-mon[35411]: pgmap v11543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:30 smithi114 ceph-mon[30820]: pgmap v11544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:48:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:30 smithi116 ceph-mon[35411]: pgmap v11544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:48:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:32 smithi114 ceph-mon[30820]: pgmap v11545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:32 smithi116 ceph-mon[35411]: pgmap v11545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:34 smithi114 ceph-mon[30820]: pgmap v11546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:34 smithi116 ceph-mon[35411]: pgmap v11546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:36 smithi114 ceph-mon[30820]: pgmap v11547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:48:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:36 smithi116 ceph-mon[35411]: pgmap v11547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:48:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:38 smithi114 ceph-mon[30820]: pgmap v11548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:38 smithi116 ceph-mon[35411]: pgmap v11548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:48:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:48:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:48:40.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:40 smithi116 ceph-mon[35411]: pgmap v11549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:48:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:40 smithi114 ceph-mon[30820]: pgmap v11549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:48:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:48:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:48:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:48:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:48:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-mon[30820]: pgmap v11550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:48:42.709+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:48:42.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:48:42.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:48:42 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:48:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:42 smithi116 ceph-mon[35411]: pgmap v11550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:43 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:48:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:48:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:48:43.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:43 smithi114 ceph-mon[30820]: pgmap v11551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:43 smithi114 ceph-mon[30820]: pgmap v11552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:48:43.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:48:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:43 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:48:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:48:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:48:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:48:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:48:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:48:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:48:43.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:48:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:48:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:48:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:48:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:48:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:43 smithi116 ceph-mon[35411]: pgmap v11551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:48:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:43 smithi116 ceph-mon[35411]: pgmap v11552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:48:43.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:48:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:46 smithi114 ceph-mon[30820]: pgmap v11553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:48:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:46 smithi116 ceph-mon[35411]: pgmap v11553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:48:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:48 smithi114 ceph-mon[30820]: pgmap v11554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:48:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:48 smithi116 ceph-mon[35411]: pgmap v11554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:48:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:50 smithi114 ceph-mon[30820]: pgmap v11555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:48:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:50 smithi116 ceph-mon[35411]: pgmap v11555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:48:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:52 smithi114 ceph-mon[30820]: pgmap v11556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:48:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:52 smithi116 ceph-mon[35411]: pgmap v11556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:48:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:54 smithi114 ceph-mon[30820]: pgmap v11557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:48:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:54 smithi116 ceph-mon[35411]: pgmap v11557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:48:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:56 smithi114 ceph-mon[30820]: pgmap v11558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:48:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:56 smithi116 ceph-mon[35411]: pgmap v11558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:48:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:48:58 smithi114 ceph-mon[30820]: pgmap v11559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:48:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:48:58 smithi116 ceph-mon[35411]: pgmap v11559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:00 smithi114 ceph-mon[30820]: pgmap v11560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:49:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:00 smithi116 ceph-mon[35411]: pgmap v11560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:49:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:02 smithi114 ceph-mon[30820]: pgmap v11561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:02 smithi116 ceph-mon[35411]: pgmap v11561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:04 smithi114 ceph-mon[30820]: pgmap v11562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:04 smithi116 ceph-mon[35411]: pgmap v11562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:06 smithi114 ceph-mon[30820]: pgmap v11563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:06 smithi116 ceph-mon[35411]: pgmap v11563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:08 smithi114 ceph-mon[30820]: pgmap v11564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:08 smithi116 ceph-mon[35411]: pgmap v11564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:10 smithi114 ceph-mon[30820]: pgmap v11565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:10 smithi116 ceph-mon[35411]: pgmap v11565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:12 smithi114 ceph-mon[30820]: pgmap v11566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:12 smithi116 ceph-mon[35411]: pgmap v11566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:14 smithi114 ceph-mon[30820]: pgmap v11567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:14 smithi116 ceph-mon[35411]: pgmap v11567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:16.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:16 smithi114 ceph-mon[30820]: pgmap v11568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:16 smithi116 ceph-mon[35411]: pgmap v11568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:18 smithi114 ceph-mon[30820]: pgmap v11569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:18 smithi116 ceph-mon[35411]: pgmap v11569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:20 smithi114 ceph-mon[30820]: pgmap v11570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:20 smithi116 ceph-mon[35411]: pgmap v11570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:22 smithi114 ceph-mon[30820]: pgmap v11571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:22 smithi116 ceph-mon[35411]: pgmap v11571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:24 smithi114 ceph-mon[30820]: pgmap v11572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:24 smithi116 ceph-mon[35411]: pgmap v11572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:26 smithi114 ceph-mon[30820]: pgmap v11573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:26 smithi116 ceph-mon[35411]: pgmap v11573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:28 smithi114 ceph-mon[30820]: pgmap v11574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:28 smithi116 ceph-mon[35411]: pgmap v11574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:30 smithi114 ceph-mon[30820]: pgmap v11575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:30 smithi116 ceph-mon[35411]: pgmap v11575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:32 smithi114 ceph-mon[30820]: pgmap v11576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:32 smithi116 ceph-mon[35411]: pgmap v11576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:34 smithi114 ceph-mon[30820]: pgmap v11577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:34 smithi116 ceph-mon[35411]: pgmap v11577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:36 smithi114 ceph-mon[30820]: pgmap v11578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:36 smithi116 ceph-mon[35411]: pgmap v11578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:38 smithi114 ceph-mon[30820]: pgmap v11579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:38 smithi116 ceph-mon[35411]: pgmap v11579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:49:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:40 smithi114 ceph-mon[30820]: pgmap v11580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:40 smithi116 ceph-mon[35411]: pgmap v11580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:42 smithi114 ceph-mon[30820]: pgmap v11581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:42 smithi116 ceph-mon[35411]: pgmap v11581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:49:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:49:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:49:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:44 smithi114 ceph-mon[30820]: pgmap v11582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:44 smithi116 ceph-mon[35411]: pgmap v11582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:45 smithi114 ceph-mon[30820]: pgmap v11583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:45 smithi116 ceph-mon[35411]: pgmap v11583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:49:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:49:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:49:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-mon[30820]: pgmap v11584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-mon[30820]: pgmap v11585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:49:46.068+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:49:46.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:49:46.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:49:46.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:49:46.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:49:46.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:49:46.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:49:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:49:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:49:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:49:46.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:46 smithi116 ceph-mon[35411]: pgmap v11584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:46 smithi116 ceph-mon[35411]: pgmap v11585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:49:46.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:49:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:49 smithi114 ceph-mon[30820]: pgmap v11586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:49:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:49 smithi116 ceph-mon[35411]: pgmap v11586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:49:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:51 smithi114 ceph-mon[30820]: pgmap v11587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:49:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:51 smithi116 ceph-mon[35411]: pgmap v11587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:49:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:53 smithi114 ceph-mon[30820]: pgmap v11588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:49:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:53 smithi116 ceph-mon[35411]: pgmap v11588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:49:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:55 smithi114 ceph-mon[30820]: pgmap v11589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:49:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:55 smithi116 ceph-mon[35411]: pgmap v11589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:49:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:57 smithi114 ceph-mon[30820]: pgmap v11590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:57 smithi116 ceph-mon[35411]: pgmap v11590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:49:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:49:59 smithi114 ceph-mon[30820]: pgmap v11591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:49:59 smithi116 ceph-mon[35411]: pgmap v11591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:50:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T12:50:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:01 smithi114 ceph-mon[30820]: pgmap v11592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:01 smithi116 ceph-mon[35411]: pgmap v11592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:03.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:03 smithi116 ceph-mon[35411]: pgmap v11593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:03 smithi114 ceph-mon[30820]: pgmap v11593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:05 smithi116 ceph-mon[35411]: pgmap v11594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:05 smithi114 ceph-mon[30820]: pgmap v11594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:07 smithi116 ceph-mon[35411]: pgmap v11595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:07 smithi114 ceph-mon[30820]: pgmap v11595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:09 smithi114 ceph-mon[30820]: pgmap v11596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:09 smithi116 ceph-mon[35411]: pgmap v11596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:11.482 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:11 smithi116 ceph-mon[35411]: pgmap v11597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:11 smithi114 ceph-mon[30820]: pgmap v11597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:13 smithi114 ceph-mon[30820]: pgmap v11598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:13 smithi116 ceph-mon[35411]: pgmap v11598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:15 smithi114 ceph-mon[30820]: pgmap v11599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:15 smithi116 ceph-mon[35411]: pgmap v11599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:17 smithi114 ceph-mon[30820]: pgmap v11600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:17 smithi116 ceph-mon[35411]: pgmap v11600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:19 smithi114 ceph-mon[30820]: pgmap v11601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:19 smithi116 ceph-mon[35411]: pgmap v11601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:21 smithi114 ceph-mon[30820]: pgmap v11602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:21 smithi116 ceph-mon[35411]: pgmap v11602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:23 smithi114 ceph-mon[30820]: pgmap v11603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:23 smithi116 ceph-mon[35411]: pgmap v11603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:25 smithi114 ceph-mon[30820]: pgmap v11604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:25 smithi116 ceph-mon[35411]: pgmap v11604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:27 smithi114 ceph-mon[30820]: pgmap v11605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:27 smithi116 ceph-mon[35411]: pgmap v11605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:29 smithi114 ceph-mon[30820]: pgmap v11606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:29 smithi116 ceph-mon[35411]: pgmap v11606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:31 smithi114 ceph-mon[30820]: pgmap v11607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:31 smithi116 ceph-mon[35411]: pgmap v11607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:33 smithi114 ceph-mon[30820]: pgmap v11608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:33 smithi116 ceph-mon[35411]: pgmap v11608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:35 smithi114 ceph-mon[30820]: pgmap v11609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:35 smithi116 ceph-mon[35411]: pgmap v11609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:37 smithi114 ceph-mon[30820]: pgmap v11610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:37 smithi116 ceph-mon[35411]: pgmap v11610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:39 smithi114 ceph-mon[30820]: pgmap v11611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:39 smithi116 ceph-mon[35411]: pgmap v11611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:41 smithi114 ceph-mon[30820]: pgmap v11612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:41 smithi116 ceph-mon[35411]: pgmap v11612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:50:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:43 smithi114 ceph-mon[30820]: pgmap v11613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:43 smithi116 ceph-mon[35411]: pgmap v11613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:50:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:45 smithi114 ceph-mon[30820]: pgmap v11614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:45 smithi116 ceph-mon[35411]: pgmap v11614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:46.554 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:50:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:50:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:47 smithi114 ceph-mon[30820]: pgmap v11615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:47 smithi116 ceph-mon[35411]: pgmap v11615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:50:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-mon[30820]: pgmap v11616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:50:49.365+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:50:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:50:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:50:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:50:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:50:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:50:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:50:49.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:50:49.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:50:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:50:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:49 smithi116 ceph-mon[35411]: pgmap v11616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:50:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:50:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:50:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:50:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:50:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:50:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:50:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:50:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:50:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:50:51.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:50:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:50:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:50:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:50:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:50:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:50:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:50:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:51 smithi114 ceph-mon[30820]: pgmap v11617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T12:50:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:51 smithi114 ceph-mon[30820]: pgmap v11618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:50:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:50:51.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:50:51.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:50:51.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:51 smithi116 ceph-mon[35411]: pgmap v11617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T12:50:51.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:51 smithi116 ceph-mon[35411]: pgmap v11618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:50:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:53 smithi114 ceph-mon[30820]: pgmap v11619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:50:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:53 smithi116 ceph-mon[35411]: pgmap v11619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:50:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:55 smithi114 ceph-mon[30820]: pgmap v11620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:50:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:55 smithi116 ceph-mon[35411]: pgmap v11620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:50:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:57 smithi114 ceph-mon[30820]: pgmap v11621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:50:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:57 smithi116 ceph-mon[35411]: pgmap v11621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:50:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:50:59 smithi114 ceph-mon[30820]: pgmap v11622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:50:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:50:59 smithi116 ceph-mon[35411]: pgmap v11622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:51:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:01 smithi114 ceph-mon[30820]: pgmap v11623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:51:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:01 smithi116 ceph-mon[35411]: pgmap v11623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:51:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:03 smithi114 ceph-mon[30820]: pgmap v11624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:03 smithi116 ceph-mon[35411]: pgmap v11624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:05 smithi114 ceph-mon[30820]: pgmap v11625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:05 smithi116 ceph-mon[35411]: pgmap v11625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:06 smithi114 ceph-mon[30820]: pgmap v11626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:06 smithi116 ceph-mon[35411]: pgmap v11626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:08 smithi114 ceph-mon[30820]: pgmap v11627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:51:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:08 smithi116 ceph-mon[35411]: pgmap v11627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:51:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:10 smithi114 ceph-mon[30820]: pgmap v11628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:10 smithi116 ceph-mon[35411]: pgmap v11628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:12.732 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:12 smithi116 ceph-mon[35411]: pgmap v11629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:12 smithi114 ceph-mon[30820]: pgmap v11629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:14 smithi114 ceph-mon[30820]: pgmap v11630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:14 smithi116 ceph-mon[35411]: pgmap v11630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:16 smithi114 ceph-mon[30820]: pgmap v11631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:16 smithi116 ceph-mon[35411]: pgmap v11631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:18 smithi114 ceph-mon[30820]: pgmap v11632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:18 smithi116 ceph-mon[35411]: pgmap v11632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:20 smithi114 ceph-mon[30820]: pgmap v11633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:20 smithi116 ceph-mon[35411]: pgmap v11633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:22 smithi114 ceph-mon[30820]: pgmap v11634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:22 smithi116 ceph-mon[35411]: pgmap v11634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:24 smithi114 ceph-mon[30820]: pgmap v11635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:24 smithi116 ceph-mon[35411]: pgmap v11635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:26 smithi114 ceph-mon[30820]: pgmap v11636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:26 smithi116 ceph-mon[35411]: pgmap v11636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:28 smithi114 ceph-mon[30820]: pgmap v11637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:28 smithi116 ceph-mon[35411]: pgmap v11637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:30 smithi114 ceph-mon[30820]: pgmap v11638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:30 smithi116 ceph-mon[35411]: pgmap v11638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:32 smithi114 ceph-mon[30820]: pgmap v11639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:32 smithi116 ceph-mon[35411]: pgmap v11639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:34 smithi114 ceph-mon[30820]: pgmap v11640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:34 smithi116 ceph-mon[35411]: pgmap v11640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:36 smithi114 ceph-mon[30820]: pgmap v11641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:36 smithi116 ceph-mon[35411]: pgmap v11641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:38 smithi114 ceph-mon[30820]: pgmap v11642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:38 smithi116 ceph-mon[35411]: pgmap v11642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:40 smithi114 ceph-mon[30820]: pgmap v11643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:40 smithi116 ceph-mon[35411]: pgmap v11643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:42 smithi114 ceph-mon[30820]: pgmap v11644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:42 smithi116 ceph-mon[35411]: pgmap v11644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:44 smithi114 ceph-mon[30820]: pgmap v11645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:44 smithi116 ceph-mon[35411]: pgmap v11645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:46.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:46 smithi114 ceph-mon[30820]: pgmap v11646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:46 smithi116 ceph-mon[35411]: pgmap v11646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:51:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:48 smithi114 ceph-mon[30820]: pgmap v11647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:48 smithi116 ceph-mon[35411]: pgmap v11647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:51:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:51:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:51:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:50 smithi114 ceph-mon[30820]: pgmap v11648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:50 smithi116 ceph-mon[35411]: pgmap v11648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:51:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-mon[30820]: pgmap v11649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:52.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:51:52.725+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:51:52.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:51:52.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:51:52.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:51:52.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:51:52.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:51:52.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:51:52.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:51:52.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:51:52.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:51:52 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:51:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:52 smithi116 ceph-mon[35411]: pgmap v11649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:51:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:53 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:51:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:51:53.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:53 smithi114 ceph-mon[30820]: pgmap v11650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:51:53.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:53 smithi114 ceph-mon[30820]: pgmap v11651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:51:53.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:51:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:53 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:51:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:51:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:51:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:51:53.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:53 smithi116 ceph-mon[35411]: pgmap v11650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:53 smithi116 ceph-mon[35411]: pgmap v11651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:51:53.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:51:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:55 smithi114 ceph-mon[30820]: pgmap v11652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:51:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:55 smithi116 ceph-mon[35411]: pgmap v11652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:51:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:57 smithi114 ceph-mon[30820]: pgmap v11653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:51:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:57 smithi116 ceph-mon[35411]: pgmap v11653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:52:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:51:59 smithi114 ceph-mon[30820]: pgmap v11654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:52:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:51:59 smithi116 ceph-mon[35411]: pgmap v11654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:52:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:01 smithi114 ceph-mon[30820]: pgmap v11655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:52:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:01 smithi116 ceph-mon[35411]: pgmap v11655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:52:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:03 smithi114 ceph-mon[30820]: pgmap v11656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:52:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:03 smithi116 ceph-mon[35411]: pgmap v11656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:52:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:05 smithi114 ceph-mon[30820]: pgmap v11657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:05 smithi116 ceph-mon[35411]: pgmap v11657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:07 smithi114 ceph-mon[30820]: pgmap v11658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:07 smithi116 ceph-mon[35411]: pgmap v11658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:09 smithi114 ceph-mon[30820]: pgmap v11659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:09 smithi116 ceph-mon[35411]: pgmap v11659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:11 smithi114 ceph-mon[30820]: pgmap v11660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:11 smithi116 ceph-mon[35411]: pgmap v11660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:13 smithi114 ceph-mon[30820]: pgmap v11661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:13 smithi116 ceph-mon[35411]: pgmap v11661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:15 smithi114 ceph-mon[30820]: pgmap v11662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:15 smithi116 ceph-mon[35411]: pgmap v11662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:17 smithi114 ceph-mon[30820]: pgmap v11663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:17 smithi116 ceph-mon[35411]: pgmap v11663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:19 smithi114 ceph-mon[30820]: pgmap v11664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:19 smithi116 ceph-mon[35411]: pgmap v11664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:21 smithi114 ceph-mon[30820]: pgmap v11665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:21 smithi116 ceph-mon[35411]: pgmap v11665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:23 smithi114 ceph-mon[30820]: pgmap v11666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:23 smithi116 ceph-mon[35411]: pgmap v11666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:25 smithi114 ceph-mon[30820]: pgmap v11667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:25 smithi116 ceph-mon[35411]: pgmap v11667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:27 smithi114 ceph-mon[30820]: pgmap v11668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:27 smithi116 ceph-mon[35411]: pgmap v11668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:29 smithi114 ceph-mon[30820]: pgmap v11669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:29 smithi116 ceph-mon[35411]: pgmap v11669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:31 smithi114 ceph-mon[30820]: pgmap v11670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:31 smithi116 ceph-mon[35411]: pgmap v11670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:33 smithi114 ceph-mon[30820]: pgmap v11671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:33 smithi116 ceph-mon[35411]: pgmap v11671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:35 smithi114 ceph-mon[30820]: pgmap v11672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:35 smithi116 ceph-mon[35411]: pgmap v11672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:37 smithi114 ceph-mon[30820]: pgmap v11673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:37 smithi116 ceph-mon[35411]: pgmap v11673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:39 smithi114 ceph-mon[30820]: pgmap v11674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:39 smithi116 ceph-mon[35411]: pgmap v11674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:41 smithi114 ceph-mon[30820]: pgmap v11675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:41 smithi116 ceph-mon[35411]: pgmap v11675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:43 smithi114 ceph-mon[30820]: pgmap v11676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:43 smithi116 ceph-mon[35411]: pgmap v11676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:45 smithi114 ceph-mon[30820]: pgmap v11677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:45 smithi116 ceph-mon[35411]: pgmap v11677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:47 smithi114 ceph-mon[30820]: pgmap v11678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:47 smithi116 ceph-mon[35411]: pgmap v11678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:52:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:49 smithi114 ceph-mon[30820]: pgmap v11679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:49 smithi116 ceph-mon[35411]: pgmap v11679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:51 smithi114 ceph-mon[30820]: pgmap v11680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:51 smithi116 ceph-mon[35411]: pgmap v11680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:52:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:52:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:52:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:53 smithi114 ceph-mon[30820]: pgmap v11681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:53 smithi116 ceph-mon[35411]: pgmap v11681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:52:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:55 smithi114 ceph-mon[30820]: pgmap v11682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:52:56.084+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:52:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:52:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:52:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:52:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:52:56.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:52:56.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:52:56.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:52:56.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:52:56.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:52:56.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:52:56.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:52:56.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:52:56.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:52:56.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:52:56.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:52:56.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:52:56.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:52:56.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:52:56.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:52:56.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:52:56.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:52:56.172 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:52:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:55 smithi116 ceph-mon[35411]: pgmap v11682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:52:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:52:57.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:52:57.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:52:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:52:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-mon[30820]: pgmap v11683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:52:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-mon[30820]: pgmap v11684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:52:57.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:52:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:52:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:52:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:52:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:52:57.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:56 smithi116 ceph-mon[35411]: pgmap v11683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:56 smithi116 ceph-mon[35411]: pgmap v11684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:52:57.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:52:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:52:59 smithi114 ceph-mon[30820]: pgmap v11685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:52:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:52:59 smithi116 ceph-mon[35411]: pgmap v11685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:53:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:01 smithi114 ceph-mon[30820]: pgmap v11686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:53:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:01 smithi116 ceph-mon[35411]: pgmap v11686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:53:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:03 smithi114 ceph-mon[30820]: pgmap v11687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:53:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:03 smithi116 ceph-mon[35411]: pgmap v11687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:53:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:05 smithi114 ceph-mon[30820]: pgmap v11688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:53:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:05 smithi116 ceph-mon[35411]: pgmap v11688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:53:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:07 smithi114 ceph-mon[30820]: pgmap v11689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:53:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:07 smithi116 ceph-mon[35411]: pgmap v11689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:53:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:09 smithi116 ceph-mon[35411]: pgmap v11690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:09 smithi114 ceph-mon[30820]: pgmap v11690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:11 smithi116 ceph-mon[35411]: pgmap v11691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:11 smithi114 ceph-mon[30820]: pgmap v11691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:13 smithi116 ceph-mon[35411]: pgmap v11692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:13 smithi114 ceph-mon[30820]: pgmap v11692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:15 smithi114 ceph-mon[30820]: pgmap v11693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:15 smithi116 ceph-mon[35411]: pgmap v11693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:17 smithi114 ceph-mon[30820]: pgmap v11694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:17 smithi116 ceph-mon[35411]: pgmap v11694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:19 smithi114 ceph-mon[30820]: pgmap v11695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:19 smithi116 ceph-mon[35411]: pgmap v11695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:21 smithi114 ceph-mon[30820]: pgmap v11696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:21 smithi116 ceph-mon[35411]: pgmap v11696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:23 smithi114 ceph-mon[30820]: pgmap v11697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:23 smithi116 ceph-mon[35411]: pgmap v11697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:25 smithi114 ceph-mon[30820]: pgmap v11698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:25 smithi116 ceph-mon[35411]: pgmap v11698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:27 smithi114 ceph-mon[30820]: pgmap v11699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:27 smithi116 ceph-mon[35411]: pgmap v11699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:29 smithi114 ceph-mon[30820]: pgmap v11700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:29 smithi116 ceph-mon[35411]: pgmap v11700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:31 smithi114 ceph-mon[30820]: pgmap v11701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:31 smithi116 ceph-mon[35411]: pgmap v11701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:33 smithi114 ceph-mon[30820]: pgmap v11702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:33 smithi116 ceph-mon[35411]: pgmap v11702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:35.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:35 smithi114 ceph-mon[30820]: pgmap v11703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:35 smithi116 ceph-mon[35411]: pgmap v11703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:37 smithi114 ceph-mon[30820]: pgmap v11704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:37 smithi116 ceph-mon[35411]: pgmap v11704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:39 smithi114 ceph-mon[30820]: pgmap v11705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:39 smithi116 ceph-mon[35411]: pgmap v11705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:41 smithi114 ceph-mon[30820]: pgmap v11706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:41 smithi116 ceph-mon[35411]: pgmap v11706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:43 smithi114 ceph-mon[30820]: pgmap v11707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:43 smithi116 ceph-mon[35411]: pgmap v11707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:45 smithi114 ceph-mon[30820]: pgmap v11708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:45 smithi116 ceph-mon[35411]: pgmap v11708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:47 smithi114 ceph-mon[30820]: pgmap v11709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:47 smithi116 ceph-mon[35411]: pgmap v11709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:49 smithi114 ceph-mon[30820]: pgmap v11710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:49 smithi116 ceph-mon[35411]: pgmap v11710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:51 smithi114 ceph-mon[30820]: pgmap v11711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:51 smithi116 ceph-mon[35411]: pgmap v11711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:53:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:53 smithi114 ceph-mon[30820]: pgmap v11712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:53 smithi116 ceph-mon[35411]: pgmap v11712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:53:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:55 smithi114 ceph-mon[30820]: pgmap v11713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:55 smithi116 ceph-mon[35411]: pgmap v11713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:53:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:53:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:57 smithi114 ceph-mon[30820]: pgmap v11714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:57 smithi116 ceph-mon[35411]: pgmap v11714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:53:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-mon[30820]: pgmap v11715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:53:59.445+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:53:59.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:53:59.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:53:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:53:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:53:59 smithi116 ceph-mon[35411]: pgmap v11715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:54:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:54:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:54:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:54:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:54:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:54:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:01 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:54:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:54:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:54:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:54:01.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:54:01.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:54:01.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:54:01.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:54:01.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:54:01.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:54:01.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:54:01.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:01 smithi114 ceph-mon[30820]: pgmap v11716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:54:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:01 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:54:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:54:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:54:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:54:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:54:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:54:01.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:54:01.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:54:01.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:54:01.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:54:01.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:54:01.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:01 smithi116 ceph-mon[35411]: pgmap v11716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:54:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:03 smithi114 ceph-mon[30820]: pgmap v11717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:54:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:03 smithi116 ceph-mon[35411]: pgmap v11717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:54:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:05 smithi114 ceph-mon[30820]: pgmap v11718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:54:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:05 smithi116 ceph-mon[35411]: pgmap v11718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:54:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:07 smithi114 ceph-mon[30820]: pgmap v11719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T12:54:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:07 smithi116 ceph-mon[35411]: pgmap v11719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T12:54:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:09 smithi114 ceph-mon[30820]: pgmap v11720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:54:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:09 smithi116 ceph-mon[35411]: pgmap v11720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:54:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:11 smithi114 ceph-mon[30820]: pgmap v11721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:54:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:11 smithi116 ceph-mon[35411]: pgmap v11721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T12:54:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:13 smithi114 ceph-mon[30820]: pgmap v11722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:13 smithi116 ceph-mon[35411]: pgmap v11722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:15 smithi114 ceph-mon[30820]: pgmap v11723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:15 smithi116 ceph-mon[35411]: pgmap v11723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:17 smithi114 ceph-mon[30820]: pgmap v11724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:17 smithi116 ceph-mon[35411]: pgmap v11724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:19 smithi114 ceph-mon[30820]: pgmap v11725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:19 smithi116 ceph-mon[35411]: pgmap v11725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:21 smithi114 ceph-mon[30820]: pgmap v11726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:54:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:21 smithi116 ceph-mon[35411]: pgmap v11726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:54:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:23 smithi114 ceph-mon[30820]: pgmap v11727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:23 smithi116 ceph-mon[35411]: pgmap v11727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:25 smithi114 ceph-mon[30820]: pgmap v11728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:25 smithi116 ceph-mon[35411]: pgmap v11728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:27 smithi114 ceph-mon[30820]: pgmap v11729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:27 smithi116 ceph-mon[35411]: pgmap v11729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:29 smithi114 ceph-mon[30820]: pgmap v11730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:29 smithi116 ceph-mon[35411]: pgmap v11730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:31 smithi114 ceph-mon[30820]: pgmap v11731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:54:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:31 smithi116 ceph-mon[35411]: pgmap v11731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:54:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:32 smithi114 ceph-mon[30820]: pgmap v11732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:32 smithi116 ceph-mon[35411]: pgmap v11732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:34 smithi114 ceph-mon[30820]: pgmap v11733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:34 smithi116 ceph-mon[35411]: pgmap v11733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:36 smithi114 ceph-mon[30820]: pgmap v11734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:36 smithi116 ceph-mon[35411]: pgmap v11734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:38 smithi114 ceph-mon[30820]: pgmap v11735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:38 smithi116 ceph-mon[35411]: pgmap v11735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:40 smithi114 ceph-mon[30820]: pgmap v11736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:54:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:40 smithi116 ceph-mon[35411]: pgmap v11736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:54:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:42 smithi114 ceph-mon[30820]: pgmap v11737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:42 smithi116 ceph-mon[35411]: pgmap v11737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:44 smithi114 ceph-mon[30820]: pgmap v11738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:44 smithi116 ceph-mon[35411]: pgmap v11738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:46 smithi114 ceph-mon[30820]: pgmap v11739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:46 smithi116 ceph-mon[35411]: pgmap v11739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:48 smithi114 ceph-mon[30820]: pgmap v11740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:48 smithi116 ceph-mon[35411]: pgmap v11740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:50 smithi114 ceph-mon[30820]: pgmap v11741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:54:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:50 smithi116 ceph-mon[35411]: pgmap v11741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:54:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:52 smithi114 ceph-mon[30820]: pgmap v11742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:52 smithi116 ceph-mon[35411]: pgmap v11742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:54:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:54 smithi114 ceph-mon[30820]: pgmap v11743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:54 smithi116 ceph-mon[35411]: pgmap v11743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:56 smithi114 ceph-mon[30820]: pgmap v11744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:56 smithi116 ceph-mon[35411]: pgmap v11744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:54:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:54:58 smithi114 ceph-mon[30820]: pgmap v11745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:54:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:54:58 smithi116 ceph-mon[35411]: pgmap v11745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:00.636 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:00 smithi116 ceph-mon[35411]: pgmap v11746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:00.636 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:55:00.912 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:00 smithi114 ceph-mon[30820]: pgmap v11746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:00.912 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:55:02.633 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:02 smithi116 ceph-mon[35411]: pgmap v11747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:02.809 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-mon[30820]: pgmap v11747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:02.809 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:55:02.804+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:55:02.809 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:55:02.809 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:55:02.809 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:55:02.809 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:55:02.810 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:55:02.811 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:55:02.811 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:55:02.811 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:55:02.811 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:55:02 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:55:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:55:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:55:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:55:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:55:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:55:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:55:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:55:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:55:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:55:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:55:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:55:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:55:03.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:03.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:05 smithi114 ceph-mon[30820]: pgmap v11748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:55:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:55:05.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:05 smithi116 ceph-mon[35411]: pgmap v11748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:55:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:55:05.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:55:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:06 smithi114 ceph-mon[30820]: pgmap v11749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T12:55:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:06 smithi114 ceph-mon[30820]: pgmap v11750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:55:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:06 smithi116 ceph-mon[35411]: pgmap v11749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T12:55:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:06 smithi116 ceph-mon[35411]: pgmap v11750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:55:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:08 smithi114 ceph-mon[30820]: pgmap v11751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:55:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:08 smithi116 ceph-mon[35411]: pgmap v11751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:55:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:10 smithi114 ceph-mon[30820]: pgmap v11752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T12:55:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:10 smithi116 ceph-mon[35411]: pgmap v11752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T12:55:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:12 smithi114 ceph-mon[30820]: pgmap v11753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:55:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:12 smithi116 ceph-mon[35411]: pgmap v11753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:14 smithi114 ceph-mon[30820]: pgmap v11754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:14 smithi116 ceph-mon[35411]: pgmap v11754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T12:55:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:16 smithi114 ceph-mon[30820]: pgmap v11755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:55:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:16 smithi116 ceph-mon[35411]: pgmap v11755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:55:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:18 smithi114 ceph-mon[30820]: pgmap v11756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:18 smithi116 ceph-mon[35411]: pgmap v11756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:20 smithi114 ceph-mon[30820]: pgmap v11757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:20 smithi116 ceph-mon[35411]: pgmap v11757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:22 smithi114 ceph-mon[30820]: pgmap v11758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:55:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:22 smithi116 ceph-mon[35411]: pgmap v11758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:55:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:24 smithi116 ceph-mon[35411]: pgmap v11759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:24 smithi114 ceph-mon[30820]: pgmap v11759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:26 smithi116 ceph-mon[35411]: pgmap v11760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:26 smithi114 ceph-mon[30820]: pgmap v11760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:28 smithi116 ceph-mon[35411]: pgmap v11761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:28 smithi114 ceph-mon[30820]: pgmap v11761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:30.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:30 smithi114 ceph-mon[30820]: pgmap v11762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:30 smithi116 ceph-mon[35411]: pgmap v11762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:32 smithi114 ceph-mon[30820]: pgmap v11763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:55:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:32 smithi116 ceph-mon[35411]: pgmap v11763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:55:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:34 smithi114 ceph-mon[30820]: pgmap v11764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:34 smithi116 ceph-mon[35411]: pgmap v11764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:36.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:36 smithi114 ceph-mon[30820]: pgmap v11765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:36 smithi116 ceph-mon[35411]: pgmap v11765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:38 smithi114 ceph-mon[30820]: pgmap v11766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:38 smithi116 ceph-mon[35411]: pgmap v11766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:40 smithi114 ceph-mon[30820]: pgmap v11767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:40 smithi116 ceph-mon[35411]: pgmap v11767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:42 smithi114 ceph-mon[30820]: pgmap v11768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:55:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:42 smithi116 ceph-mon[35411]: pgmap v11768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:55:44.482 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:44 smithi116 ceph-mon[35411]: pgmap v11769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:44 smithi114 ceph-mon[30820]: pgmap v11769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:46 smithi114 ceph-mon[30820]: pgmap v11770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:46 smithi116 ceph-mon[35411]: pgmap v11770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:48 smithi114 ceph-mon[30820]: pgmap v11771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:48 smithi116 ceph-mon[35411]: pgmap v11771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:50 smithi114 ceph-mon[30820]: pgmap v11772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:50 smithi116 ceph-mon[35411]: pgmap v11772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:52 smithi114 ceph-mon[30820]: pgmap v11773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:55:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:52 smithi116 ceph-mon[35411]: pgmap v11773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:55:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:54 smithi114 ceph-mon[30820]: pgmap v11774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:54 smithi116 ceph-mon[35411]: pgmap v11774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:55:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:56 smithi114 ceph-mon[30820]: pgmap v11775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:56 smithi116 ceph-mon[35411]: pgmap v11775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:55:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:55:58 smithi114 ceph-mon[30820]: pgmap v11776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:55:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:55:58 smithi116 ceph-mon[35411]: pgmap v11776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:56:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:00 smithi114 ceph-mon[30820]: pgmap v11777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:00 smithi116 ceph-mon[35411]: pgmap v11777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:02 smithi114 ceph-mon[30820]: pgmap v11778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:02 smithi116 ceph-mon[35411]: pgmap v11778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:04 smithi114 ceph-mon[30820]: pgmap v11779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:04 smithi116 ceph-mon[35411]: pgmap v11779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:56:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:56:06.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-mon[30820]: pgmap v11780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:56:06.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:56:06.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:56:06.568 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:56:06.569 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:56:06.228+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:56:06.569 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:56:06.569 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:56:06.569 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:56:06.569 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:56:06.569 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:56:06.569 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:56:06.570 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:56:06.571 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:56:06.571 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:56:06.571 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:56:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:56:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:06 smithi116 ceph-mon[35411]: pgmap v11780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:56:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:56:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:56:06.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:56:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:07 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:56:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:56:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:56:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:56:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:56:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:56:07.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:56:07.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:56:07.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:56:07.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:56:07.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:07 smithi114 ceph-mon[30820]: pgmap v11781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T12:56:07.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:07 smithi114 ceph-mon[30820]: pgmap v11782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:56:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:07 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:56:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:56:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:56:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:56:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:56:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:56:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:56:07.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:56:07.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:56:07.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:56:07.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:56:07.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:56:07.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:56:07.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:07 smithi116 ceph-mon[35411]: pgmap v11781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 6 op/s 2024-09-19T12:56:07.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:07 smithi116 ceph-mon[35411]: pgmap v11782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:56:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:09 smithi114 ceph-mon[30820]: pgmap v11783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:56:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:09 smithi116 ceph-mon[35411]: pgmap v11783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:56:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:11 smithi114 ceph-mon[30820]: pgmap v11784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:56:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:11 smithi116 ceph-mon[35411]: pgmap v11784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:56:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:13 smithi114 ceph-mon[30820]: pgmap v11785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:56:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:13 smithi116 ceph-mon[35411]: pgmap v11785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T12:56:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:15 smithi114 ceph-mon[30820]: pgmap v11786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T12:56:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:15 smithi116 ceph-mon[35411]: pgmap v11786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T12:56:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:17 smithi114 ceph-mon[30820]: pgmap v11787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:56:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:17 smithi116 ceph-mon[35411]: pgmap v11787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:56:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:19 smithi114 ceph-mon[30820]: pgmap v11788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:19 smithi116 ceph-mon[35411]: pgmap v11788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:21 smithi114 ceph-mon[30820]: pgmap v11789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:56:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:21 smithi116 ceph-mon[35411]: pgmap v11789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:56:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:23 smithi114 ceph-mon[30820]: pgmap v11790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:56:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:23 smithi116 ceph-mon[35411]: pgmap v11790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:56:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:25 smithi114 ceph-mon[30820]: pgmap v11791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:25 smithi116 ceph-mon[35411]: pgmap v11791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:27 smithi114 ceph-mon[30820]: pgmap v11792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:27 smithi116 ceph-mon[35411]: pgmap v11792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:29 smithi114 ceph-mon[30820]: pgmap v11793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:29 smithi116 ceph-mon[35411]: pgmap v11793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:31 smithi114 ceph-mon[30820]: pgmap v11794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:56:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:31 smithi116 ceph-mon[35411]: pgmap v11794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:56:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:33 smithi114 ceph-mon[30820]: pgmap v11795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:56:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:33 smithi116 ceph-mon[35411]: pgmap v11795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:56:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:35 smithi114 ceph-mon[30820]: pgmap v11796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:35 smithi116 ceph-mon[35411]: pgmap v11796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:37 smithi114 ceph-mon[30820]: pgmap v11797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:37 smithi116 ceph-mon[35411]: pgmap v11797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:39 smithi114 ceph-mon[30820]: pgmap v11798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:39 smithi116 ceph-mon[35411]: pgmap v11798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:41 smithi114 ceph-mon[30820]: pgmap v11799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:56:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:41 smithi116 ceph-mon[35411]: pgmap v11799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:56:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:43 smithi114 ceph-mon[30820]: pgmap v11800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:56:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:43 smithi116 ceph-mon[35411]: pgmap v11800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:56:45.166 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:45 smithi114 ceph-mon[30820]: pgmap v11801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:45 smithi116 ceph-mon[35411]: pgmap v11801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:47 smithi114 ceph-mon[30820]: pgmap v11802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:47 smithi116 ceph-mon[35411]: pgmap v11802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:49 smithi114 ceph-mon[30820]: pgmap v11803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:49 smithi116 ceph-mon[35411]: pgmap v11803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:51 smithi114 ceph-mon[30820]: pgmap v11804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:56:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:51 smithi116 ceph-mon[35411]: pgmap v11804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:56:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:53 smithi114 ceph-mon[30820]: pgmap v11805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:56:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:53 smithi116 ceph-mon[35411]: pgmap v11805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:56:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:55 smithi114 ceph-mon[30820]: pgmap v11806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:55 smithi116 ceph-mon[35411]: pgmap v11806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:57 smithi114 ceph-mon[30820]: pgmap v11807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:57 smithi116 ceph-mon[35411]: pgmap v11807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:56:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:56:59 smithi114 ceph-mon[30820]: pgmap v11808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:56:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:56:59 smithi116 ceph-mon[35411]: pgmap v11808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:01 smithi114 ceph-mon[30820]: pgmap v11809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:57:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:01 smithi116 ceph-mon[35411]: pgmap v11809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:57:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:03 smithi114 ceph-mon[30820]: pgmap v11810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:57:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:03 smithi116 ceph-mon[35411]: pgmap v11810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:57:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:05 smithi116 ceph-mon[35411]: pgmap v11811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:05 smithi114 ceph-mon[30820]: pgmap v11811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: pgmap v11812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: pgmap v11812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:57:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:07.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T12:57:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:08 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:57:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:08 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:57:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:08 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:57:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:08 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:57:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:08 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T12:57:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:08 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T12:57:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:08 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T12:57:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:08 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T12:57:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-mon[30820]: pgmap v11813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:57:09.524+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:57:09.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:57:09.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:57:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:57:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:09 smithi116 ceph-mon[35411]: pgmap v11813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:57:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:57:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:57:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:57:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:57:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:57:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:11 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:57:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:57:11.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:11 smithi114 ceph-mon[30820]: pgmap v11814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T12:57:11.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:11 smithi114 ceph-mon[30820]: pgmap v11815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T12:57:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:11 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:57:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:57:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:57:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:11 smithi116 ceph-mon[35411]: pgmap v11814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T12:57:11.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:11 smithi116 ceph-mon[35411]: pgmap v11815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T12:57:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:13 smithi114 ceph-mon[30820]: pgmap v11816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:57:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:13 smithi116 ceph-mon[35411]: pgmap v11816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:57:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:15 smithi114 ceph-mon[30820]: pgmap v11817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:57:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:15 smithi116 ceph-mon[35411]: pgmap v11817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:57:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:17 smithi114 ceph-mon[30820]: pgmap v11818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:57:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:17 smithi116 ceph-mon[35411]: pgmap v11818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:57:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:19 smithi114 ceph-mon[30820]: pgmap v11819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:57:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:19 smithi116 ceph-mon[35411]: pgmap v11819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:57:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:21 smithi114 ceph-mon[30820]: pgmap v11820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:57:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:21 smithi116 ceph-mon[35411]: pgmap v11820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:57:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:23 smithi114 ceph-mon[30820]: pgmap v11821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:57:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:23 smithi116 ceph-mon[35411]: pgmap v11821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:57:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:25 smithi114 ceph-mon[30820]: pgmap v11822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:25 smithi116 ceph-mon[35411]: pgmap v11822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:27 smithi114 ceph-mon[30820]: pgmap v11823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:27 smithi116 ceph-mon[35411]: pgmap v11823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:29 smithi114 ceph-mon[30820]: pgmap v11824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:29 smithi116 ceph-mon[35411]: pgmap v11824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:31 smithi114 ceph-mon[30820]: pgmap v11825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:57:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:31 smithi116 ceph-mon[35411]: pgmap v11825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:57:33.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:33 smithi114 ceph-mon[30820]: pgmap v11826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:57:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:33 smithi116 ceph-mon[35411]: pgmap v11826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:57:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:34 smithi116 ceph-mon[35411]: pgmap v11827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:34 smithi114 ceph-mon[30820]: pgmap v11827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:36 smithi114 ceph-mon[30820]: pgmap v11828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:36 smithi116 ceph-mon[35411]: pgmap v11828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:38 smithi114 ceph-mon[30820]: pgmap v11829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:38 smithi116 ceph-mon[35411]: pgmap v11829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:40 smithi114 ceph-mon[30820]: pgmap v11830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:57:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:40 smithi116 ceph-mon[35411]: pgmap v11830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:57:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:42 smithi114 ceph-mon[30820]: pgmap v11831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:57:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:42 smithi116 ceph-mon[35411]: pgmap v11831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:57:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:44 smithi114 ceph-mon[30820]: pgmap v11832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:57:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:44 smithi116 ceph-mon[35411]: pgmap v11832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:57:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:46 smithi114 ceph-mon[30820]: pgmap v11833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:46 smithi116 ceph-mon[35411]: pgmap v11833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:48 smithi114 ceph-mon[30820]: pgmap v11834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:57:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:48 smithi116 ceph-mon[35411]: pgmap v11834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:57:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:50 smithi114 ceph-mon[30820]: pgmap v11835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:57:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:50 smithi116 ceph-mon[35411]: pgmap v11835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:57:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:52 smithi114 ceph-mon[30820]: pgmap v11836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:57:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:52 smithi116 ceph-mon[35411]: pgmap v11836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:57:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:54 smithi114 ceph-mon[30820]: pgmap v11837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:54 smithi116 ceph-mon[35411]: pgmap v11837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:56 smithi114 ceph-mon[30820]: pgmap v11838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:56 smithi116 ceph-mon[35411]: pgmap v11838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:57:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:57:58 smithi114 ceph-mon[30820]: pgmap v11839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:57:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:57:58 smithi116 ceph-mon[35411]: pgmap v11839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:00 smithi114 ceph-mon[30820]: pgmap v11840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:58:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:00 smithi116 ceph-mon[35411]: pgmap v11840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:58:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:02 smithi114 ceph-mon[30820]: pgmap v11841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:02 smithi116 ceph-mon[35411]: pgmap v11841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:04 smithi114 ceph-mon[30820]: pgmap v11842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:58:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:04 smithi116 ceph-mon[35411]: pgmap v11842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:58:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:06 smithi114 ceph-mon[30820]: pgmap v11843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:58:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:06 smithi116 ceph-mon[35411]: pgmap v11843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:58:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:08 smithi114 ceph-mon[30820]: pgmap v11844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:58:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:08 smithi116 ceph-mon[35411]: pgmap v11844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T12:58:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:09 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:58:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:09 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:58:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:10 smithi114 ceph-mon[30820]: pgmap v11845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:58:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:10 smithi116 ceph-mon[35411]: pgmap v11845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T12:58:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:12 smithi116 ceph-mon[35411]: pgmap v11846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-mon[30820]: pgmap v11846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:13.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:58:12.885+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:58:13.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:58:13.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:58:12 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:58:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:13 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:58:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:58:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:13 smithi116 ceph-mon[35411]: pgmap v11847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:58:13.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:13 smithi116 ceph-mon[35411]: pgmap v11848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:58:13.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:58:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:13 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:58:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:58:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:13 smithi114 ceph-mon[30820]: pgmap v11847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:13 smithi114 ceph-mon[30820]: pgmap v11848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:58:14.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:58:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:15 smithi114 ceph-mon[30820]: pgmap v11849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:58:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:15 smithi116 ceph-mon[35411]: pgmap v11849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T12:58:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:17 smithi114 ceph-mon[30820]: pgmap v11850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:58:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:17 smithi116 ceph-mon[35411]: pgmap v11850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:58:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:19 smithi114 ceph-mon[30820]: pgmap v11851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:58:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:19 smithi116 ceph-mon[35411]: pgmap v11851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:58:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:21 smithi114 ceph-mon[30820]: pgmap v11852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:58:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:21 smithi116 ceph-mon[35411]: pgmap v11852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:58:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:23 smithi114 ceph-mon[30820]: pgmap v11853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:58:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:23 smithi116 ceph-mon[35411]: pgmap v11853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:58:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:25 smithi114 ceph-mon[30820]: pgmap v11854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T12:58:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:25 smithi116 ceph-mon[35411]: pgmap v11854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 5 op/s 2024-09-19T12:58:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:27 smithi114 ceph-mon[30820]: pgmap v11855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 5 op/s 2024-09-19T12:58:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:27 smithi116 ceph-mon[35411]: pgmap v11855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 5 op/s 2024-09-19T12:58:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:29 smithi114 ceph-mon[30820]: pgmap v11856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:29 smithi116 ceph-mon[35411]: pgmap v11856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:31 smithi114 ceph-mon[30820]: pgmap v11857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:58:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:31 smithi116 ceph-mon[35411]: pgmap v11857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:58:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:33 smithi114 ceph-mon[30820]: pgmap v11858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:33 smithi116 ceph-mon[35411]: pgmap v11858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:35 smithi114 ceph-mon[30820]: pgmap v11859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:58:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:35 smithi116 ceph-mon[35411]: pgmap v11859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:58:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:37 smithi114 ceph-mon[30820]: pgmap v11860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:37 smithi116 ceph-mon[35411]: pgmap v11860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:39 smithi114 ceph-mon[30820]: pgmap v11861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:39 smithi116 ceph-mon[35411]: pgmap v11861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:41 smithi114 ceph-mon[30820]: pgmap v11862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:58:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:41 smithi116 ceph-mon[35411]: pgmap v11862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:58:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:44 smithi114 ceph-mon[30820]: pgmap v11863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:44 smithi116 ceph-mon[35411]: pgmap v11863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:46 smithi114 ceph-mon[30820]: pgmap v11864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:58:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:46 smithi116 ceph-mon[35411]: pgmap v11864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:58:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:48 smithi114 ceph-mon[30820]: pgmap v11865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:48 smithi116 ceph-mon[35411]: pgmap v11865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:50 smithi114 ceph-mon[30820]: pgmap v11866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:50 smithi116 ceph-mon[35411]: pgmap v11866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:52 smithi114 ceph-mon[30820]: pgmap v11867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:58:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:52 smithi116 ceph-mon[35411]: pgmap v11867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:58:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:54 smithi114 ceph-mon[30820]: pgmap v11868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:54 smithi116 ceph-mon[35411]: pgmap v11868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:58:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:56 smithi114 ceph-mon[30820]: pgmap v11869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:58:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:56 smithi116 ceph-mon[35411]: pgmap v11869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:58:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:58:58 smithi114 ceph-mon[30820]: pgmap v11870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:58:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:58:58 smithi116 ceph-mon[35411]: pgmap v11870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:59:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:00 smithi114 ceph-mon[30820]: pgmap v11871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:00 smithi116 ceph-mon[35411]: pgmap v11871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:02 smithi114 ceph-mon[30820]: pgmap v11872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:02 smithi116 ceph-mon[35411]: pgmap v11872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:04 smithi114 ceph-mon[30820]: pgmap v11873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:04 smithi116 ceph-mon[35411]: pgmap v11873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:06 smithi114 ceph-mon[30820]: pgmap v11874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:59:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:06 smithi116 ceph-mon[35411]: pgmap v11874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:59:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:08 smithi114 ceph-mon[30820]: pgmap v11875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:59:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:08 smithi116 ceph-mon[35411]: pgmap v11875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T12:59:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:10 smithi114 ceph-mon[30820]: pgmap v11876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:10 smithi116 ceph-mon[35411]: pgmap v11876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:12 smithi114 ceph-mon[30820]: pgmap v11877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:12 smithi116 ceph-mon[35411]: pgmap v11877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:59:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T12:59:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:14 smithi114 ceph-mon[30820]: pgmap v11878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:14 smithi116 ceph-mon[35411]: pgmap v11878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-mon[30820]: pgmap v11879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:59:16.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T12:59:16.245+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:59:16.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T12:59:16.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:59:16.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T12:59:16.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:59:16.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:59:16.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T12:59:16.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:59:16.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T12:59:16.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:59:16.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:59:16.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 12:59:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:59:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:16 smithi116 ceph-mon[35411]: pgmap v11879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T12:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:17 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T12:59:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:59:17.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:17 smithi114 ceph-mon[30820]: pgmap v11880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:17 smithi114 ceph-mon[30820]: pgmap v11881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:59:17.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:17 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T12:59:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T12:59:17.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:17 smithi116 ceph-mon[35411]: pgmap v11880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:17 smithi116 ceph-mon[35411]: pgmap v11881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:59:17.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T12:59:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:19 smithi114 ceph-mon[30820]: pgmap v11882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:59:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:19 smithi116 ceph-mon[35411]: pgmap v11882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:59:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:21 smithi114 ceph-mon[30820]: pgmap v11883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:59:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:21 smithi116 ceph-mon[35411]: pgmap v11883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T12:59:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:23 smithi114 ceph-mon[30820]: pgmap v11884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:59:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:23 smithi116 ceph-mon[35411]: pgmap v11884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T12:59:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:25 smithi114 ceph-mon[30820]: pgmap v11885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:59:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:25 smithi116 ceph-mon[35411]: pgmap v11885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T12:59:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:27 smithi114 ceph-mon[30820]: pgmap v11886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:59:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:27 smithi116 ceph-mon[35411]: pgmap v11886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T12:59:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:29 smithi114 ceph-mon[30820]: pgmap v11887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:29 smithi116 ceph-mon[35411]: pgmap v11887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:31 smithi114 ceph-mon[30820]: pgmap v11888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:59:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:31 smithi116 ceph-mon[35411]: pgmap v11888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:59:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:33 smithi114 ceph-mon[30820]: pgmap v11889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:59:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:33 smithi116 ceph-mon[35411]: pgmap v11889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:59:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:35 smithi114 ceph-mon[30820]: pgmap v11890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:35.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:35 smithi116 ceph-mon[35411]: pgmap v11890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:37 smithi114 ceph-mon[30820]: pgmap v11891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:37 smithi116 ceph-mon[35411]: pgmap v11891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:39 smithi114 ceph-mon[30820]: pgmap v11892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:39 smithi116 ceph-mon[35411]: pgmap v11892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:41 smithi114 ceph-mon[30820]: pgmap v11893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:59:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:41 smithi116 ceph-mon[35411]: pgmap v11893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:59:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:43 smithi114 ceph-mon[30820]: pgmap v11894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:59:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:43 smithi116 ceph-mon[35411]: pgmap v11894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:59:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:45 smithi114 ceph-mon[30820]: pgmap v11895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:45 smithi116 ceph-mon[35411]: pgmap v11895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:47 smithi114 ceph-mon[30820]: pgmap v11896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:47 smithi116 ceph-mon[35411]: pgmap v11896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:49 smithi114 ceph-mon[30820]: pgmap v11897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:49 smithi116 ceph-mon[35411]: pgmap v11897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:51.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:51 smithi114 ceph-mon[30820]: pgmap v11898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:59:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:51 smithi116 ceph-mon[35411]: pgmap v11898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T12:59:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:53 smithi114 ceph-mon[30820]: pgmap v11899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:59:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:53 smithi116 ceph-mon[35411]: pgmap v11899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T12:59:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:55 smithi114 ceph-mon[30820]: pgmap v11900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:55 smithi116 ceph-mon[35411]: pgmap v11900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:57 smithi114 ceph-mon[30820]: pgmap v11901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:57 smithi116 ceph-mon[35411]: pgmap v11901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T12:59:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 12:59:59 smithi114 ceph-mon[30820]: pgmap v11902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T12:59:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 12:59:59 smithi116 ceph-mon[35411]: pgmap v11902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:00:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:00:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:01 smithi114 ceph-mon[30820]: pgmap v11903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:00:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:01 smithi116 ceph-mon[35411]: pgmap v11903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:00:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:03 smithi114 ceph-mon[30820]: pgmap v11904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:00:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:03 smithi116 ceph-mon[35411]: pgmap v11904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:00:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:04 smithi114 ceph-mon[30820]: pgmap v11905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:04 smithi116 ceph-mon[35411]: pgmap v11905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:07 smithi114 ceph-mon[30820]: pgmap v11906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:07 smithi116 ceph-mon[35411]: pgmap v11906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:09 smithi114 ceph-mon[30820]: pgmap v11907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:09 smithi116 ceph-mon[35411]: pgmap v11907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:11 smithi114 ceph-mon[30820]: pgmap v11908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:00:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:11 smithi116 ceph-mon[35411]: pgmap v11908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:00:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:13 smithi114 ceph-mon[30820]: pgmap v11909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:00:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:13 smithi116 ceph-mon[35411]: pgmap v11909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:00:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:15 smithi114 ceph-mon[30820]: pgmap v11910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:15 smithi116 ceph-mon[35411]: pgmap v11910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:17 smithi114 ceph-mon[30820]: pgmap v11911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:00:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:17 smithi116 ceph-mon[35411]: pgmap v11911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:00:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:18 smithi114 ceph-mon[30820]: pgmap v11912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:00:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:18 smithi116 ceph-mon[35411]: pgmap v11912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:00:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:00:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:00:19.604+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:00:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:00:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:00:19.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:00:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:00:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:00:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:00:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:00:19.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:00:19.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:00:19.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:00:19.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:00:19.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:00:19.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:00:19.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:00:19.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:00:19.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:00:19.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:00:19.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:00:19.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:00:19.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:00:19.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:00:19.922 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:00:19 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:00:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:00:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:00:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:00:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:20 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:00:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:00:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:00:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:00:21.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:00:21.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:20 smithi114 ceph-mon[30820]: pgmap v11913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:00:21.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:20 smithi114 ceph-mon[30820]: pgmap v11914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-09-19T13:00:21.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:20 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:00:21.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:20 smithi116 ceph-mon[35411]: pgmap v11913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:00:21.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:20 smithi116 ceph-mon[35411]: pgmap v11914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 5 op/s 2024-09-19T13:00:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:22 smithi114 ceph-mon[30820]: pgmap v11915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:00:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:22 smithi116 ceph-mon[35411]: pgmap v11915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:00:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:24 smithi114 ceph-mon[30820]: pgmap v11916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:00:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:24 smithi116 ceph-mon[35411]: pgmap v11916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:00:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:26 smithi114 ceph-mon[30820]: pgmap v11917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:00:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:26 smithi116 ceph-mon[35411]: pgmap v11917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:00:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:28 smithi114 ceph-mon[30820]: pgmap v11918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:00:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:28 smithi116 ceph-mon[35411]: pgmap v11918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:00:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:30 smithi114 ceph-mon[30820]: pgmap v11919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:00:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:30 smithi116 ceph-mon[35411]: pgmap v11919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:00:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:32 smithi114 ceph-mon[30820]: pgmap v11920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:00:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:32 smithi116 ceph-mon[35411]: pgmap v11920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:00:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:34 smithi114 ceph-mon[30820]: pgmap v11921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:34 smithi116 ceph-mon[35411]: pgmap v11921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:36 smithi114 ceph-mon[30820]: pgmap v11922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:36 smithi116 ceph-mon[35411]: pgmap v11922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:38 smithi114 ceph-mon[30820]: pgmap v11923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:38 smithi116 ceph-mon[35411]: pgmap v11923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:40 smithi114 ceph-mon[30820]: pgmap v11924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:00:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:40 smithi116 ceph-mon[35411]: pgmap v11924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:00:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:42 smithi114 ceph-mon[30820]: pgmap v11925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:00:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:42 smithi116 ceph-mon[35411]: pgmap v11925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:00:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:44 smithi114 ceph-mon[30820]: pgmap v11926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:44 smithi116 ceph-mon[35411]: pgmap v11926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:46 smithi114 ceph-mon[30820]: pgmap v11927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:46 smithi116 ceph-mon[35411]: pgmap v11927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:48 smithi114 ceph-mon[30820]: pgmap v11928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:48 smithi116 ceph-mon[35411]: pgmap v11928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:50 smithi114 ceph-mon[30820]: pgmap v11929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:00:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:50 smithi116 ceph-mon[35411]: pgmap v11929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:00:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:52 smithi114 ceph-mon[30820]: pgmap v11930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:00:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:52 smithi116 ceph-mon[35411]: pgmap v11930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:00:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:54 smithi114 ceph-mon[30820]: pgmap v11931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:54 smithi116 ceph-mon[35411]: pgmap v11931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:56 smithi114 ceph-mon[30820]: pgmap v11932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:56 smithi116 ceph-mon[35411]: pgmap v11932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:00:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:00:58 smithi114 ceph-mon[30820]: pgmap v11933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:00:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:00:58 smithi116 ceph-mon[35411]: pgmap v11933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:00 smithi114 ceph-mon[30820]: pgmap v11934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:00 smithi116 ceph-mon[35411]: pgmap v11934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:02 smithi114 ceph-mon[30820]: pgmap v11935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:02 smithi116 ceph-mon[35411]: pgmap v11935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:05 smithi114 ceph-mon[30820]: pgmap v11936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:05 smithi116 ceph-mon[35411]: pgmap v11936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:07 smithi114 ceph-mon[30820]: pgmap v11937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:01:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:07 smithi116 ceph-mon[35411]: pgmap v11937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:01:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:09 smithi114 ceph-mon[30820]: pgmap v11938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:09 smithi116 ceph-mon[35411]: pgmap v11938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:11.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:11 smithi114 ceph-mon[30820]: pgmap v11939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:11 smithi116 ceph-mon[35411]: pgmap v11939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:13 smithi114 ceph-mon[30820]: pgmap v11940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:13 smithi116 ceph-mon[35411]: pgmap v11940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:15 smithi114 ceph-mon[30820]: pgmap v11941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:15 smithi116 ceph-mon[35411]: pgmap v11941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:17 smithi114 ceph-mon[30820]: pgmap v11942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:01:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:17 smithi116 ceph-mon[35411]: pgmap v11942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:01:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:19 smithi114 ceph-mon[30820]: pgmap v11943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:19 smithi116 ceph-mon[35411]: pgmap v11943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:01:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:01:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:20.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:20.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:21 smithi114 ceph-mon[30820]: pgmap v11944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:21 smithi116 ceph-mon[35411]: pgmap v11944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:23.250 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:01:22.965+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:01:23.251 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:01:23.252 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:01:22 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:01:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:23 smithi114 ceph-mon[30820]: pgmap v11945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:01:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:01:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:23 smithi116 ceph-mon[35411]: pgmap v11945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:01:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:01:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:01:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:01:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:01:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:01:24.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:01:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:01:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:01:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:01:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:01:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:01:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:01:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:01:24.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:01:24.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:24 smithi114 ceph-mon[30820]: pgmap v11946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:01:24.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:24 smithi114 ceph-mon[30820]: pgmap v11947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:01:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:01:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:01:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:01:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:01:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:01:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:01:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:01:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:01:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:01:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:01:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:01:24.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:01:24.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:01:24.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:01:24.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:24 smithi116 ceph-mon[35411]: pgmap v11946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:01:24.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:24 smithi116 ceph-mon[35411]: pgmap v11947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:01:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:26 smithi114 ceph-mon[30820]: pgmap v11948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:01:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:26 smithi116 ceph-mon[35411]: pgmap v11948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:01:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:28 smithi114 ceph-mon[30820]: pgmap v11949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:01:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:28 smithi116 ceph-mon[35411]: pgmap v11949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:01:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:30 smithi114 ceph-mon[30820]: pgmap v11950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:01:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:30 smithi116 ceph-mon[35411]: pgmap v11950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:01:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:32 smithi114 ceph-mon[30820]: pgmap v11951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:01:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:32 smithi116 ceph-mon[35411]: pgmap v11951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:01:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:34 smithi114 ceph-mon[30820]: pgmap v11952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:01:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:34 smithi116 ceph-mon[35411]: pgmap v11952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:01:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:36 smithi114 ceph-mon[30820]: pgmap v11953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:36 smithi116 ceph-mon[35411]: pgmap v11953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:38 smithi114 ceph-mon[30820]: pgmap v11954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:38 smithi116 ceph-mon[35411]: pgmap v11954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:40 smithi114 ceph-mon[30820]: pgmap v11955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:40 smithi116 ceph-mon[35411]: pgmap v11955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:42 smithi114 ceph-mon[30820]: pgmap v11956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:01:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:42 smithi116 ceph-mon[35411]: pgmap v11956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:01:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:44 smithi114 ceph-mon[30820]: pgmap v11957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:44 smithi116 ceph-mon[35411]: pgmap v11957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:46 smithi114 ceph-mon[30820]: pgmap v11958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:46 smithi116 ceph-mon[35411]: pgmap v11958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:48 smithi114 ceph-mon[30820]: pgmap v11959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:48 smithi116 ceph-mon[35411]: pgmap v11959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:50 smithi114 ceph-mon[30820]: pgmap v11960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:50 smithi116 ceph-mon[35411]: pgmap v11960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:52 smithi114 ceph-mon[30820]: pgmap v11961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:01:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:52 smithi116 ceph-mon[35411]: pgmap v11961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:01:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:54 smithi114 ceph-mon[30820]: pgmap v11962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:54 smithi116 ceph-mon[35411]: pgmap v11962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:01:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:56 smithi114 ceph-mon[30820]: pgmap v11963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:56 smithi116 ceph-mon[35411]: pgmap v11963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:01:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:01:58 smithi114 ceph-mon[30820]: pgmap v11964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:01:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:01:58 smithi116 ceph-mon[35411]: pgmap v11964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:00 smithi114 ceph-mon[30820]: pgmap v11965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:00 smithi116 ceph-mon[35411]: pgmap v11965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:02 smithi114 ceph-mon[30820]: pgmap v11966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:02:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:02 smithi116 ceph-mon[35411]: pgmap v11966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:02:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:04 smithi114 ceph-mon[30820]: pgmap v11967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:04 smithi116 ceph-mon[35411]: pgmap v11967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:06 smithi114 ceph-mon[30820]: pgmap v11968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:06 smithi116 ceph-mon[35411]: pgmap v11968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:08 smithi114 ceph-mon[30820]: pgmap v11969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:08 smithi116 ceph-mon[35411]: pgmap v11969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:10 smithi114 ceph-mon[30820]: pgmap v11970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:10 smithi116 ceph-mon[35411]: pgmap v11970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:12 smithi114 ceph-mon[30820]: pgmap v11971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:02:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:12 smithi116 ceph-mon[35411]: pgmap v11971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:02:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:14 smithi114 ceph-mon[30820]: pgmap v11972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:14 smithi116 ceph-mon[35411]: pgmap v11972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:16 smithi114 ceph-mon[30820]: pgmap v11973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:16 smithi116 ceph-mon[35411]: pgmap v11973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:18 smithi114 ceph-mon[30820]: pgmap v11974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:18 smithi116 ceph-mon[35411]: pgmap v11974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:20 smithi114 ceph-mon[30820]: pgmap v11975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:20 smithi116 ceph-mon[35411]: pgmap v11975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:22 smithi114 ceph-mon[30820]: pgmap v11976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:02:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:22 smithi116 ceph-mon[35411]: pgmap v11976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:02:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:02:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:02:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:24 smithi114 ceph-mon[30820]: pgmap v11977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:24 smithi116 ceph-mon[35411]: pgmap v11977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-mon[30820]: pgmap v11978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:02:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:02:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:02:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:02:26.324+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:02:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:02:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:02:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:02:26.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:02:26.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:02:26.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:02:26.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:02:26.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:02:26.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:02:26.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:02:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:02:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:26 smithi116 ceph-mon[35411]: pgmap v11978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:02:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:02:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:02:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:02:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:02:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:02:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:02:27.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:27 smithi114 ceph-mon[30820]: pgmap v11979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:02:27.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:27 smithi114 ceph-mon[30820]: pgmap v11980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:02:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:02:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:02:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:02:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:02:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:02:27.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:02:27.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:02:27.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:02:27.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:02:27.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:02:27.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:02:27.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:27 smithi116 ceph-mon[35411]: pgmap v11979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:02:27.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:27 smithi116 ceph-mon[35411]: pgmap v11980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:02:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:29 smithi114 ceph-mon[30820]: pgmap v11981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:02:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:29 smithi116 ceph-mon[35411]: pgmap v11981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:02:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:31 smithi114 ceph-mon[30820]: pgmap v11982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:02:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:31 smithi116 ceph-mon[35411]: pgmap v11982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:02:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:33 smithi114 ceph-mon[30820]: pgmap v11983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:02:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:33 smithi116 ceph-mon[35411]: pgmap v11983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:02:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:35 smithi114 ceph-mon[30820]: pgmap v11984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:02:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:35 smithi116 ceph-mon[35411]: pgmap v11984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:02:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:37 smithi114 ceph-mon[30820]: pgmap v11985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:02:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:37 smithi116 ceph-mon[35411]: pgmap v11985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:02:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:39 smithi114 ceph-mon[30820]: pgmap v11986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:39 smithi116 ceph-mon[35411]: pgmap v11986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:41 smithi114 ceph-mon[30820]: pgmap v11987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:02:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:41 smithi116 ceph-mon[35411]: pgmap v11987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:02:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:43 smithi114 ceph-mon[30820]: pgmap v11988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:02:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:43 smithi116 ceph-mon[35411]: pgmap v11988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:02:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:45 smithi114 ceph-mon[30820]: pgmap v11989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:45 smithi116 ceph-mon[35411]: pgmap v11989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:47 smithi114 ceph-mon[30820]: pgmap v11990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:47 smithi116 ceph-mon[35411]: pgmap v11990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:49 smithi114 ceph-mon[30820]: pgmap v11991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:49 smithi116 ceph-mon[35411]: pgmap v11991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:51 smithi114 ceph-mon[30820]: pgmap v11992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:02:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:51 smithi116 ceph-mon[35411]: pgmap v11992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:02:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:53 smithi116 ceph-mon[35411]: pgmap v11993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:02:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:53 smithi114 ceph-mon[30820]: pgmap v11993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:02:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:55 smithi116 ceph-mon[35411]: pgmap v11994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:55 smithi114 ceph-mon[30820]: pgmap v11994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:02:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:56 smithi114 ceph-mon[30820]: pgmap v11995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:56 smithi116 ceph-mon[35411]: pgmap v11995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:02:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:02:59 smithi114 ceph-mon[30820]: pgmap v11996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:02:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:02:59 smithi116 ceph-mon[35411]: pgmap v11996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:01 smithi114 ceph-mon[30820]: pgmap v11997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:03:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:01 smithi116 ceph-mon[35411]: pgmap v11997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:03:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:03 smithi114 ceph-mon[30820]: pgmap v11998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:03:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:03 smithi116 ceph-mon[35411]: pgmap v11998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:03:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:05 smithi114 ceph-mon[30820]: pgmap v11999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:03:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:05 smithi116 ceph-mon[35411]: pgmap v11999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:03:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:07 smithi114 ceph-mon[30820]: pgmap v12000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:07 smithi116 ceph-mon[35411]: pgmap v12000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:09 smithi116 ceph-mon[35411]: pgmap v12001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:09 smithi114 ceph-mon[30820]: pgmap v12001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:11 smithi116 ceph-mon[35411]: pgmap v12002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:03:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:11 smithi114 ceph-mon[30820]: pgmap v12002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:03:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:13 smithi116 ceph-mon[35411]: pgmap v12003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:03:13.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:13 smithi114 ceph-mon[30820]: pgmap v12003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:03:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:15 smithi116 ceph-mon[35411]: pgmap v12004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:03:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:15 smithi114 ceph-mon[30820]: pgmap v12004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:03:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:16 smithi114 ceph-mon[30820]: pgmap v12005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:16 smithi116 ceph-mon[35411]: pgmap v12005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:19 smithi114 ceph-mon[30820]: pgmap v12006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:19 smithi116 ceph-mon[35411]: pgmap v12006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:21 smithi114 ceph-mon[30820]: pgmap v12007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:03:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:21 smithi116 ceph-mon[35411]: pgmap v12007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:03:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:23 smithi114 ceph-mon[30820]: pgmap v12008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:03:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:23 smithi116 ceph-mon[35411]: pgmap v12008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:03:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:25 smithi114 ceph-mon[30820]: pgmap v12009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:03:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:25 smithi116 ceph-mon[35411]: pgmap v12009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:03:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:27 smithi116 ceph-mon[35411]: pgmap v12010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:03:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:27 smithi114 ceph-mon[30820]: pgmap v12010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:03:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:29 smithi116 ceph-mon[35411]: pgmap v12011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:29.687 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-mon[30820]: pgmap v12011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:03:29.684+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:03:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:03:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:03:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:03:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:03:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:03:30.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:03:30.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:03:30.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:03:30.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:03:30.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:03:29 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:03:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:03:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:03:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:03:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:03:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:03:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:03:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:03:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:03:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:03:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:03:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:03:31.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:03:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:03:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:03:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:03:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:03:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:03:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:31 smithi114 ceph-mon[30820]: pgmap v12012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:03:31.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:31 smithi114 ceph-mon[30820]: pgmap v12013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:03:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:03:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:03:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:03:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:03:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:03:31.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:03:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:03:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:03:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:03:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:03:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:31 smithi116 ceph-mon[35411]: pgmap v12012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:03:31.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:31 smithi116 ceph-mon[35411]: pgmap v12013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:03:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:32 smithi114 ceph-mon[30820]: pgmap v12014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:03:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:32 smithi116 ceph-mon[35411]: pgmap v12014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:03:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:34 smithi114 ceph-mon[30820]: pgmap v12015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:03:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:34 smithi116 ceph-mon[35411]: pgmap v12015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:03:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:36 smithi114 ceph-mon[30820]: pgmap v12016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:03:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:36 smithi116 ceph-mon[35411]: pgmap v12016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:03:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:38 smithi114 ceph-mon[30820]: pgmap v12017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:03:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:38 smithi116 ceph-mon[35411]: pgmap v12017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:03:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:40 smithi114 ceph-mon[30820]: pgmap v12018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:03:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:40 smithi116 ceph-mon[35411]: pgmap v12018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:03:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:42 smithi114 ceph-mon[30820]: pgmap v12019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:42 smithi116 ceph-mon[35411]: pgmap v12019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:44 smithi114 ceph-mon[30820]: pgmap v12020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:44 smithi116 ceph-mon[35411]: pgmap v12020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:46 smithi114 ceph-mon[30820]: pgmap v12021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:03:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:46 smithi116 ceph-mon[35411]: pgmap v12021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:03:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:48 smithi114 ceph-mon[30820]: pgmap v12022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:48 smithi116 ceph-mon[35411]: pgmap v12022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:50 smithi114 ceph-mon[30820]: pgmap v12023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:03:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:50 smithi116 ceph-mon[35411]: pgmap v12023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:03:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:52 smithi114 ceph-mon[30820]: pgmap v12024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:52 smithi116 ceph-mon[35411]: pgmap v12024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:03:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:54 smithi114 ceph-mon[30820]: pgmap v12025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:54 smithi116 ceph-mon[35411]: pgmap v12025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:56 smithi114 ceph-mon[30820]: pgmap v12026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:03:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:56 smithi116 ceph-mon[35411]: pgmap v12026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:03:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:03:58 smithi114 ceph-mon[30820]: pgmap v12027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:03:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:03:58 smithi116 ceph-mon[35411]: pgmap v12027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:00 smithi114 ceph-mon[30820]: pgmap v12028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:00 smithi116 ceph-mon[35411]: pgmap v12028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:02 smithi114 ceph-mon[30820]: pgmap v12029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:02 smithi116 ceph-mon[35411]: pgmap v12029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:04 smithi114 ceph-mon[30820]: pgmap v12030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:04 smithi116 ceph-mon[35411]: pgmap v12030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:06 smithi114 ceph-mon[30820]: pgmap v12031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:04:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:06 smithi116 ceph-mon[35411]: pgmap v12031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:04:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:08 smithi114 ceph-mon[30820]: pgmap v12032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:08 smithi116 ceph-mon[35411]: pgmap v12032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:10 smithi114 ceph-mon[30820]: pgmap v12033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:10 smithi116 ceph-mon[35411]: pgmap v12033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:12 smithi114 ceph-mon[30820]: pgmap v12034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:12 smithi116 ceph-mon[35411]: pgmap v12034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:14 smithi114 ceph-mon[30820]: pgmap v12035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:14 smithi116 ceph-mon[35411]: pgmap v12035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:16 smithi114 ceph-mon[30820]: pgmap v12036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:04:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:16 smithi116 ceph-mon[35411]: pgmap v12036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:04:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:18 smithi114 ceph-mon[30820]: pgmap v12037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:18 smithi116 ceph-mon[35411]: pgmap v12037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:20 smithi114 ceph-mon[30820]: pgmap v12038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:20 smithi116 ceph-mon[35411]: pgmap v12038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:22 smithi114 ceph-mon[30820]: pgmap v12039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:22 smithi116 ceph-mon[35411]: pgmap v12039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:24 smithi114 ceph-mon[30820]: pgmap v12040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:24 smithi116 ceph-mon[35411]: pgmap v12040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:26 smithi114 ceph-mon[30820]: pgmap v12041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:04:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:26 smithi116 ceph-mon[35411]: pgmap v12041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:04:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:28 smithi114 ceph-mon[30820]: pgmap v12042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:28 smithi116 ceph-mon[35411]: pgmap v12042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:04:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:04:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:30 smithi114 ceph-mon[30820]: pgmap v12043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:30 smithi116 ceph-mon[35411]: pgmap v12043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:32 smithi114 ceph-mon[30820]: pgmap v12044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:04:33.044+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:04:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:04:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:04:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:04:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:04:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:04:33.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:04:33.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:04:33.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:04:33.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:04:33.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:04:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:32 smithi116 ceph-mon[35411]: pgmap v12044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:04:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:04:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:04:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:04:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-mon[30820]: pgmap v12045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:04:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:04:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:33 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:04:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:04:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:04:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:04:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:04:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:04:34.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:04:34.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:04:34.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:04:34.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:04:34.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:04:34.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:04:34.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:33 smithi116 ceph-mon[35411]: pgmap v12045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:04:34.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:04:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:36 smithi114 ceph-mon[30820]: pgmap v12046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:04:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:36 smithi116 ceph-mon[35411]: pgmap v12046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:04:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:38 smithi114 ceph-mon[30820]: pgmap v12047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:04:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:38 smithi116 ceph-mon[35411]: pgmap v12047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:04:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:40 smithi114 ceph-mon[30820]: pgmap v12048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:04:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:40 smithi116 ceph-mon[35411]: pgmap v12048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:04:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:42 smithi114 ceph-mon[30820]: pgmap v12049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:04:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:42 smithi116 ceph-mon[35411]: pgmap v12049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:04:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:44 smithi114 ceph-mon[30820]: pgmap v12050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:04:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:44 smithi116 ceph-mon[35411]: pgmap v12050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:04:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:46 smithi114 ceph-mon[30820]: pgmap v12051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:46 smithi116 ceph-mon[35411]: pgmap v12051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:48 smithi114 ceph-mon[30820]: pgmap v12052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:48 smithi116 ceph-mon[35411]: pgmap v12052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:50 smithi114 ceph-mon[30820]: pgmap v12053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:50 smithi116 ceph-mon[35411]: pgmap v12053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:52 smithi114 ceph-mon[30820]: pgmap v12054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:04:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:52 smithi116 ceph-mon[35411]: pgmap v12054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:04:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:54 smithi114 ceph-mon[30820]: pgmap v12055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:54 smithi116 ceph-mon[35411]: pgmap v12055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:04:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:56 smithi114 ceph-mon[30820]: pgmap v12056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:56 smithi116 ceph-mon[35411]: pgmap v12056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:04:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:04:58 smithi114 ceph-mon[30820]: pgmap v12057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:04:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:04:58 smithi116 ceph-mon[35411]: pgmap v12057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:00 smithi114 ceph-mon[30820]: pgmap v12058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:00 smithi116 ceph-mon[35411]: pgmap v12058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:02 smithi114 ceph-mon[30820]: pgmap v12059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:05:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:02 smithi116 ceph-mon[35411]: pgmap v12059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:05:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:04 smithi114 ceph-mon[30820]: pgmap v12060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:04 smithi116 ceph-mon[35411]: pgmap v12060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:06 smithi114 ceph-mon[30820]: pgmap v12061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:06 smithi116 ceph-mon[35411]: pgmap v12061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:08 smithi114 ceph-mon[30820]: pgmap v12062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:08 smithi116 ceph-mon[35411]: pgmap v12062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:10 smithi114 ceph-mon[30820]: pgmap v12063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:10 smithi116 ceph-mon[35411]: pgmap v12063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:12 smithi114 ceph-mon[30820]: pgmap v12064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:05:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:12 smithi116 ceph-mon[35411]: pgmap v12064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:05:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:14 smithi114 ceph-mon[30820]: pgmap v12065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:14 smithi116 ceph-mon[35411]: pgmap v12065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:16 smithi114 ceph-mon[30820]: pgmap v12066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:16 smithi116 ceph-mon[35411]: pgmap v12066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:18 smithi114 ceph-mon[30820]: pgmap v12067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:18 smithi116 ceph-mon[35411]: pgmap v12067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:20 smithi114 ceph-mon[30820]: pgmap v12068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:20 smithi116 ceph-mon[35411]: pgmap v12068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:22 smithi114 ceph-mon[30820]: pgmap v12069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:05:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:22 smithi116 ceph-mon[35411]: pgmap v12069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:05:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:24 smithi114 ceph-mon[30820]: pgmap v12070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:24 smithi116 ceph-mon[35411]: pgmap v12070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:26 smithi114 ceph-mon[30820]: pgmap v12071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:26 smithi116 ceph-mon[35411]: pgmap v12071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:28 smithi114 ceph-mon[30820]: pgmap v12072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:28 smithi116 ceph-mon[35411]: pgmap v12072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:30 smithi114 ceph-mon[30820]: pgmap v12073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:30 smithi116 ceph-mon[35411]: pgmap v12073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:32 smithi114 ceph-mon[30820]: pgmap v12074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:05:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:32 smithi116 ceph-mon[35411]: pgmap v12074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:05:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:05:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:05:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:34 smithi114 ceph-mon[30820]: pgmap v12075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:34 smithi116 ceph-mon[35411]: pgmap v12075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-mon[30820]: pgmap v12076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:05:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:05:36.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:05:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:05:36.340+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:05:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:05:36.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:05:36.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:05:36.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:05:36.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:05:36.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:05:36.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:05:36.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:05:36.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:05:36.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:05:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:05:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:36 smithi116 ceph-mon[35411]: pgmap v12076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:05:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:05:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:37 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:05:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:05:37.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:05:37.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:37 smithi114 ceph-mon[30820]: pgmap v12077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:05:37.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:37 smithi114 ceph-mon[30820]: pgmap v12078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:37 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:05:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:05:37.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:05:37.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:05:37.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:05:37.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:05:37.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:05:37.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:05:37.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:37 smithi116 ceph-mon[35411]: pgmap v12077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:05:37.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:37 smithi116 ceph-mon[35411]: pgmap v12078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:05:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:39 smithi114 ceph-mon[30820]: pgmap v12079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:05:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:39 smithi116 ceph-mon[35411]: pgmap v12079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:05:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:41 smithi114 ceph-mon[30820]: pgmap v12080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:05:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:41 smithi116 ceph-mon[35411]: pgmap v12080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:05:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:43 smithi114 ceph-mon[30820]: pgmap v12081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:05:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:43 smithi116 ceph-mon[35411]: pgmap v12081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:05:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:45 smithi114 ceph-mon[30820]: pgmap v12082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:05:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:45 smithi116 ceph-mon[35411]: pgmap v12082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:05:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:47 smithi114 ceph-mon[30820]: pgmap v12083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:05:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:47 smithi116 ceph-mon[35411]: pgmap v12083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:05:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:49 smithi114 ceph-mon[30820]: pgmap v12084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:49 smithi116 ceph-mon[35411]: pgmap v12084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:51 smithi114 ceph-mon[30820]: pgmap v12085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:05:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:51 smithi116 ceph-mon[35411]: pgmap v12085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:05:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:53 smithi114 ceph-mon[30820]: pgmap v12086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:05:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:53 smithi116 ceph-mon[35411]: pgmap v12086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:05:55.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:55 smithi114 ceph-mon[30820]: pgmap v12087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:55 smithi116 ceph-mon[35411]: pgmap v12087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:05:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:57 smithi116 ceph-mon[35411]: pgmap v12088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:57 smithi114 ceph-mon[30820]: pgmap v12088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:05:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:05:59 smithi116 ceph-mon[35411]: pgmap v12089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:05:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:05:59 smithi114 ceph-mon[30820]: pgmap v12089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:01 smithi116 ceph-mon[35411]: pgmap v12090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:06:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:01 smithi114 ceph-mon[30820]: pgmap v12090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:06:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:03 smithi114 ceph-mon[30820]: pgmap v12091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:06:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:03 smithi116 ceph-mon[35411]: pgmap v12091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:06:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:05 smithi114 ceph-mon[30820]: pgmap v12092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:06:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:05 smithi116 ceph-mon[35411]: pgmap v12092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:06:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:07 smithi114 ceph-mon[30820]: pgmap v12093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:07 smithi116 ceph-mon[35411]: pgmap v12093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:09 smithi114 ceph-mon[30820]: pgmap v12094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:09 smithi116 ceph-mon[35411]: pgmap v12094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:11 smithi116 ceph-mon[35411]: pgmap v12095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:06:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:11 smithi114 ceph-mon[30820]: pgmap v12095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:06:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:13 smithi116 ceph-mon[35411]: pgmap v12096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:06:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:13 smithi114 ceph-mon[30820]: pgmap v12096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:06:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:15 smithi116 ceph-mon[35411]: pgmap v12097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:06:15.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:15 smithi114 ceph-mon[30820]: pgmap v12097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:06:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:17 smithi114 ceph-mon[30820]: pgmap v12098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:17 smithi116 ceph-mon[35411]: pgmap v12098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:19 smithi114 ceph-mon[30820]: pgmap v12099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:19 smithi116 ceph-mon[35411]: pgmap v12099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:21 smithi114 ceph-mon[30820]: pgmap v12100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:06:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:21 smithi116 ceph-mon[35411]: pgmap v12100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:06:23.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:23 smithi116 ceph-mon[35411]: pgmap v12101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:06:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:23 smithi114 ceph-mon[30820]: pgmap v12101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:06:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:25 smithi114 ceph-mon[30820]: pgmap v12102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:06:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:25 smithi116 ceph-mon[35411]: pgmap v12102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:06:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:27 smithi114 ceph-mon[30820]: pgmap v12103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:27 smithi116 ceph-mon[35411]: pgmap v12103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:29 smithi114 ceph-mon[30820]: pgmap v12104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:29 smithi116 ceph-mon[35411]: pgmap v12104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:31 smithi114 ceph-mon[30820]: pgmap v12105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:06:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:31 smithi116 ceph-mon[35411]: pgmap v12105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:06:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:33 smithi114 ceph-mon[30820]: pgmap v12106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:06:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:33 smithi116 ceph-mon[35411]: pgmap v12106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:06:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:35 smithi114 ceph-mon[30820]: pgmap v12107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:06:35.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:35 smithi116 ceph-mon[35411]: pgmap v12107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:06:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:36 smithi114 ceph-mon[30820]: pgmap v12108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:36 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:06:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:36 smithi116 ceph-mon[35411]: pgmap v12108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:36 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:06:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:39 smithi116 ceph-mon[35411]: pgmap v12109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:39.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-mon[30820]: pgmap v12109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:40.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:06:39.700+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:06:40.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:06:40.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:06:40.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:06:40.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:06:40.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:06:40.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:06:40.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:06:40.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:06:40.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:06:40.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:06:40.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:06:39 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:06:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:06:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:06:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:06:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:06:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:06:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:06:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:06:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:06:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:06:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:06:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:06:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:06:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:06:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:06:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:06:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:06:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:41 smithi114 ceph-mon[30820]: pgmap v12110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:06:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:41 smithi114 ceph-mon[30820]: pgmap v12111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:06:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:06:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:06:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:06:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:06:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:06:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:06:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:06:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:06:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:06:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:06:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:06:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:41 smithi116 ceph-mon[35411]: pgmap v12110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:06:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:41 smithi116 ceph-mon[35411]: pgmap v12111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:06:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:43 smithi114 ceph-mon[30820]: pgmap v12112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:06:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:43 smithi116 ceph-mon[35411]: pgmap v12112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:06:45.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:45 smithi114 ceph-mon[30820]: pgmap v12113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:06:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:45 smithi116 ceph-mon[35411]: pgmap v12113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:06:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:47 smithi114 ceph-mon[30820]: pgmap v12114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:06:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:47 smithi116 ceph-mon[35411]: pgmap v12114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:06:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:49 smithi114 ceph-mon[30820]: pgmap v12115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:06:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:49 smithi116 ceph-mon[35411]: pgmap v12115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:06:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:51 smithi114 ceph-mon[30820]: pgmap v12116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:06:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:51 smithi116 ceph-mon[35411]: pgmap v12116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:06:53.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:53 smithi116 ceph-mon[35411]: pgmap v12117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:53 smithi114 ceph-mon[30820]: pgmap v12117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:06:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:55 smithi114 ceph-mon[30820]: pgmap v12118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:55 smithi116 ceph-mon[35411]: pgmap v12118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:57 smithi114 ceph-mon[30820]: pgmap v12119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:06:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:57 smithi116 ceph-mon[35411]: pgmap v12119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:06:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:06:58 smithi114 ceph-mon[30820]: pgmap v12120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:06:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:06:58 smithi116 ceph-mon[35411]: pgmap v12120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:00 smithi114 ceph-mon[30820]: pgmap v12121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:00 smithi116 ceph-mon[35411]: pgmap v12121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:02 smithi114 ceph-mon[30820]: pgmap v12122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:02 smithi116 ceph-mon[35411]: pgmap v12122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:04 smithi114 ceph-mon[30820]: pgmap v12123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:04 smithi116 ceph-mon[35411]: pgmap v12123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:06 smithi114 ceph-mon[30820]: pgmap v12124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:07:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:06 smithi116 ceph-mon[35411]: pgmap v12124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:07:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:08 smithi114 ceph-mon[30820]: pgmap v12125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:08 smithi116 ceph-mon[35411]: pgmap v12125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:10 smithi114 ceph-mon[30820]: pgmap v12126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:10 smithi116 ceph-mon[35411]: pgmap v12126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:12 smithi114 ceph-mon[30820]: pgmap v12127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:12 smithi116 ceph-mon[35411]: pgmap v12127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:14 smithi114 ceph-mon[30820]: pgmap v12128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:14 smithi116 ceph-mon[35411]: pgmap v12128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:16 smithi114 ceph-mon[30820]: pgmap v12129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:07:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:16 smithi116 ceph-mon[35411]: pgmap v12129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:07:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:18 smithi114 ceph-mon[30820]: pgmap v12130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:18 smithi116 ceph-mon[35411]: pgmap v12130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:20 smithi114 ceph-mon[30820]: pgmap v12131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:20 smithi116 ceph-mon[35411]: pgmap v12131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:22 smithi114 ceph-mon[30820]: pgmap v12132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:22 smithi116 ceph-mon[35411]: pgmap v12132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:24 smithi114 ceph-mon[30820]: pgmap v12133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:24 smithi116 ceph-mon[35411]: pgmap v12133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:26 smithi114 ceph-mon[30820]: pgmap v12134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:07:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:26 smithi116 ceph-mon[35411]: pgmap v12134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:07:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:28 smithi114 ceph-mon[30820]: pgmap v12135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:28 smithi116 ceph-mon[35411]: pgmap v12135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:30 smithi114 ceph-mon[30820]: pgmap v12136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:30 smithi116 ceph-mon[35411]: pgmap v12136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:32 smithi114 ceph-mon[30820]: pgmap v12137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:32 smithi116 ceph-mon[35411]: pgmap v12137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:34 smithi114 ceph-mon[30820]: pgmap v12138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:34 smithi116 ceph-mon[35411]: pgmap v12138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:36 smithi114 ceph-mon[30820]: pgmap v12139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:07:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:36 smithi116 ceph-mon[35411]: pgmap v12139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:07:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:38 smithi114 ceph-mon[30820]: pgmap v12140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:38 smithi116 ceph-mon[35411]: pgmap v12140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:07:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:39 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:07:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:39 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:07:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: pgmap v12141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:07:41.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:07:41.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: pgmap v12141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:07:41.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:07:41.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:41.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:07:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:41 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:07:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:41 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:07:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:41 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:07:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:41 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:07:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:42 smithi114 ceph-mon[30820]: pgmap v12142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:07:43.060+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:07:43.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:07:43.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:07:43.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:07:43.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:07:43.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:07:43.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:07:43.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:07:43.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:07:43.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:07:43.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:07:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:42 smithi116 ceph-mon[35411]: pgmap v12142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:07:44.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:07:44.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:07:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:07:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-mon[30820]: pgmap v12143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:07:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-mon[30820]: pgmap v12144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:07:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:07:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:43 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:07:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:07:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:07:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:07:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:07:44.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:43 smithi116 ceph-mon[35411]: pgmap v12143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:07:44.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:43 smithi116 ceph-mon[35411]: pgmap v12144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:07:44.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:07:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:46 smithi114 ceph-mon[30820]: pgmap v12145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:07:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:46 smithi116 ceph-mon[35411]: pgmap v12145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:07:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:48 smithi114 ceph-mon[30820]: pgmap v12146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:07:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:48 smithi116 ceph-mon[35411]: pgmap v12146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:07:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:50 smithi114 ceph-mon[30820]: pgmap v12147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:07:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:50 smithi116 ceph-mon[35411]: pgmap v12147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:07:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:52 smithi114 ceph-mon[30820]: pgmap v12148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:07:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:52 smithi116 ceph-mon[35411]: pgmap v12148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:07:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:54 smithi114 ceph-mon[30820]: pgmap v12149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:07:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:54 smithi116 ceph-mon[35411]: pgmap v12149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:07:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:56 smithi114 ceph-mon[30820]: pgmap v12150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:56 smithi116 ceph-mon[35411]: pgmap v12150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:07:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:07:58 smithi114 ceph-mon[30820]: pgmap v12151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:07:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:07:58 smithi116 ceph-mon[35411]: pgmap v12151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:00 smithi114 ceph-mon[30820]: pgmap v12152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:00 smithi116 ceph-mon[35411]: pgmap v12152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:02 smithi114 ceph-mon[30820]: pgmap v12153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:02 smithi116 ceph-mon[35411]: pgmap v12153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:04 smithi114 ceph-mon[30820]: pgmap v12154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:04 smithi116 ceph-mon[35411]: pgmap v12154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:06 smithi114 ceph-mon[30820]: pgmap v12155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:08:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:06 smithi116 ceph-mon[35411]: pgmap v12155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:08:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:08 smithi114 ceph-mon[30820]: pgmap v12156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:08 smithi116 ceph-mon[35411]: pgmap v12156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:10 smithi116 ceph-mon[35411]: pgmap v12157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:10 smithi114 ceph-mon[30820]: pgmap v12157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:12 smithi116 ceph-mon[35411]: pgmap v12158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:12 smithi114 ceph-mon[30820]: pgmap v12158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:14 smithi116 ceph-mon[35411]: pgmap v12159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:14 smithi114 ceph-mon[30820]: pgmap v12159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:16.472 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:16 smithi114 ceph-mon[30820]: pgmap v12160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:08:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:16 smithi116 ceph-mon[35411]: pgmap v12160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:08:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:18 smithi114 ceph-mon[30820]: pgmap v12161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:18 smithi116 ceph-mon[35411]: pgmap v12161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:20 smithi114 ceph-mon[30820]: pgmap v12162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:20 smithi116 ceph-mon[35411]: pgmap v12162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:22 smithi114 ceph-mon[30820]: pgmap v12163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:22 smithi116 ceph-mon[35411]: pgmap v12163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:24.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:24 smithi116 ceph-mon[35411]: pgmap v12164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:08:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:24 smithi114 ceph-mon[30820]: pgmap v12164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:08:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:26 smithi114 ceph-mon[30820]: pgmap v12165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:08:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:26 smithi116 ceph-mon[35411]: pgmap v12165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:08:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:28 smithi114 ceph-mon[30820]: pgmap v12166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:28 smithi116 ceph-mon[35411]: pgmap v12166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:30 smithi114 ceph-mon[30820]: pgmap v12167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:08:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:30 smithi116 ceph-mon[35411]: pgmap v12167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:08:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:32 smithi114 ceph-mon[30820]: pgmap v12168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:32 smithi116 ceph-mon[35411]: pgmap v12168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:34 smithi114 ceph-mon[30820]: pgmap v12169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:34 smithi116 ceph-mon[35411]: pgmap v12169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:36 smithi114 ceph-mon[30820]: pgmap v12170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:08:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:36 smithi116 ceph-mon[35411]: pgmap v12170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:08:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:38 smithi114 ceph-mon[30820]: pgmap v12171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:38 smithi116 ceph-mon[35411]: pgmap v12171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:08:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:40 smithi114 ceph-mon[30820]: pgmap v12172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:40 smithi116 ceph-mon[35411]: pgmap v12172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:42 smithi114 ceph-mon[30820]: pgmap v12173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:42 smithi116 ceph-mon[35411]: pgmap v12173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:08:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:43 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:08:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:43 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:08:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:44 smithi114 ceph-mon[30820]: pgmap v12174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:44 smithi116 ceph-mon[35411]: pgmap v12174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-mon[30820]: pgmap v12175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:08:46.420+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:08:46.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:08:46.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:08:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:08:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:46 smithi116 ceph-mon[35411]: pgmap v12175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:08:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:47 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:08:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:08:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:08:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:47 smithi114 ceph-mon[30820]: pgmap v12176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:47 smithi114 ceph-mon[30820]: pgmap v12177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:47.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:08:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:47 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:08:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:08:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:08:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:08:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:08:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:47 smithi116 ceph-mon[35411]: pgmap v12176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:47 smithi116 ceph-mon[35411]: pgmap v12177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:47.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:08:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:49 smithi114 ceph-mon[30820]: pgmap v12178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:49 smithi116 ceph-mon[35411]: pgmap v12178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:51 smithi114 ceph-mon[30820]: pgmap v12179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:51 smithi116 ceph-mon[35411]: pgmap v12179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:52 smithi114 ceph-mon[30820]: pgmap v12180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:52 smithi116 ceph-mon[35411]: pgmap v12180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:55 smithi114 ceph-mon[30820]: pgmap v12181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:55 smithi116 ceph-mon[35411]: pgmap v12181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:08:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:57 smithi114 ceph-mon[30820]: pgmap v12182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:08:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:57 smithi116 ceph-mon[35411]: pgmap v12182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:08:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:08:58 smithi114 ceph-mon[30820]: pgmap v12183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:08:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:08:58 smithi116 ceph-mon[35411]: pgmap v12183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:01 smithi114 ceph-mon[30820]: pgmap v12184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:01 smithi116 ceph-mon[35411]: pgmap v12184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:03 smithi114 ceph-mon[30820]: pgmap v12185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:03 smithi116 ceph-mon[35411]: pgmap v12185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:05 smithi114 ceph-mon[30820]: pgmap v12186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:05 smithi116 ceph-mon[35411]: pgmap v12186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:07 smithi114 ceph-mon[30820]: pgmap v12187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:07 smithi116 ceph-mon[35411]: pgmap v12187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:08 smithi114 ceph-mon[30820]: pgmap v12188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:08 smithi116 ceph-mon[35411]: pgmap v12188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:11 smithi114 ceph-mon[30820]: pgmap v12189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:11 smithi116 ceph-mon[35411]: pgmap v12189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:12 smithi114 ceph-mon[30820]: pgmap v12190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:12 smithi116 ceph-mon[35411]: pgmap v12190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:15 smithi114 ceph-mon[30820]: pgmap v12191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:15 smithi116 ceph-mon[35411]: pgmap v12191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:16.819 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:16 smithi114 ceph-mon[30820]: pgmap v12192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:16 smithi116 ceph-mon[35411]: pgmap v12192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:19 smithi114 ceph-mon[30820]: pgmap v12193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:19 smithi116 ceph-mon[35411]: pgmap v12193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:20 smithi114 ceph-mon[30820]: pgmap v12194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:20 smithi116 ceph-mon[35411]: pgmap v12194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:22 smithi114 ceph-mon[30820]: pgmap v12195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:22 smithi116 ceph-mon[35411]: pgmap v12195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:24 smithi114 ceph-mon[30820]: pgmap v12196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:24 smithi116 ceph-mon[35411]: pgmap v12196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:26 smithi114 ceph-mon[30820]: pgmap v12197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:26 smithi116 ceph-mon[35411]: pgmap v12197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:28 smithi114 ceph-mon[30820]: pgmap v12198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:28 smithi116 ceph-mon[35411]: pgmap v12198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:30 smithi114 ceph-mon[30820]: pgmap v12199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:30 smithi116 ceph-mon[35411]: pgmap v12199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:32 smithi114 ceph-mon[30820]: pgmap v12200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:32 smithi116 ceph-mon[35411]: pgmap v12200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:34 smithi114 ceph-mon[30820]: pgmap v12201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:34 smithi116 ceph-mon[35411]: pgmap v12201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:36 smithi114 ceph-mon[30820]: pgmap v12202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:36 smithi116 ceph-mon[35411]: pgmap v12202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:38 smithi114 ceph-mon[30820]: pgmap v12203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:38 smithi116 ceph-mon[35411]: pgmap v12203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:40 smithi114 ceph-mon[30820]: pgmap v12204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:40 smithi116 ceph-mon[35411]: pgmap v12204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:09:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:42 smithi114 ceph-mon[30820]: pgmap v12205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:42 smithi116 ceph-mon[35411]: pgmap v12205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:44 smithi114 ceph-mon[30820]: pgmap v12206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:44 smithi116 ceph-mon[35411]: pgmap v12206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:09:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:46 smithi114 ceph-mon[30820]: pgmap v12207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:46 smithi116 ceph-mon[35411]: pgmap v12207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:09:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:09:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:09:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:48 smithi114 ceph-mon[30820]: pgmap v12208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:48 smithi116 ceph-mon[35411]: pgmap v12208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:09:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:09:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:09:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:09:49.780+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:09:50.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:09:50.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:09:49 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:09:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:09:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:09:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:09:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:50 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:09:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:09:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:09:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:09:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:09:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:09:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:09:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:09:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:09:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:09:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:09:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:50 smithi114 ceph-mon[30820]: pgmap v12209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:09:51.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:50 smithi114 ceph-mon[30820]: pgmap v12210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:09:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:50 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:09:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:09:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:09:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:09:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:09:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:09:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:09:51.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:09:51.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:09:51.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:09:51.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:50 smithi116 ceph-mon[35411]: pgmap v12209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:09:51.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:50 smithi116 ceph-mon[35411]: pgmap v12210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:09:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:52 smithi114 ceph-mon[30820]: pgmap v12211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:09:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:52 smithi116 ceph-mon[35411]: pgmap v12211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:09:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:54 smithi114 ceph-mon[30820]: pgmap v12212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:09:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:54 smithi116 ceph-mon[35411]: pgmap v12212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:09:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:56 smithi114 ceph-mon[30820]: pgmap v12213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:09:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:56 smithi116 ceph-mon[35411]: pgmap v12213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:09:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:09:58 smithi114 ceph-mon[30820]: pgmap v12214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:09:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:09:58 smithi116 ceph-mon[35411]: pgmap v12214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:10:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:00 smithi114 ceph-mon[30820]: pgmap v12215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:10:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:10:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:00 smithi116 ceph-mon[35411]: pgmap v12215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:10:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:10:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:02 smithi114 ceph-mon[30820]: pgmap v12216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:02 smithi116 ceph-mon[35411]: pgmap v12216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:04 smithi114 ceph-mon[30820]: pgmap v12217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:04 smithi116 ceph-mon[35411]: pgmap v12217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:06 smithi114 ceph-mon[30820]: pgmap v12218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:06 smithi116 ceph-mon[35411]: pgmap v12218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:08 smithi114 ceph-mon[30820]: pgmap v12219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:08 smithi116 ceph-mon[35411]: pgmap v12219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:10 smithi114 ceph-mon[30820]: pgmap v12220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:10:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:10 smithi116 ceph-mon[35411]: pgmap v12220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:10:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:12 smithi114 ceph-mon[30820]: pgmap v12221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:12 smithi116 ceph-mon[35411]: pgmap v12221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:14 smithi114 ceph-mon[30820]: pgmap v12222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:14 smithi116 ceph-mon[35411]: pgmap v12222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:16 smithi114 ceph-mon[30820]: pgmap v12223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:17.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:16 smithi116 ceph-mon[35411]: pgmap v12223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:18 smithi114 ceph-mon[30820]: pgmap v12224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:18 smithi116 ceph-mon[35411]: pgmap v12224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:20 smithi114 ceph-mon[30820]: pgmap v12225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:10:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:20 smithi116 ceph-mon[35411]: pgmap v12225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:10:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:22 smithi114 ceph-mon[30820]: pgmap v12226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:22 smithi116 ceph-mon[35411]: pgmap v12226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:24 smithi114 ceph-mon[30820]: pgmap v12227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:24 smithi116 ceph-mon[35411]: pgmap v12227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:26 smithi114 ceph-mon[30820]: pgmap v12228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:26 smithi116 ceph-mon[35411]: pgmap v12228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:28 smithi114 ceph-mon[30820]: pgmap v12229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:28 smithi116 ceph-mon[35411]: pgmap v12229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:30 smithi114 ceph-mon[30820]: pgmap v12230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:10:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:30 smithi116 ceph-mon[35411]: pgmap v12230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:10:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:32 smithi116 ceph-mon[35411]: pgmap v12231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:32 smithi114 ceph-mon[30820]: pgmap v12231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:34 smithi116 ceph-mon[35411]: pgmap v12232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:34 smithi114 ceph-mon[30820]: pgmap v12232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:36 smithi116 ceph-mon[35411]: pgmap v12233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:36 smithi114 ceph-mon[30820]: pgmap v12233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:38 smithi114 ceph-mon[30820]: pgmap v12234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:38 smithi116 ceph-mon[35411]: pgmap v12234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:10:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:40 smithi114 ceph-mon[30820]: pgmap v12235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:10:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:40 smithi116 ceph-mon[35411]: pgmap v12235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:10:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:42 smithi114 ceph-mon[30820]: pgmap v12236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:42 smithi116 ceph-mon[35411]: pgmap v12236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:44 smithi114 ceph-mon[30820]: pgmap v12237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:10:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:44 smithi116 ceph-mon[35411]: pgmap v12237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:10:47.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:46 smithi114 ceph-mon[30820]: pgmap v12238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:46 smithi116 ceph-mon[35411]: pgmap v12238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:10:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:48 smithi114 ceph-mon[30820]: pgmap v12239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:10:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:48 smithi116 ceph-mon[35411]: pgmap v12239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:10:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:10:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:10:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:51 smithi114 ceph-mon[30820]: pgmap v12240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:10:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:51 smithi116 ceph-mon[35411]: pgmap v12240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:10:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:53.403 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:10:53.140+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:10:53.403 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:10:53.404 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:10:53.404 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:10:53.404 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:10:53.404 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:10:53.404 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:10:53.404 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:10:53.404 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:10:53.404 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:10:53.405 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:10:53.406 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:10:53.406 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:10:53.406 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:10:53.406 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:10:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-mon[30820]: pgmap v12241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:10:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:10:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:53 smithi116 ceph-mon[35411]: pgmap v12241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:10:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:10:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:10:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:10:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:10:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:10:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:54 smithi114 ceph-mon[30820]: pgmap v12242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:10:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:54 smithi114 ceph-mon[30820]: pgmap v12243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:10:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:10:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:10:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:10:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:10:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:54 smithi116 ceph-mon[35411]: pgmap v12242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:10:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:54 smithi116 ceph-mon[35411]: pgmap v12243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:10:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:56 smithi114 ceph-mon[30820]: pgmap v12244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:10:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:56 smithi116 ceph-mon[35411]: pgmap v12244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:10:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:10:58 smithi114 ceph-mon[30820]: pgmap v12245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:10:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:10:58 smithi116 ceph-mon[35411]: pgmap v12245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:11:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:00 smithi114 ceph-mon[30820]: pgmap v12246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:11:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:00 smithi116 ceph-mon[35411]: pgmap v12246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:11:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:02 smithi116 ceph-mon[35411]: pgmap v12247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:11:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:02 smithi114 ceph-mon[30820]: pgmap v12247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:11:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:04 smithi116 ceph-mon[35411]: pgmap v12248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:11:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:04 smithi114 ceph-mon[30820]: pgmap v12248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:11:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:06 smithi116 ceph-mon[35411]: pgmap v12249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:06 smithi114 ceph-mon[30820]: pgmap v12249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:08 smithi114 ceph-mon[30820]: pgmap v12250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:08 smithi116 ceph-mon[35411]: pgmap v12250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:10 smithi114 ceph-mon[30820]: pgmap v12251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:10 smithi116 ceph-mon[35411]: pgmap v12251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:12 smithi114 ceph-mon[30820]: pgmap v12252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:12 smithi116 ceph-mon[35411]: pgmap v12252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:14 smithi114 ceph-mon[30820]: pgmap v12253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:14 smithi116 ceph-mon[35411]: pgmap v12253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:16 smithi114 ceph-mon[30820]: pgmap v12254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:16 smithi116 ceph-mon[35411]: pgmap v12254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:18.720 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:18 smithi114 ceph-mon[30820]: pgmap v12255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:18 smithi116 ceph-mon[35411]: pgmap v12255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:20 smithi114 ceph-mon[30820]: pgmap v12256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:20 smithi116 ceph-mon[35411]: pgmap v12256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:22 smithi114 ceph-mon[30820]: pgmap v12257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:22 smithi116 ceph-mon[35411]: pgmap v12257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:24 smithi114 ceph-mon[30820]: pgmap v12258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:24 smithi116 ceph-mon[35411]: pgmap v12258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:26.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:26 smithi116 ceph-mon[35411]: pgmap v12259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:26 smithi114 ceph-mon[30820]: pgmap v12259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:28 smithi114 ceph-mon[30820]: pgmap v12260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:28.933 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:28 smithi116 ceph-mon[35411]: pgmap v12260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:30.729 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:30 smithi114 ceph-mon[30820]: pgmap v12261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:30 smithi116 ceph-mon[35411]: pgmap v12261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:32 smithi114 ceph-mon[30820]: pgmap v12262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:32 smithi116 ceph-mon[35411]: pgmap v12262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:34 smithi114 ceph-mon[30820]: pgmap v12263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:11:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:34 smithi116 ceph-mon[35411]: pgmap v12263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:11:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:36 smithi114 ceph-mon[30820]: pgmap v12264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:11:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:36 smithi116 ceph-mon[35411]: pgmap v12264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:11:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:38 smithi114 ceph-mon[30820]: pgmap v12265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:38 smithi116 ceph-mon[35411]: pgmap v12265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:40 smithi114 ceph-mon[30820]: pgmap v12266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:11:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:40 smithi116 ceph-mon[35411]: pgmap v12266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:11:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:42 smithi114 ceph-mon[30820]: pgmap v12267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:42 smithi116 ceph-mon[35411]: pgmap v12267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:44.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:44 smithi114 ceph-mon[30820]: pgmap v12268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:44 smithi116 ceph-mon[35411]: pgmap v12268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:46 smithi114 ceph-mon[30820]: pgmap v12269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:46 smithi116 ceph-mon[35411]: pgmap v12269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:48 smithi114 ceph-mon[30820]: pgmap v12270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:48 smithi116 ceph-mon[35411]: pgmap v12270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:11:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:50 smithi114 ceph-mon[30820]: pgmap v12271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:50 smithi116 ceph-mon[35411]: pgmap v12271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:52 smithi114 ceph-mon[30820]: pgmap v12272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:52 smithi116 ceph-mon[35411]: pgmap v12272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:11:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:11:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:11:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:54 smithi114 ceph-mon[30820]: pgmap v12273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:54 smithi116 ceph-mon[35411]: pgmap v12273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:11:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:54.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:55 smithi114 ceph-mon[30820]: pgmap v12274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:55 smithi116 ceph-mon[35411]: pgmap v12274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:11:56.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:11:56.831 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:11:56.831 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:11:56.831 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:11:56.831 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:11:56.831 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:11:56.832 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:56 smithi116 ceph-mon[35411]: pgmap v12275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:56 smithi116 ceph-mon[35411]: pgmap v12276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:11:56.833 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:11:56.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:11:56.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-mon[30820]: pgmap v12275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-mon[30820]: pgmap v12276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:11:56.500+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:11:56.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:11:56.921 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:11:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:11:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:11:58 smithi114 ceph-mon[30820]: pgmap v12277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:11:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:11:58 smithi116 ceph-mon[35411]: pgmap v12277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:12:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:00 smithi114 ceph-mon[30820]: pgmap v12278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:12:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:00 smithi116 ceph-mon[35411]: pgmap v12278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:12:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:02 smithi114 ceph-mon[30820]: pgmap v12279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:12:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:02 smithi116 ceph-mon[35411]: pgmap v12279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:12:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:04 smithi114 ceph-mon[30820]: pgmap v12280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:12:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:04 smithi116 ceph-mon[35411]: pgmap v12280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:12:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:06 smithi114 ceph-mon[30820]: pgmap v12281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:12:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:06 smithi116 ceph-mon[35411]: pgmap v12281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:12:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:08 smithi114 ceph-mon[30820]: pgmap v12282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:08 smithi116 ceph-mon[35411]: pgmap v12282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:10 smithi114 ceph-mon[30820]: pgmap v12283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:10 smithi116 ceph-mon[35411]: pgmap v12283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:12 smithi114 ceph-mon[30820]: pgmap v12284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:12 smithi116 ceph-mon[35411]: pgmap v12284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:14 smithi114 ceph-mon[30820]: pgmap v12285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:14 smithi116 ceph-mon[35411]: pgmap v12285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:16 smithi114 ceph-mon[30820]: pgmap v12286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:16 smithi116 ceph-mon[35411]: pgmap v12286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:18 smithi114 ceph-mon[30820]: pgmap v12287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:18 smithi116 ceph-mon[35411]: pgmap v12287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:20 smithi114 ceph-mon[30820]: pgmap v12288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:20 smithi116 ceph-mon[35411]: pgmap v12288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:22 smithi114 ceph-mon[30820]: pgmap v12289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:22 smithi116 ceph-mon[35411]: pgmap v12289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:24 smithi114 ceph-mon[30820]: pgmap v12290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:24 smithi116 ceph-mon[35411]: pgmap v12290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:26 smithi114 ceph-mon[30820]: pgmap v12291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:26 smithi116 ceph-mon[35411]: pgmap v12291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:28 smithi114 ceph-mon[30820]: pgmap v12292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:28 smithi116 ceph-mon[35411]: pgmap v12292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:30 smithi114 ceph-mon[30820]: pgmap v12293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:30 smithi116 ceph-mon[35411]: pgmap v12293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:32 smithi114 ceph-mon[30820]: pgmap v12294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:32 smithi116 ceph-mon[35411]: pgmap v12294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:35 smithi114 ceph-mon[30820]: pgmap v12295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:35 smithi116 ceph-mon[35411]: pgmap v12295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:36 smithi114 ceph-mon[30820]: pgmap v12296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:36 smithi116 ceph-mon[35411]: pgmap v12296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:38 smithi114 ceph-mon[30820]: pgmap v12297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:38 smithi116 ceph-mon[35411]: pgmap v12297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:40 smithi114 ceph-mon[30820]: pgmap v12298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:40 smithi116 ceph-mon[35411]: pgmap v12298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:42 smithi114 ceph-mon[30820]: pgmap v12299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:42 smithi116 ceph-mon[35411]: pgmap v12299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:44 smithi114 ceph-mon[30820]: pgmap v12300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:44 smithi116 ceph-mon[35411]: pgmap v12300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:46 smithi114 ceph-mon[30820]: pgmap v12301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:46 smithi116 ceph-mon[35411]: pgmap v12301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:48 smithi114 ceph-mon[30820]: pgmap v12302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:48 smithi116 ceph-mon[35411]: pgmap v12302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:50 smithi114 ceph-mon[30820]: pgmap v12303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:50 smithi116 ceph-mon[35411]: pgmap v12303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:12:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:52 smithi114 ceph-mon[30820]: pgmap v12304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:52 smithi116 ceph-mon[35411]: pgmap v12304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:54 smithi114 ceph-mon[30820]: pgmap v12305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:54 smithi116 ceph-mon[35411]: pgmap v12305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:12:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:56 smithi114 ceph-mon[30820]: pgmap v12306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:56 smithi116 ceph-mon[35411]: pgmap v12306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:12:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:12:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:12:59.863 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-mon[30820]: pgmap v12307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:12:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:12:59 smithi116 ceph-mon[35411]: pgmap v12307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:00.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:12:59.862+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:13:00.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:13:00.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:12:59 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:13:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:00 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:13:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:13:00.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:13:00.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:00 smithi114 ceph-mon[30820]: pgmap v12308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:13:00.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:00 smithi114 ceph-mon[30820]: pgmap v12309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:13:00.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:13:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:00 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:13:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:00 smithi116 ceph-mon[35411]: pgmap v12308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:13:00.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:00 smithi116 ceph-mon[35411]: pgmap v12309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:13:00.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:13:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:02 smithi116 ceph-mon[35411]: pgmap v12310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:13:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:02 smithi114 ceph-mon[30820]: pgmap v12310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:13:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:05 smithi114 ceph-mon[30820]: pgmap v12311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:13:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:05 smithi116 ceph-mon[35411]: pgmap v12311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:13:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:07 smithi114 ceph-mon[30820]: pgmap v12312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:13:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:07 smithi116 ceph-mon[35411]: pgmap v12312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:13:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:09 smithi114 ceph-mon[30820]: pgmap v12313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:13:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:09 smithi116 ceph-mon[35411]: pgmap v12313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:13:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:11 smithi114 ceph-mon[30820]: pgmap v12314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:13:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:11 smithi116 ceph-mon[35411]: pgmap v12314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:13:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:13 smithi114 ceph-mon[30820]: pgmap v12315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:13 smithi116 ceph-mon[35411]: pgmap v12315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:15 smithi114 ceph-mon[30820]: pgmap v12316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:15 smithi116 ceph-mon[35411]: pgmap v12316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:17 smithi114 ceph-mon[30820]: pgmap v12317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:17 smithi116 ceph-mon[35411]: pgmap v12317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:19 smithi114 ceph-mon[30820]: pgmap v12318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:19 smithi116 ceph-mon[35411]: pgmap v12318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:21 smithi114 ceph-mon[30820]: pgmap v12319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:13:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:21 smithi116 ceph-mon[35411]: pgmap v12319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:13:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:23 smithi114 ceph-mon[30820]: pgmap v12320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:23 smithi116 ceph-mon[35411]: pgmap v12320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:25 smithi114 ceph-mon[30820]: pgmap v12321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:25 smithi116 ceph-mon[35411]: pgmap v12321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:27 smithi116 ceph-mon[35411]: pgmap v12322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:27 smithi114 ceph-mon[30820]: pgmap v12322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:29 smithi116 ceph-mon[35411]: pgmap v12323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:29 smithi114 ceph-mon[30820]: pgmap v12323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:31 smithi116 ceph-mon[35411]: pgmap v12324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:13:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:31 smithi114 ceph-mon[30820]: pgmap v12324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:13:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:33 smithi114 ceph-mon[30820]: pgmap v12325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:33 smithi116 ceph-mon[35411]: pgmap v12325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:35 smithi114 ceph-mon[30820]: pgmap v12326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:35 smithi116 ceph-mon[35411]: pgmap v12326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:37 smithi114 ceph-mon[30820]: pgmap v12327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:37 smithi116 ceph-mon[35411]: pgmap v12327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:39 smithi114 ceph-mon[30820]: pgmap v12328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:39 smithi116 ceph-mon[35411]: pgmap v12328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:41 smithi114 ceph-mon[30820]: pgmap v12329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:13:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:41 smithi116 ceph-mon[35411]: pgmap v12329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:13:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:43 smithi114 ceph-mon[30820]: pgmap v12330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:43 smithi116 ceph-mon[35411]: pgmap v12330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:45 smithi114 ceph-mon[30820]: pgmap v12331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:45 smithi116 ceph-mon[35411]: pgmap v12331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:47 smithi114 ceph-mon[30820]: pgmap v12332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:47 smithi116 ceph-mon[35411]: pgmap v12332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:49.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:49 smithi114 ceph-mon[30820]: pgmap v12333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:49 smithi116 ceph-mon[35411]: pgmap v12333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:51 smithi114 ceph-mon[30820]: pgmap v12334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:13:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:51 smithi116 ceph-mon[35411]: pgmap v12334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:13:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:53 smithi114 ceph-mon[30820]: pgmap v12335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:53 smithi116 ceph-mon[35411]: pgmap v12335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:13:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:55 smithi114 ceph-mon[30820]: pgmap v12336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:55 smithi116 ceph-mon[35411]: pgmap v12336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:57.732 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:57 smithi116 ceph-mon[35411]: pgmap v12337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:57 smithi114 ceph-mon[30820]: pgmap v12337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:13:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:13:59 smithi114 ceph-mon[30820]: pgmap v12338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:13:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:13:59 smithi116 ceph-mon[35411]: pgmap v12338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:14:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:14:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:01 smithi114 ceph-mon[30820]: pgmap v12339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:14:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:01 smithi116 ceph-mon[35411]: pgmap v12339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:14:03.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:14:03.220+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:14:03.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:14:03.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:14:03.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:14:03.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:14:03.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:14:03.494 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:14:03.495 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:14:03.496 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:14:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-mon[30820]: pgmap v12340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:14:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:14:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:03 smithi116 ceph-mon[35411]: pgmap v12340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:14:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:14:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:14:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:14:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:14:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:14:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:14:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:14:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:14:04.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:14:04.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:14:04.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:14:04.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:04 smithi114 ceph-mon[30820]: pgmap v12341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:14:04.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:04 smithi114 ceph-mon[30820]: pgmap v12342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:14:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:14:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:14:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:14:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:14:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:14:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:14:04.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:14:04.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:14:04.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:14:04.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:14:04.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:14:04.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:14:04.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:04 smithi116 ceph-mon[35411]: pgmap v12341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:14:04.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:04 smithi116 ceph-mon[35411]: pgmap v12342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:14:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:06 smithi114 ceph-mon[30820]: pgmap v12343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:14:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:06 smithi116 ceph-mon[35411]: pgmap v12343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:14:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:08 smithi114 ceph-mon[30820]: pgmap v12344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:14:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:08 smithi116 ceph-mon[35411]: pgmap v12344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:14:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:10 smithi114 ceph-mon[30820]: pgmap v12345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:14:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:10 smithi116 ceph-mon[35411]: pgmap v12345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:14:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:12 smithi114 ceph-mon[30820]: pgmap v12346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:14:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:12 smithi116 ceph-mon[35411]: pgmap v12346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:14 smithi114 ceph-mon[30820]: pgmap v12347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:14:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:14 smithi116 ceph-mon[35411]: pgmap v12347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:14:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:16 smithi114 ceph-mon[30820]: pgmap v12348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:14:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:16 smithi116 ceph-mon[35411]: pgmap v12348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:14:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:18 smithi114 ceph-mon[30820]: pgmap v12349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:18 smithi116 ceph-mon[35411]: pgmap v12349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:20 smithi114 ceph-mon[30820]: pgmap v12350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:20 smithi116 ceph-mon[35411]: pgmap v12350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:22 smithi114 ceph-mon[30820]: pgmap v12351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:14:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:22 smithi116 ceph-mon[35411]: pgmap v12351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:14:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:24 smithi114 ceph-mon[30820]: pgmap v12352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:24 smithi116 ceph-mon[35411]: pgmap v12352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:26 smithi114 ceph-mon[30820]: pgmap v12353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:14:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:26 smithi116 ceph-mon[35411]: pgmap v12353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:14:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:28 smithi114 ceph-mon[30820]: pgmap v12354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:28 smithi116 ceph-mon[35411]: pgmap v12354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:30 smithi114 ceph-mon[30820]: pgmap v12355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:30 smithi116 ceph-mon[35411]: pgmap v12355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:32 smithi114 ceph-mon[30820]: pgmap v12356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:14:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:32 smithi116 ceph-mon[35411]: pgmap v12356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:14:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:34 smithi114 ceph-mon[30820]: pgmap v12357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:34 smithi116 ceph-mon[35411]: pgmap v12357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:35 smithi114 ceph-mon[30820]: pgmap v12358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:14:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:35 smithi116 ceph-mon[35411]: pgmap v12358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:14:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:38 smithi114 ceph-mon[30820]: pgmap v12359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:38 smithi116 ceph-mon[35411]: pgmap v12359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:40 smithi114 ceph-mon[30820]: pgmap v12360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:40 smithi116 ceph-mon[35411]: pgmap v12360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:42 smithi114 ceph-mon[30820]: pgmap v12361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:14:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:42 smithi116 ceph-mon[35411]: pgmap v12361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:14:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:44 smithi114 ceph-mon[30820]: pgmap v12362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:44 smithi116 ceph-mon[35411]: pgmap v12362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:46 smithi114 ceph-mon[30820]: pgmap v12363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:14:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:46 smithi116 ceph-mon[35411]: pgmap v12363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:14:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:48 smithi114 ceph-mon[30820]: pgmap v12364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:48 smithi116 ceph-mon[35411]: pgmap v12364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:14:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:50 smithi114 ceph-mon[30820]: pgmap v12365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:50 smithi116 ceph-mon[35411]: pgmap v12365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:52 smithi114 ceph-mon[30820]: pgmap v12366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:14:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:52 smithi116 ceph-mon[35411]: pgmap v12366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:14:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:54 smithi114 ceph-mon[30820]: pgmap v12367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:54 smithi116 ceph-mon[35411]: pgmap v12367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:14:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:56 smithi114 ceph-mon[30820]: pgmap v12368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:14:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:56 smithi116 ceph-mon[35411]: pgmap v12368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:14:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:14:58 smithi114 ceph-mon[30820]: pgmap v12369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:14:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:14:58 smithi116 ceph-mon[35411]: pgmap v12369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:15:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:00 smithi114 ceph-mon[30820]: pgmap v12370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:00 smithi116 ceph-mon[35411]: pgmap v12370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:02 smithi114 ceph-mon[30820]: pgmap v12371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:02 smithi116 ceph-mon[35411]: pgmap v12371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:04 smithi114 ceph-mon[30820]: pgmap v12372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:04 smithi116 ceph-mon[35411]: pgmap v12372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:15:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-mon[30820]: pgmap v12373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:15:06.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:15:06.581+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:15:06.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:15:06.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:15:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:15:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:06 smithi116 ceph-mon[35411]: pgmap v12373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:15:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:15:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:15:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:15:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:15:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:15:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:15:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:08 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:15:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:15:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:15:08.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:08 smithi114 ceph-mon[30820]: pgmap v12374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T13:15:08.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:08 smithi114 ceph-mon[30820]: pgmap v12375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:15:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:08 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:15:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:15:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:15:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:15:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:15:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:15:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:15:08.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:15:08.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:15:08.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:15:08.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:15:08.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:08 smithi116 ceph-mon[35411]: pgmap v12374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T13:15:08.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:08 smithi116 ceph-mon[35411]: pgmap v12375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:15:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:10 smithi114 ceph-mon[30820]: pgmap v12376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:15:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:10 smithi116 ceph-mon[35411]: pgmap v12376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:15:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:12 smithi114 ceph-mon[30820]: pgmap v12377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:15:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:12 smithi116 ceph-mon[35411]: pgmap v12377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:15:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:14 smithi114 ceph-mon[30820]: pgmap v12378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:15:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:14 smithi116 ceph-mon[35411]: pgmap v12378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:15:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:16 smithi114 ceph-mon[30820]: pgmap v12379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:15:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:16 smithi116 ceph-mon[35411]: pgmap v12379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:15:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:18 smithi114 ceph-mon[30820]: pgmap v12380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:15:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:18 smithi116 ceph-mon[35411]: pgmap v12380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:15:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:20 smithi114 ceph-mon[30820]: pgmap v12381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:20 smithi116 ceph-mon[35411]: pgmap v12381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:22 smithi114 ceph-mon[30820]: pgmap v12382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:22 smithi116 ceph-mon[35411]: pgmap v12382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:24 smithi114 ceph-mon[30820]: pgmap v12383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:24 smithi116 ceph-mon[35411]: pgmap v12383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:26 smithi114 ceph-mon[30820]: pgmap v12384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:15:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:26 smithi116 ceph-mon[35411]: pgmap v12384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:15:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:28 smithi114 ceph-mon[30820]: pgmap v12385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:15:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:28 smithi116 ceph-mon[35411]: pgmap v12385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:15:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:30 smithi114 ceph-mon[30820]: pgmap v12386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:30 smithi116 ceph-mon[35411]: pgmap v12386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:32 smithi114 ceph-mon[30820]: pgmap v12387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:32 smithi116 ceph-mon[35411]: pgmap v12387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:34 smithi114 ceph-mon[30820]: pgmap v12388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:34 smithi116 ceph-mon[35411]: pgmap v12388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:36 smithi114 ceph-mon[30820]: pgmap v12389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:15:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:36 smithi116 ceph-mon[35411]: pgmap v12389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:15:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:38 smithi114 ceph-mon[30820]: pgmap v12390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:15:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:38 smithi116 ceph-mon[35411]: pgmap v12390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:15:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:40 smithi114 ceph-mon[30820]: pgmap v12391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:40 smithi116 ceph-mon[35411]: pgmap v12391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:42 smithi116 ceph-mon[35411]: pgmap v12392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:42 smithi114 ceph-mon[30820]: pgmap v12392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:44 smithi116 ceph-mon[35411]: pgmap v12393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:44 smithi114 ceph-mon[30820]: pgmap v12393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:46 smithi116 ceph-mon[35411]: pgmap v12394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:15:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:46 smithi114 ceph-mon[30820]: pgmap v12394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:15:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:48 smithi114 ceph-mon[30820]: pgmap v12395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:15:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:48 smithi116 ceph-mon[35411]: pgmap v12395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:15:50.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:50 smithi114 ceph-mon[30820]: pgmap v12396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:50 smithi116 ceph-mon[35411]: pgmap v12396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:52 smithi114 ceph-mon[30820]: pgmap v12397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:52 smithi116 ceph-mon[35411]: pgmap v12397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:15:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:54 smithi114 ceph-mon[30820]: pgmap v12398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:54 smithi116 ceph-mon[35411]: pgmap v12398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:15:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:56 smithi114 ceph-mon[30820]: pgmap v12399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:15:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:56 smithi116 ceph-mon[35411]: pgmap v12399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:15:58.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:15:58 smithi116 ceph-mon[35411]: pgmap v12400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:15:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:15:58 smithi114 ceph-mon[30820]: pgmap v12400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:00 smithi114 ceph-mon[30820]: pgmap v12401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:00 smithi116 ceph-mon[35411]: pgmap v12401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:02 smithi114 ceph-mon[30820]: pgmap v12402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:02 smithi116 ceph-mon[35411]: pgmap v12402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:04 smithi114 ceph-mon[30820]: pgmap v12403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:04 smithi116 ceph-mon[35411]: pgmap v12403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:06 smithi114 ceph-mon[30820]: pgmap v12404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:16:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:06 smithi116 ceph-mon[35411]: pgmap v12404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:16:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:16:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:16:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:08 smithi114 ceph-mon[30820]: pgmap v12405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:08 smithi116 ceph-mon[35411]: pgmap v12405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:10.417 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:16:09.940+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:16:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:16:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:16:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:16:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:16:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:16:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:16:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:16:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:16:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:16:09 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:16:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:10 smithi114 ceph-mon[30820]: pgmap v12406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:16:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:16:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:16:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:10 smithi116 ceph-mon[35411]: pgmap v12406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:16:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:16:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:16:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:11 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:16:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:16:11.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:11 smithi114 ceph-mon[30820]: pgmap v12407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:16:11.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:11 smithi114 ceph-mon[30820]: pgmap v12408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:16:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:11 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:16:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:16:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:16:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:16:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:16:11.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:16:11.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:16:11.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:16:11.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:16:11.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:11 smithi116 ceph-mon[35411]: pgmap v12407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:16:11.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:11 smithi116 ceph-mon[35411]: pgmap v12408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:16:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:13 smithi114 ceph-mon[30820]: pgmap v12409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:16:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:13 smithi116 ceph-mon[35411]: pgmap v12409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:16:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:15 smithi114 ceph-mon[30820]: pgmap v12410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:16:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:15 smithi116 ceph-mon[35411]: pgmap v12410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:16:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:17 smithi114 ceph-mon[30820]: pgmap v12411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:16:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:17 smithi116 ceph-mon[35411]: pgmap v12411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:16:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:19 smithi114 ceph-mon[30820]: pgmap v12412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:16:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:19 smithi116 ceph-mon[35411]: pgmap v12412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:16:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:21 smithi114 ceph-mon[30820]: pgmap v12413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:16:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:21 smithi116 ceph-mon[35411]: pgmap v12413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:16:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:23 smithi114 ceph-mon[30820]: pgmap v12414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:23 smithi116 ceph-mon[35411]: pgmap v12414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:25 smithi114 ceph-mon[30820]: pgmap v12415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:25 smithi116 ceph-mon[35411]: pgmap v12415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:27 smithi114 ceph-mon[30820]: pgmap v12416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:27 smithi116 ceph-mon[35411]: pgmap v12416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:29 smithi114 ceph-mon[30820]: pgmap v12417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:29 smithi116 ceph-mon[35411]: pgmap v12417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:31 smithi114 ceph-mon[30820]: pgmap v12418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:16:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:31 smithi116 ceph-mon[35411]: pgmap v12418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:16:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:33 smithi114 ceph-mon[30820]: pgmap v12419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:33 smithi116 ceph-mon[35411]: pgmap v12419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:35 smithi114 ceph-mon[30820]: pgmap v12420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:35 smithi116 ceph-mon[35411]: pgmap v12420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:37 smithi114 ceph-mon[30820]: pgmap v12421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:37 smithi116 ceph-mon[35411]: pgmap v12421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:39 smithi114 ceph-mon[30820]: pgmap v12422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:39 smithi116 ceph-mon[35411]: pgmap v12422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:41 smithi114 ceph-mon[30820]: pgmap v12423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:16:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:41 smithi116 ceph-mon[35411]: pgmap v12423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:16:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:43 smithi114 ceph-mon[30820]: pgmap v12424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:43 smithi116 ceph-mon[35411]: pgmap v12424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:45 smithi114 ceph-mon[30820]: pgmap v12425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:45 smithi116 ceph-mon[35411]: pgmap v12425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:47 smithi114 ceph-mon[30820]: pgmap v12426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:47 smithi116 ceph-mon[35411]: pgmap v12426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:49 smithi114 ceph-mon[30820]: pgmap v12427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:49 smithi116 ceph-mon[35411]: pgmap v12427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:51 smithi114 ceph-mon[30820]: pgmap v12428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:16:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:51 smithi116 ceph-mon[35411]: pgmap v12428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:16:53.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:53 smithi114 ceph-mon[30820]: pgmap v12429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:53 smithi116 ceph-mon[35411]: pgmap v12429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:16:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:55 smithi114 ceph-mon[30820]: pgmap v12430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:55 smithi116 ceph-mon[35411]: pgmap v12430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:57 smithi114 ceph-mon[30820]: pgmap v12431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:57 smithi116 ceph-mon[35411]: pgmap v12431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:16:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:16:59 smithi114 ceph-mon[30820]: pgmap v12432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:16:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:16:59 smithi116 ceph-mon[35411]: pgmap v12432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:00 smithi114 ceph-mon[30820]: pgmap v12433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:00 smithi116 ceph-mon[35411]: pgmap v12433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:03 smithi114 ceph-mon[30820]: pgmap v12434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:03 smithi116 ceph-mon[35411]: pgmap v12434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:05 smithi114 ceph-mon[30820]: pgmap v12435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:05 smithi116 ceph-mon[35411]: pgmap v12435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:07 smithi116 ceph-mon[35411]: pgmap v12436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:17:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:07 smithi114 ceph-mon[30820]: pgmap v12436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:17:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:09 smithi116 ceph-mon[35411]: pgmap v12437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:09 smithi114 ceph-mon[30820]: pgmap v12437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:17:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:17:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:11 smithi114 ceph-mon[30820]: pgmap v12438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:11 smithi116 ceph-mon[35411]: pgmap v12438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-mon[30820]: pgmap v12439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:17:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:17:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:17:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:17:13.300+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:17:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:17:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:17:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:17:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:17:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:17:13.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:17:13.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:17:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:17:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:17:13.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:17:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:17:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:13 smithi116 ceph-mon[35411]: pgmap v12439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:17:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:17:13.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:17:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:17:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:17:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:17:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:14 smithi114 ceph-mon[30820]: pgmap v12440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:17:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:14 smithi114 ceph-mon[30820]: pgmap v12441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:17:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:17:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:17:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:17:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:17:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:17:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:17:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:17:14.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:17:14.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:17:14.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:17:14.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:17:14.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:14 smithi116 ceph-mon[35411]: pgmap v12440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:17:14.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:14 smithi116 ceph-mon[35411]: pgmap v12441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:17:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:16 smithi114 ceph-mon[30820]: pgmap v12442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:17:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:16 smithi116 ceph-mon[35411]: pgmap v12442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:17:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:18 smithi114 ceph-mon[30820]: pgmap v12443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:17:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:18 smithi116 ceph-mon[35411]: pgmap v12443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:17:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:20 smithi114 ceph-mon[30820]: pgmap v12444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:17:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:20 smithi116 ceph-mon[35411]: pgmap v12444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:17:22.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:22 smithi114 ceph-mon[30820]: pgmap v12445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:17:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:22 smithi116 ceph-mon[35411]: pgmap v12445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:17:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:24 smithi114 ceph-mon[30820]: pgmap v12446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:17:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:24 smithi116 ceph-mon[35411]: pgmap v12446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:17:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:26 smithi114 ceph-mon[30820]: pgmap v12447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:17:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:26 smithi116 ceph-mon[35411]: pgmap v12447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:17:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:28 smithi114 ceph-mon[30820]: pgmap v12448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:17:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:28 smithi116 ceph-mon[35411]: pgmap v12448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:17:30.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:30 smithi116 ceph-mon[35411]: pgmap v12449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:30 smithi114 ceph-mon[30820]: pgmap v12449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:32 smithi114 ceph-mon[30820]: pgmap v12450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:32 smithi116 ceph-mon[35411]: pgmap v12450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:34 smithi114 ceph-mon[30820]: pgmap v12451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:34 smithi116 ceph-mon[35411]: pgmap v12451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:36 smithi114 ceph-mon[30820]: pgmap v12452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:17:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:36 smithi116 ceph-mon[35411]: pgmap v12452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:17:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:38 smithi114 ceph-mon[30820]: pgmap v12453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:17:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:38 smithi116 ceph-mon[35411]: pgmap v12453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:17:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:40 smithi114 ceph-mon[30820]: pgmap v12454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:40 smithi116 ceph-mon[35411]: pgmap v12454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:42 smithi114 ceph-mon[30820]: pgmap v12455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:42 smithi116 ceph-mon[35411]: pgmap v12455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:44 smithi114 ceph-mon[30820]: pgmap v12456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:44 smithi116 ceph-mon[35411]: pgmap v12456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:46 smithi114 ceph-mon[30820]: pgmap v12457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:17:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:46 smithi116 ceph-mon[35411]: pgmap v12457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:17:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:48 smithi114 ceph-mon[30820]: pgmap v12458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:17:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:48 smithi116 ceph-mon[35411]: pgmap v12458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:17:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:50 smithi114 ceph-mon[30820]: pgmap v12459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:50 smithi116 ceph-mon[35411]: pgmap v12459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:17:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:52 smithi114 ceph-mon[30820]: pgmap v12460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:52 smithi116 ceph-mon[35411]: pgmap v12460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:17:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:54 smithi114 ceph-mon[30820]: pgmap v12461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:54 smithi116 ceph-mon[35411]: pgmap v12461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:17:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:56 smithi114 ceph-mon[30820]: pgmap v12462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:17:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:56 smithi116 ceph-mon[35411]: pgmap v12462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:17:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:17:58 smithi114 ceph-mon[30820]: pgmap v12463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:17:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:17:58 smithi116 ceph-mon[35411]: pgmap v12463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:18:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:00 smithi114 ceph-mon[30820]: pgmap v12464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:18:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:00 smithi116 ceph-mon[35411]: pgmap v12464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:18:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:02 smithi114 ceph-mon[30820]: pgmap v12465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:02.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:02 smithi116 ceph-mon[35411]: pgmap v12465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:04 smithi114 ceph-mon[30820]: pgmap v12466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:04 smithi116 ceph-mon[35411]: pgmap v12466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:06 smithi114 ceph-mon[30820]: pgmap v12467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:18:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:06 smithi116 ceph-mon[35411]: pgmap v12467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:18:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:08 smithi114 ceph-mon[30820]: pgmap v12468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:18:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:08 smithi116 ceph-mon[35411]: pgmap v12468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:18:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:10 smithi114 ceph-mon[30820]: pgmap v12469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:18:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:10 smithi116 ceph-mon[35411]: pgmap v12469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:18:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:12 smithi114 ceph-mon[30820]: pgmap v12470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:12 smithi116 ceph-mon[35411]: pgmap v12470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:18:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:18:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: pgmap v12471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: pgmap v12471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:14.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:18:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:15 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:18:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:15 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:18:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:15 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:18:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:15 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:18:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:15 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:18:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:15 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:18:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:15 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:18:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:15 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:18:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-mon[30820]: pgmap v12472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:18:16.917 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:18:16.661+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:18:16.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:18:16.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:18:16 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:18:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:16 smithi116 ceph-mon[35411]: pgmap v12472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:18:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:18:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:18:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:18:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:18:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:18:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:18:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:18 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:18:18.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:18:18.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:18:18.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:18:18.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:18:18.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:18:18.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:18:18.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:18:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:18:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:18:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:18:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:18:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:18:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:18:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:18 smithi114 ceph-mon[30820]: pgmap v12473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T13:18:18.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:18 smithi114 ceph-mon[30820]: pgmap v12474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:18:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:18 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:18:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:18:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:18:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:18:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:18:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:18:18.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:18:18.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:18:18.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:18:18.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:18 smithi116 ceph-mon[35411]: pgmap v12473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T13:18:18.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:18 smithi116 ceph-mon[35411]: pgmap v12474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:18:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:20 smithi114 ceph-mon[30820]: pgmap v12475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:18:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:20 smithi116 ceph-mon[35411]: pgmap v12475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:18:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:22 smithi114 ceph-mon[30820]: pgmap v12476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:18:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:22 smithi116 ceph-mon[35411]: pgmap v12476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:18:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:24 smithi114 ceph-mon[30820]: pgmap v12477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:18:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:24 smithi116 ceph-mon[35411]: pgmap v12477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:18:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:26 smithi114 ceph-mon[30820]: pgmap v12478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:18:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:26 smithi116 ceph-mon[35411]: pgmap v12478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:18:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:27 smithi114 ceph-mon[30820]: pgmap v12479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:18:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:27 smithi116 ceph-mon[35411]: pgmap v12479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:18:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:29 smithi114 ceph-mon[30820]: pgmap v12480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:29 smithi116 ceph-mon[35411]: pgmap v12480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:31.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:31 smithi116 ceph-mon[35411]: pgmap v12481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:18:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:31 smithi114 ceph-mon[30820]: pgmap v12481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:18:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:33 smithi114 ceph-mon[30820]: pgmap v12482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:33 smithi116 ceph-mon[35411]: pgmap v12482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:35 smithi114 ceph-mon[30820]: pgmap v12483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:18:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:35 smithi116 ceph-mon[35411]: pgmap v12483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:18:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:37 smithi114 ceph-mon[30820]: pgmap v12484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:37 smithi116 ceph-mon[35411]: pgmap v12484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:39 smithi114 ceph-mon[30820]: pgmap v12485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:39 smithi116 ceph-mon[35411]: pgmap v12485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:41 smithi114 ceph-mon[30820]: pgmap v12486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:18:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:41 smithi116 ceph-mon[35411]: pgmap v12486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:18:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:43 smithi114 ceph-mon[30820]: pgmap v12487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:18:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:43 smithi116 ceph-mon[35411]: pgmap v12487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:18:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:45 smithi114 ceph-mon[30820]: pgmap v12488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:18:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:45 smithi116 ceph-mon[35411]: pgmap v12488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:18:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:47 smithi114 ceph-mon[30820]: pgmap v12489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:47 smithi116 ceph-mon[35411]: pgmap v12489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:49 smithi114 ceph-mon[30820]: pgmap v12490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:18:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:49 smithi116 ceph-mon[35411]: pgmap v12490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:18:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:51 smithi114 ceph-mon[30820]: pgmap v12491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:18:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:51 smithi116 ceph-mon[35411]: pgmap v12491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:18:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:53 smithi114 ceph-mon[30820]: pgmap v12492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:53 smithi116 ceph-mon[35411]: pgmap v12492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:18:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:55 smithi114 ceph-mon[30820]: pgmap v12493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:18:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:55 smithi116 ceph-mon[35411]: pgmap v12493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:18:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:57 smithi114 ceph-mon[30820]: pgmap v12494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:18:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:57 smithi116 ceph-mon[35411]: pgmap v12494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:18:59 smithi114 ceph-mon[30820]: pgmap v12495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:18:59 smithi116 ceph-mon[35411]: pgmap v12495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:01 smithi114 ceph-mon[30820]: pgmap v12496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:01 smithi116 ceph-mon[35411]: pgmap v12496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:03 smithi114 ceph-mon[30820]: pgmap v12497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:03 smithi116 ceph-mon[35411]: pgmap v12497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:06 smithi114 ceph-mon[30820]: pgmap v12498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:19:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:06 smithi116 ceph-mon[35411]: pgmap v12498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:19:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:08 smithi114 ceph-mon[30820]: pgmap v12499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:08 smithi116 ceph-mon[35411]: pgmap v12499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:10 smithi114 ceph-mon[30820]: pgmap v12500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:10 smithi116 ceph-mon[35411]: pgmap v12500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:12 smithi114 ceph-mon[30820]: pgmap v12501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:12 smithi116 ceph-mon[35411]: pgmap v12501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:14 smithi114 ceph-mon[30820]: pgmap v12502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:14 smithi116 ceph-mon[35411]: pgmap v12502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:16 smithi114 ceph-mon[30820]: pgmap v12503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:19:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:16 smithi116 ceph-mon[35411]: pgmap v12503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:19:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:19:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:19:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:18 smithi114 ceph-mon[30820]: pgmap v12504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:19:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:19:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:18 smithi116 ceph-mon[35411]: pgmap v12504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:19:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:19:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:19:20.020+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:19:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:19:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:19:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:19:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:19:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:19:20.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:19:20.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:19:20.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:19:20.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:19:20.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:19:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-mon[30820]: pgmap v12505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:19:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:19:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:19:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:20 smithi116 ceph-mon[35411]: pgmap v12505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:19:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:19:20.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:19:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:19:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:19:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:19:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:19:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:19:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:19:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:19:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:19:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:19:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:19:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:21 smithi114 ceph-mon[30820]: pgmap v12506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:19:21.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:21 smithi114 ceph-mon[30820]: pgmap v12507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:19:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:19:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:19:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:19:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:19:21.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:19:21.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:19:21.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:19:21.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:19:21.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:19:21.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:19:21.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:21 smithi116 ceph-mon[35411]: pgmap v12506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:19:21.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:21 smithi116 ceph-mon[35411]: pgmap v12507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:19:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:23 smithi114 ceph-mon[30820]: pgmap v12508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:19:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:23 smithi116 ceph-mon[35411]: pgmap v12508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:19:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:25 smithi114 ceph-mon[30820]: pgmap v12509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:19:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:25 smithi116 ceph-mon[35411]: pgmap v12509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:19:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:27 smithi114 ceph-mon[30820]: pgmap v12510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:19:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:27 smithi116 ceph-mon[35411]: pgmap v12510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:19:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:29 smithi114 ceph-mon[30820]: pgmap v12511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:19:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:29 smithi116 ceph-mon[35411]: pgmap v12511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:19:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:31 smithi114 ceph-mon[30820]: pgmap v12512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:19:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:31 smithi116 ceph-mon[35411]: pgmap v12512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:19:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:33 smithi114 ceph-mon[30820]: pgmap v12513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:33 smithi116 ceph-mon[35411]: pgmap v12513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:35 smithi114 ceph-mon[30820]: pgmap v12514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:35 smithi116 ceph-mon[35411]: pgmap v12514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:37 smithi114 ceph-mon[30820]: pgmap v12515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:37 smithi116 ceph-mon[35411]: pgmap v12515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:39 smithi114 ceph-mon[30820]: pgmap v12516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:39 smithi116 ceph-mon[35411]: pgmap v12516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:41 smithi114 ceph-mon[30820]: pgmap v12517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:19:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:41 smithi116 ceph-mon[35411]: pgmap v12517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:19:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:43 smithi114 ceph-mon[30820]: pgmap v12518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:43 smithi116 ceph-mon[35411]: pgmap v12518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:45 smithi114 ceph-mon[30820]: pgmap v12519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:45 smithi116 ceph-mon[35411]: pgmap v12519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:47 smithi114 ceph-mon[30820]: pgmap v12520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:47 smithi116 ceph-mon[35411]: pgmap v12520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:49 smithi114 ceph-mon[30820]: pgmap v12521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:49 smithi116 ceph-mon[35411]: pgmap v12521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:51 smithi114 ceph-mon[30820]: pgmap v12522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:19:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:51 smithi116 ceph-mon[35411]: pgmap v12522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:19:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:53 smithi114 ceph-mon[30820]: pgmap v12523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:53 smithi116 ceph-mon[35411]: pgmap v12523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:19:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:55 smithi114 ceph-mon[30820]: pgmap v12524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:55 smithi116 ceph-mon[35411]: pgmap v12524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:57 smithi114 ceph-mon[30820]: pgmap v12525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:57 smithi116 ceph-mon[35411]: pgmap v12525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:19:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:19:59 smithi114 ceph-mon[30820]: pgmap v12526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:19:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:19:59 smithi116 ceph-mon[35411]: pgmap v12526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:20:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:20:01.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:01 smithi116 ceph-mon[35411]: pgmap v12527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:01 smithi114 ceph-mon[30820]: pgmap v12527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:03 smithi114 ceph-mon[30820]: pgmap v12528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:03 smithi116 ceph-mon[35411]: pgmap v12528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:05 smithi114 ceph-mon[30820]: pgmap v12529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:05 smithi116 ceph-mon[35411]: pgmap v12529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:07 smithi114 ceph-mon[30820]: pgmap v12530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:07 smithi116 ceph-mon[35411]: pgmap v12530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:09 smithi114 ceph-mon[30820]: pgmap v12531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:09 smithi116 ceph-mon[35411]: pgmap v12531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:11 smithi114 ceph-mon[30820]: pgmap v12532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:11 smithi116 ceph-mon[35411]: pgmap v12532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:13 smithi114 ceph-mon[30820]: pgmap v12533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:13 smithi116 ceph-mon[35411]: pgmap v12533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:15 smithi114 ceph-mon[30820]: pgmap v12534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:15 smithi116 ceph-mon[35411]: pgmap v12534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:17 smithi114 ceph-mon[30820]: pgmap v12535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:17 smithi116 ceph-mon[35411]: pgmap v12535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:19 smithi114 ceph-mon[30820]: pgmap v12536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:19 smithi116 ceph-mon[35411]: pgmap v12536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:20:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:20:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:21 smithi114 ceph-mon[30820]: pgmap v12537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:20:21.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:20:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:21 smithi116 ceph-mon[35411]: pgmap v12537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:20:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:20:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-mon[30820]: pgmap v12538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:20:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:20:23.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:20:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:20:23.381+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:20:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:20:23.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:20:23.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:20:23.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:20:23.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:20:23.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:20:23.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:20:23.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:20:23.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:20:23.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:20:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:20:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:23 smithi116 ceph-mon[35411]: pgmap v12538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:20:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:20:23.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:20:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:20:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:20:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:20:24.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:24 smithi114 ceph-mon[30820]: pgmap v12539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:20:24.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:24 smithi114 ceph-mon[30820]: pgmap v12540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:20:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:20:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:20:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:20:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:20:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:20:24.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:20:24.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:20:24.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:20:24.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:20:24.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:24 smithi116 ceph-mon[35411]: pgmap v12539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:20:24.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:24 smithi116 ceph-mon[35411]: pgmap v12540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:26 smithi114 ceph-mon[30820]: pgmap v12541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:26 smithi116 ceph-mon[35411]: pgmap v12541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:27 smithi114 ceph-mon[30820]: pgmap v12542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:27 smithi116 ceph-mon[35411]: pgmap v12542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:30 smithi114 ceph-mon[30820]: pgmap v12543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:30 smithi116 ceph-mon[35411]: pgmap v12543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:32.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:32 smithi116 ceph-mon[35411]: pgmap v12544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:32 smithi114 ceph-mon[30820]: pgmap v12544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:34 smithi114 ceph-mon[30820]: pgmap v12545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:34 smithi116 ceph-mon[35411]: pgmap v12545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:20:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:36 smithi114 ceph-mon[30820]: pgmap v12546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:36 smithi116 ceph-mon[35411]: pgmap v12546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:38 smithi114 ceph-mon[30820]: pgmap v12547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:38 smithi116 ceph-mon[35411]: pgmap v12547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:40 smithi114 ceph-mon[30820]: pgmap v12548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:40 smithi116 ceph-mon[35411]: pgmap v12548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:42 smithi114 ceph-mon[30820]: pgmap v12549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:42 smithi116 ceph-mon[35411]: pgmap v12549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:44 smithi114 ceph-mon[30820]: pgmap v12550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:44 smithi116 ceph-mon[35411]: pgmap v12550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:46 smithi114 ceph-mon[30820]: pgmap v12551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:46 smithi116 ceph-mon[35411]: pgmap v12551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:48 smithi114 ceph-mon[30820]: pgmap v12552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:48 smithi116 ceph-mon[35411]: pgmap v12552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:50 smithi114 ceph-mon[30820]: pgmap v12553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:50 smithi116 ceph-mon[35411]: pgmap v12553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:20:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:52 smithi114 ceph-mon[30820]: pgmap v12554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:52 smithi116 ceph-mon[35411]: pgmap v12554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:20:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:54 smithi114 ceph-mon[30820]: pgmap v12555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:54 smithi116 ceph-mon[35411]: pgmap v12555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:20:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:56 smithi114 ceph-mon[30820]: pgmap v12556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:56 smithi116 ceph-mon[35411]: pgmap v12556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:20:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:20:58 smithi114 ceph-mon[30820]: pgmap v12557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:20:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:20:58 smithi116 ceph-mon[35411]: pgmap v12557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:21:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:00 smithi114 ceph-mon[30820]: pgmap v12558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:21:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:00 smithi116 ceph-mon[35411]: pgmap v12558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:21:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:02 smithi114 ceph-mon[30820]: pgmap v12559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:02 smithi116 ceph-mon[35411]: pgmap v12559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:04 smithi114 ceph-mon[30820]: pgmap v12560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:21:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:04 smithi116 ceph-mon[35411]: pgmap v12560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:21:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:06 smithi114 ceph-mon[30820]: pgmap v12561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:06 smithi116 ceph-mon[35411]: pgmap v12561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:08 smithi114 ceph-mon[30820]: pgmap v12562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:08 smithi116 ceph-mon[35411]: pgmap v12562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:10 smithi114 ceph-mon[30820]: pgmap v12563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:21:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:10 smithi116 ceph-mon[35411]: pgmap v12563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:21:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:12 smithi114 ceph-mon[30820]: pgmap v12564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:12 smithi116 ceph-mon[35411]: pgmap v12564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:14 smithi114 ceph-mon[30820]: pgmap v12565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:14 smithi116 ceph-mon[35411]: pgmap v12565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:16 smithi114 ceph-mon[30820]: pgmap v12566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:16 smithi116 ceph-mon[35411]: pgmap v12566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:18 smithi114 ceph-mon[30820]: pgmap v12567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:18 smithi116 ceph-mon[35411]: pgmap v12567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:20 smithi114 ceph-mon[30820]: pgmap v12568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:21:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:20 smithi116 ceph-mon[35411]: pgmap v12568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:21:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:22 smithi114 ceph-mon[30820]: pgmap v12569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:22 smithi116 ceph-mon[35411]: pgmap v12569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:21:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:21:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:24 smithi114 ceph-mon[30820]: pgmap v12570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:24 smithi116 ceph-mon[35411]: pgmap v12570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:25 smithi114 ceph-mon[30820]: pgmap v12571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:25 smithi116 ceph-mon[35411]: pgmap v12571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:21:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:21:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:21:26.740+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:21:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:21:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:21:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:21:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:21:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:21:27.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:21:27.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:21:27.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:21:27.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:21:26 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:21:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:26 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:21:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:27 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:21:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:21:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:21:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:21:28.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:27 smithi114 ceph-mon[30820]: pgmap v12572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:21:28.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:27 smithi114 ceph-mon[30820]: pgmap v12573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:27 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:21:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:21:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:27 smithi116 ceph-mon[35411]: pgmap v12572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:21:28.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:27 smithi116 ceph-mon[35411]: pgmap v12573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:21:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:29 smithi114 ceph-mon[30820]: pgmap v12574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:21:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:29 smithi116 ceph-mon[35411]: pgmap v12574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:21:32.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:31 smithi116 ceph-mon[35411]: pgmap v12575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:21:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:31 smithi114 ceph-mon[30820]: pgmap v12575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:21:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:33 smithi114 ceph-mon[30820]: pgmap v12576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:21:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:33 smithi116 ceph-mon[35411]: pgmap v12576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:21:36.354 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:35 smithi114 ceph-mon[30820]: pgmap v12577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:21:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:35 smithi116 ceph-mon[35411]: pgmap v12577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:21:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:37 smithi114 ceph-mon[30820]: pgmap v12578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:21:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:37 smithi116 ceph-mon[35411]: pgmap v12578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:21:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:39 smithi114 ceph-mon[30820]: pgmap v12579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:39 smithi116 ceph-mon[35411]: pgmap v12579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:41 smithi114 ceph-mon[30820]: pgmap v12580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:41 smithi116 ceph-mon[35411]: pgmap v12580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:43 smithi114 ceph-mon[30820]: pgmap v12581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:43 smithi116 ceph-mon[35411]: pgmap v12581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:45 smithi114 ceph-mon[30820]: pgmap v12582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:21:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:45 smithi116 ceph-mon[35411]: pgmap v12582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:21:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:47 smithi114 ceph-mon[30820]: pgmap v12583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:47 smithi116 ceph-mon[35411]: pgmap v12583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:49 smithi114 ceph-mon[30820]: pgmap v12584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:49 smithi116 ceph-mon[35411]: pgmap v12584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:51 smithi114 ceph-mon[30820]: pgmap v12585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:51 smithi116 ceph-mon[35411]: pgmap v12585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:21:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:54 smithi114 ceph-mon[30820]: pgmap v12586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:54 smithi116 ceph-mon[35411]: pgmap v12586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:21:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:56 smithi114 ceph-mon[30820]: pgmap v12587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:21:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:56 smithi116 ceph-mon[35411]: pgmap v12587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:21:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:21:58 smithi114 ceph-mon[30820]: pgmap v12588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:21:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:21:58 smithi116 ceph-mon[35411]: pgmap v12588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:00 smithi114 ceph-mon[30820]: pgmap v12589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:00 smithi116 ceph-mon[35411]: pgmap v12589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:02 smithi114 ceph-mon[30820]: pgmap v12590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:02 smithi116 ceph-mon[35411]: pgmap v12590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:04 smithi114 ceph-mon[30820]: pgmap v12591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:04 smithi116 ceph-mon[35411]: pgmap v12591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:06 smithi114 ceph-mon[30820]: pgmap v12592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:22:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:06 smithi116 ceph-mon[35411]: pgmap v12592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:22:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:08 smithi114 ceph-mon[30820]: pgmap v12593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:08 smithi116 ceph-mon[35411]: pgmap v12593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:10 smithi114 ceph-mon[30820]: pgmap v12594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:10 smithi116 ceph-mon[35411]: pgmap v12594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:12 smithi114 ceph-mon[30820]: pgmap v12595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:12 smithi116 ceph-mon[35411]: pgmap v12595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:14 smithi114 ceph-mon[30820]: pgmap v12596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:14 smithi116 ceph-mon[35411]: pgmap v12596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:16 smithi114 ceph-mon[30820]: pgmap v12597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:22:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:16 smithi116 ceph-mon[35411]: pgmap v12597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:22:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:18 smithi114 ceph-mon[30820]: pgmap v12598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:18 smithi116 ceph-mon[35411]: pgmap v12598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:20 smithi114 ceph-mon[30820]: pgmap v12599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:20 smithi116 ceph-mon[35411]: pgmap v12599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:22 smithi114 ceph-mon[30820]: pgmap v12600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:22 smithi116 ceph-mon[35411]: pgmap v12600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:24 smithi114 ceph-mon[30820]: pgmap v12601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:24 smithi116 ceph-mon[35411]: pgmap v12601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:26 smithi114 ceph-mon[30820]: pgmap v12602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:22:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:26 smithi116 ceph-mon[35411]: pgmap v12602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:22:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:22:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:22:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:28 smithi116 ceph-mon[35411]: pgmap v12603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:28.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:28 smithi114 ceph-mon[30820]: pgmap v12603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-mon[30820]: pgmap v12604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:22:30.036+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:22:30.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:22:30.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:22:30.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:22:30.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:22:30.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:22:30.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:22:30.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:22:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:30 smithi116 ceph-mon[35411]: pgmap v12604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:22:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:22:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:22:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:22:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:22:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:22:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:22:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:22:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:22:31.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:22:31.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:22:31.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:22:31.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:31 smithi114 ceph-mon[30820]: pgmap v12605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T13:22:31.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:31 smithi114 ceph-mon[30820]: pgmap v12606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:22:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:22:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:22:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:22:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:22:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:22:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:22:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:22:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:22:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:22:31.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:22:31.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:22:31.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:31 smithi116 ceph-mon[35411]: pgmap v12605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T13:22:31.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:31 smithi116 ceph-mon[35411]: pgmap v12606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:22:33.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:33 smithi116 ceph-mon[35411]: pgmap v12607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:22:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:33 smithi114 ceph-mon[30820]: pgmap v12607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:22:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:35 smithi114 ceph-mon[30820]: pgmap v12608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:22:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:35 smithi116 ceph-mon[35411]: pgmap v12608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:22:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:37 smithi114 ceph-mon[30820]: pgmap v12609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:22:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:37 smithi116 ceph-mon[35411]: pgmap v12609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:22:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:39 smithi114 ceph-mon[30820]: pgmap v12610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:22:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:39 smithi116 ceph-mon[35411]: pgmap v12610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:22:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:41 smithi114 ceph-mon[30820]: pgmap v12611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:22:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:41 smithi116 ceph-mon[35411]: pgmap v12611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:22:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:43 smithi114 ceph-mon[30820]: pgmap v12612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:43 smithi116 ceph-mon[35411]: pgmap v12612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:45 smithi114 ceph-mon[30820]: pgmap v12613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:45 smithi116 ceph-mon[35411]: pgmap v12613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:47 smithi114 ceph-mon[30820]: pgmap v12614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:47 smithi116 ceph-mon[35411]: pgmap v12614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:49 smithi114 ceph-mon[30820]: pgmap v12615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:49 smithi116 ceph-mon[35411]: pgmap v12615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:51 smithi114 ceph-mon[30820]: pgmap v12616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:22:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:51 smithi116 ceph-mon[35411]: pgmap v12616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:22:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:53 smithi114 ceph-mon[30820]: pgmap v12617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:53 smithi116 ceph-mon[35411]: pgmap v12617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:22:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:55 smithi114 ceph-mon[30820]: pgmap v12618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:55 smithi116 ceph-mon[35411]: pgmap v12618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:57 smithi114 ceph-mon[30820]: pgmap v12619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:57 smithi116 ceph-mon[35411]: pgmap v12619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:22:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:22:59 smithi114 ceph-mon[30820]: pgmap v12620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:22:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:22:59 smithi116 ceph-mon[35411]: pgmap v12620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:01.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:01 smithi114 ceph-mon[30820]: pgmap v12621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:01 smithi116 ceph-mon[35411]: pgmap v12621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:03 smithi114 ceph-mon[30820]: pgmap v12622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:03 smithi116 ceph-mon[35411]: pgmap v12622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:05 smithi114 ceph-mon[30820]: pgmap v12623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:05 smithi116 ceph-mon[35411]: pgmap v12623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:07 smithi114 ceph-mon[30820]: pgmap v12624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:07 smithi116 ceph-mon[35411]: pgmap v12624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:09 smithi114 ceph-mon[30820]: pgmap v12625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:09 smithi116 ceph-mon[35411]: pgmap v12625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:11 smithi114 ceph-mon[30820]: pgmap v12626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:11 smithi116 ceph-mon[35411]: pgmap v12626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:13 smithi114 ceph-mon[30820]: pgmap v12627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:13 smithi116 ceph-mon[35411]: pgmap v12627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:15 smithi114 ceph-mon[30820]: pgmap v12628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:15 smithi116 ceph-mon[35411]: pgmap v12628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:17 smithi114 ceph-mon[30820]: pgmap v12629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:17 smithi116 ceph-mon[35411]: pgmap v12629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:19 smithi114 ceph-mon[30820]: pgmap v12630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:19 smithi116 ceph-mon[35411]: pgmap v12630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:21 smithi114 ceph-mon[30820]: pgmap v12631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:21 smithi116 ceph-mon[35411]: pgmap v12631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:23 smithi114 ceph-mon[30820]: pgmap v12632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:23 smithi116 ceph-mon[35411]: pgmap v12632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:25 smithi114 ceph-mon[30820]: pgmap v12633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:25 smithi116 ceph-mon[35411]: pgmap v12633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:27 smithi114 ceph-mon[30820]: pgmap v12634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:27 smithi116 ceph-mon[35411]: pgmap v12634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:29 smithi114 ceph-mon[30820]: pgmap v12635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:29 smithi116 ceph-mon[35411]: pgmap v12635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:23:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:23:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:31 smithi114 ceph-mon[30820]: pgmap v12636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:31 smithi116 ceph-mon[35411]: pgmap v12636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-mon[30820]: pgmap v12637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:23:33.396+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:23:33.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:23:33.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:23:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:23:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:33 smithi116 ceph-mon[35411]: pgmap v12637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:23:34.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:23:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:23:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:23:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:23:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:23:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:34 smithi114 ceph-mon[30820]: pgmap v12638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:23:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:34 smithi114 ceph-mon[30820]: pgmap v12639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:34.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:23:34.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:23:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:23:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:23:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:23:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:23:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:34 smithi116 ceph-mon[35411]: pgmap v12638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:23:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:34 smithi116 ceph-mon[35411]: pgmap v12639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:34.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:23:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:36 smithi114 ceph-mon[30820]: pgmap v12640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:36 smithi116 ceph-mon[35411]: pgmap v12640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:38 smithi114 ceph-mon[30820]: pgmap v12641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:38 smithi116 ceph-mon[35411]: pgmap v12641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:40 smithi114 ceph-mon[30820]: pgmap v12642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:40 smithi116 ceph-mon[35411]: pgmap v12642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:42 smithi114 ceph-mon[30820]: pgmap v12643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:42 smithi116 ceph-mon[35411]: pgmap v12643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:23:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:44 smithi114 ceph-mon[30820]: pgmap v12644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:23:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:44 smithi116 ceph-mon[35411]: pgmap v12644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:23:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:46 smithi114 ceph-mon[30820]: pgmap v12645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:46 smithi116 ceph-mon[35411]: pgmap v12645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:48 smithi114 ceph-mon[30820]: pgmap v12646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:48 smithi116 ceph-mon[35411]: pgmap v12646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:50 smithi114 ceph-mon[30820]: pgmap v12647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:50 smithi116 ceph-mon[35411]: pgmap v12647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:23:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:52 smithi114 ceph-mon[30820]: pgmap v12648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:52 smithi116 ceph-mon[35411]: pgmap v12648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:23:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:54 smithi114 ceph-mon[30820]: pgmap v12649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:54 smithi116 ceph-mon[35411]: pgmap v12649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:56.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:56 smithi114 ceph-mon[30820]: pgmap v12650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:56 smithi116 ceph-mon[35411]: pgmap v12650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:23:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:23:58 smithi114 ceph-mon[30820]: pgmap v12651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:23:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:23:58 smithi116 ceph-mon[35411]: pgmap v12651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:00 smithi114 ceph-mon[30820]: pgmap v12652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:00 smithi116 ceph-mon[35411]: pgmap v12652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:02 smithi114 ceph-mon[30820]: pgmap v12653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:02 smithi116 ceph-mon[35411]: pgmap v12653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:04 smithi114 ceph-mon[30820]: pgmap v12654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:04 smithi116 ceph-mon[35411]: pgmap v12654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:06 smithi114 ceph-mon[30820]: pgmap v12655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:06 smithi116 ceph-mon[35411]: pgmap v12655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:08 smithi114 ceph-mon[30820]: pgmap v12656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:08 smithi116 ceph-mon[35411]: pgmap v12656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:10 smithi114 ceph-mon[30820]: pgmap v12657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:10 smithi116 ceph-mon[35411]: pgmap v12657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:12 smithi114 ceph-mon[30820]: pgmap v12658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:12 smithi116 ceph-mon[35411]: pgmap v12658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:14.921 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:14 smithi114 ceph-mon[30820]: pgmap v12659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:14 smithi116 ceph-mon[35411]: pgmap v12659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:16 smithi114 ceph-mon[30820]: pgmap v12660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:16 smithi116 ceph-mon[35411]: pgmap v12660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:18 smithi114 ceph-mon[30820]: pgmap v12661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:18 smithi116 ceph-mon[35411]: pgmap v12661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:20 smithi114 ceph-mon[30820]: pgmap v12662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:20 smithi116 ceph-mon[35411]: pgmap v12662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:22 smithi114 ceph-mon[30820]: pgmap v12663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:22 smithi116 ceph-mon[35411]: pgmap v12663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:24 smithi114 ceph-mon[30820]: pgmap v12664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:24 smithi116 ceph-mon[35411]: pgmap v12664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:26 smithi114 ceph-mon[30820]: pgmap v12665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:26 smithi116 ceph-mon[35411]: pgmap v12665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:28 smithi114 ceph-mon[30820]: pgmap v12666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:28 smithi116 ceph-mon[35411]: pgmap v12666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:30 smithi114 ceph-mon[30820]: pgmap v12667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:30 smithi116 ceph-mon[35411]: pgmap v12667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:32 smithi114 ceph-mon[30820]: pgmap v12668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:32 smithi116 ceph-mon[35411]: pgmap v12668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:24:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:24:34.830 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:34 smithi116 ceph-mon[35411]: pgmap v12669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:34 smithi114 ceph-mon[30820]: pgmap v12669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-mon[30820]: pgmap v12670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:24:36.756+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:24:36.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:24:36.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:24:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:24:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:24:36.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:24:36 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:24:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:36 smithi116 ceph-mon[35411]: pgmap v12670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:24:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:24:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:24:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:24:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:24:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:24:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:38 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:24:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:24:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:24:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:24:38.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:24:38.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:24:38.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:24:38.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:24:38.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:24:38.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:24:38.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:24:38.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:38 smithi114 ceph-mon[30820]: pgmap v12671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:24:38.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:38 smithi114 ceph-mon[30820]: pgmap v12672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:24:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:38 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:24:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:24:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:24:38.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:24:38.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:24:38.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:24:38.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:24:38.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:24:38.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:24:38.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:24:38.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:38 smithi116 ceph-mon[35411]: pgmap v12671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:24:38.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:38 smithi116 ceph-mon[35411]: pgmap v12672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:24:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:40 smithi114 ceph-mon[30820]: pgmap v12673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:24:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:40 smithi116 ceph-mon[35411]: pgmap v12673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:24:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:42 smithi114 ceph-mon[30820]: pgmap v12674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:24:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:42 smithi116 ceph-mon[35411]: pgmap v12674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:24:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:44 smithi114 ceph-mon[30820]: pgmap v12675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:24:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:44 smithi116 ceph-mon[35411]: pgmap v12675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:24:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:46 smithi114 ceph-mon[30820]: pgmap v12676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:24:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:46 smithi116 ceph-mon[35411]: pgmap v12676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:24:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:48 smithi114 ceph-mon[30820]: pgmap v12677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:24:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:48 smithi116 ceph-mon[35411]: pgmap v12677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:24:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:50 smithi114 ceph-mon[30820]: pgmap v12678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:50 smithi116 ceph-mon[35411]: pgmap v12678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:52 smithi114 ceph-mon[30820]: pgmap v12679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:52 smithi116 ceph-mon[35411]: pgmap v12679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:24:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:54 smithi114 ceph-mon[30820]: pgmap v12680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:54 smithi116 ceph-mon[35411]: pgmap v12680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:24:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:56 smithi114 ceph-mon[30820]: pgmap v12681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:56 smithi116 ceph-mon[35411]: pgmap v12681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:24:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:24:58 smithi114 ceph-mon[30820]: pgmap v12682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:24:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:24:58 smithi116 ceph-mon[35411]: pgmap v12682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:00 smithi114 ceph-mon[30820]: pgmap v12683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:00 smithi116 ceph-mon[35411]: pgmap v12683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:02 smithi114 ceph-mon[30820]: pgmap v12684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:02 smithi116 ceph-mon[35411]: pgmap v12684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:04 smithi114 ceph-mon[30820]: pgmap v12685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:04 smithi116 ceph-mon[35411]: pgmap v12685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:06 smithi114 ceph-mon[30820]: pgmap v12686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:25:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:06 smithi116 ceph-mon[35411]: pgmap v12686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:25:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:08 smithi114 ceph-mon[30820]: pgmap v12687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:08 smithi116 ceph-mon[35411]: pgmap v12687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:10 smithi114 ceph-mon[30820]: pgmap v12688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:10 smithi116 ceph-mon[35411]: pgmap v12688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:12 smithi114 ceph-mon[30820]: pgmap v12689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:12 smithi116 ceph-mon[35411]: pgmap v12689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:14 smithi114 ceph-mon[30820]: pgmap v12690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:14 smithi116 ceph-mon[35411]: pgmap v12690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:16 smithi114 ceph-mon[30820]: pgmap v12691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:25:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:16 smithi116 ceph-mon[35411]: pgmap v12691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:25:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:18 smithi114 ceph-mon[30820]: pgmap v12692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:18 smithi116 ceph-mon[35411]: pgmap v12692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:19 smithi114 ceph-mon[30820]: pgmap v12693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:25:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:19 smithi116 ceph-mon[35411]: pgmap v12693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:25:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:21 smithi114 ceph-mon[30820]: pgmap v12694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:21 smithi116 ceph-mon[35411]: pgmap v12694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:23 smithi114 ceph-mon[30820]: pgmap v12695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:25:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:23 smithi116 ceph-mon[35411]: pgmap v12695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:25:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:25 smithi114 ceph-mon[30820]: pgmap v12696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:25:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:25 smithi116 ceph-mon[35411]: pgmap v12696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:25:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:27 smithi114 ceph-mon[30820]: pgmap v12697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:27 smithi116 ceph-mon[35411]: pgmap v12697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:29 smithi114 ceph-mon[30820]: pgmap v12698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:29 smithi116 ceph-mon[35411]: pgmap v12698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:31 smithi114 ceph-mon[30820]: pgmap v12699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:31 smithi116 ceph-mon[35411]: pgmap v12699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:33 smithi114 ceph-mon[30820]: pgmap v12700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:33 smithi116 ceph-mon[35411]: pgmap v12700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:35 smithi114 ceph-mon[30820]: pgmap v12701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:25:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:35 smithi116 ceph-mon[35411]: pgmap v12701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:25:37.910 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:37 smithi116 ceph-mon[35411]: pgmap v12702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:37.911 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:25:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:37 smithi114 ceph-mon[30820]: pgmap v12702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:25:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:39 smithi114 ceph-mon[30820]: pgmap v12703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:39 smithi116 ceph-mon[35411]: pgmap v12703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:40.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:25:40.116+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:25:40.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:25:40.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:25:40.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:25:40.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:25:40.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:25:40.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:25:40.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:25:40.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:25:40.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:25:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:25:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:25:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:25:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:25:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:25:41.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:41.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:25:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:25:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:25:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:25:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:25:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:25:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:25:41.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:25:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:25:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:25:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:41.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:42 smithi114 ceph-mon[30820]: pgmap v12704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:25:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:25:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:42 smithi116 ceph-mon[35411]: pgmap v12704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:25:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:25:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:25:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:25:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:43 smithi114 ceph-mon[30820]: pgmap v12705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:25:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:43 smithi114 ceph-mon[30820]: pgmap v12706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:25:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:43 smithi116 ceph-mon[35411]: pgmap v12705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:25:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:43 smithi116 ceph-mon[35411]: pgmap v12706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:25:45.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:45 smithi114 ceph-mon[30820]: pgmap v12707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:25:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:45 smithi116 ceph-mon[35411]: pgmap v12707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:25:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:47 smithi114 ceph-mon[30820]: pgmap v12708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:25:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:47 smithi116 ceph-mon[35411]: pgmap v12708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:25:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:49 smithi114 ceph-mon[30820]: pgmap v12709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:25:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:49 smithi116 ceph-mon[35411]: pgmap v12709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:25:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:51 smithi114 ceph-mon[30820]: pgmap v12710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:25:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:51 smithi116 ceph-mon[35411]: pgmap v12710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:25:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:53 smithi114 ceph-mon[30820]: pgmap v12711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:25:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:53 smithi116 ceph-mon[35411]: pgmap v12711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:25:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:55 smithi116 ceph-mon[35411]: pgmap v12712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:25:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:55 smithi114 ceph-mon[30820]: pgmap v12712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:25:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:56 smithi116 ceph-mon[35411]: pgmap v12713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:57.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:56 smithi114 ceph-mon[30820]: pgmap v12713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:25:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:25:59 smithi114 ceph-mon[30820]: pgmap v12714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:25:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:25:59 smithi116 ceph-mon[35411]: pgmap v12714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:01 smithi114 ceph-mon[30820]: pgmap v12715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:01 smithi116 ceph-mon[35411]: pgmap v12715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:03 smithi114 ceph-mon[30820]: pgmap v12716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:26:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:03 smithi116 ceph-mon[35411]: pgmap v12716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:26:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:05 smithi114 ceph-mon[30820]: pgmap v12717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:05 smithi116 ceph-mon[35411]: pgmap v12717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:07 smithi114 ceph-mon[30820]: pgmap v12718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:26:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:07 smithi116 ceph-mon[35411]: pgmap v12718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:26:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:09 smithi114 ceph-mon[30820]: pgmap v12719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:09 smithi116 ceph-mon[35411]: pgmap v12719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:11 smithi114 ceph-mon[30820]: pgmap v12720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:11 smithi116 ceph-mon[35411]: pgmap v12720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:13 smithi114 ceph-mon[30820]: pgmap v12721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:13 smithi116 ceph-mon[35411]: pgmap v12721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:15 smithi114 ceph-mon[30820]: pgmap v12722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:15.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:15 smithi116 ceph-mon[35411]: pgmap v12722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:17 smithi114 ceph-mon[30820]: pgmap v12723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:26:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:17 smithi116 ceph-mon[35411]: pgmap v12723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:26:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:19 smithi114 ceph-mon[30820]: pgmap v12724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:19 smithi116 ceph-mon[35411]: pgmap v12724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:21 smithi114 ceph-mon[30820]: pgmap v12725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:21 smithi116 ceph-mon[35411]: pgmap v12725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:23 smithi114 ceph-mon[30820]: pgmap v12726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:23 smithi116 ceph-mon[35411]: pgmap v12726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:25 smithi114 ceph-mon[30820]: pgmap v12727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:25 smithi116 ceph-mon[35411]: pgmap v12727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:27 smithi114 ceph-mon[30820]: pgmap v12728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:26:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:27 smithi116 ceph-mon[35411]: pgmap v12728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:26:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:29 smithi114 ceph-mon[30820]: pgmap v12729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:29 smithi116 ceph-mon[35411]: pgmap v12729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:31 smithi114 ceph-mon[30820]: pgmap v12730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:31 smithi116 ceph-mon[35411]: pgmap v12730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:33 smithi114 ceph-mon[30820]: pgmap v12731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:33 smithi116 ceph-mon[35411]: pgmap v12731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:35 smithi114 ceph-mon[30820]: pgmap v12732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:35 smithi116 ceph-mon[35411]: pgmap v12732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:37 smithi114 ceph-mon[30820]: pgmap v12733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:26:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:37 smithi116 ceph-mon[35411]: pgmap v12733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:26:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:39 smithi114 ceph-mon[30820]: pgmap v12734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:39 smithi116 ceph-mon[35411]: pgmap v12734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:41 smithi114 ceph-mon[30820]: pgmap v12735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:41 smithi116 ceph-mon[35411]: pgmap v12735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:26:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:26:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:26:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:43 smithi114 ceph-mon[30820]: pgmap v12736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:43 smithi116 ceph-mon[35411]: pgmap v12736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-mon[30820]: pgmap v12737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:26:45.717+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:26:45.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:26:45.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:26:45 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:26:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:45 smithi116 ceph-mon[35411]: pgmap v12737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:26:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:26:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:26:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:26:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:26:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:26:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:26:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:47 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:26:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:26:47.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:47 smithi114 ceph-mon[30820]: pgmap v12738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:26:47.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:47 smithi114 ceph-mon[30820]: pgmap v12739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:26:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:47 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:26:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:26:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:26:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:26:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:26:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:26:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:26:47.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:26:47.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:26:47.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:47 smithi116 ceph-mon[35411]: pgmap v12738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:26:47.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:47 smithi116 ceph-mon[35411]: pgmap v12739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:26:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:49 smithi114 ceph-mon[30820]: pgmap v12740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:26:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:49 smithi116 ceph-mon[35411]: pgmap v12740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:26:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:51 smithi114 ceph-mon[30820]: pgmap v12741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:26:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:51 smithi116 ceph-mon[35411]: pgmap v12741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:26:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:53 smithi114 ceph-mon[30820]: pgmap v12742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:26:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:53 smithi116 ceph-mon[35411]: pgmap v12742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:26:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:55 smithi114 ceph-mon[30820]: pgmap v12743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:26:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:55 smithi116 ceph-mon[35411]: pgmap v12743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:26:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:57 smithi114 ceph-mon[30820]: pgmap v12744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:26:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:57 smithi116 ceph-mon[35411]: pgmap v12744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:26:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:26:59 smithi114 ceph-mon[30820]: pgmap v12745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:26:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:26:59 smithi116 ceph-mon[35411]: pgmap v12745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:01 smithi114 ceph-mon[30820]: pgmap v12746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:01 smithi116 ceph-mon[35411]: pgmap v12746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:03 smithi114 ceph-mon[30820]: pgmap v12747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:03 smithi116 ceph-mon[35411]: pgmap v12747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:05 smithi114 ceph-mon[30820]: pgmap v12748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:05 smithi116 ceph-mon[35411]: pgmap v12748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:07 smithi114 ceph-mon[30820]: pgmap v12749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:07 smithi116 ceph-mon[35411]: pgmap v12749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:09 smithi114 ceph-mon[30820]: pgmap v12750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:09 smithi116 ceph-mon[35411]: pgmap v12750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:11 smithi114 ceph-mon[30820]: pgmap v12751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:11 smithi116 ceph-mon[35411]: pgmap v12751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:13 smithi114 ceph-mon[30820]: pgmap v12752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:13 smithi116 ceph-mon[35411]: pgmap v12752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:15 smithi114 ceph-mon[30820]: pgmap v12753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:15 smithi116 ceph-mon[35411]: pgmap v12753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:17 smithi114 ceph-mon[30820]: pgmap v12754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:17 smithi116 ceph-mon[35411]: pgmap v12754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:19 smithi114 ceph-mon[30820]: pgmap v12755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:19 smithi116 ceph-mon[35411]: pgmap v12755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:21 smithi114 ceph-mon[30820]: pgmap v12756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:21 smithi116 ceph-mon[35411]: pgmap v12756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:23 smithi114 ceph-mon[30820]: pgmap v12757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:23 smithi116 ceph-mon[35411]: pgmap v12757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:25 smithi114 ceph-mon[30820]: pgmap v12758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:25 smithi116 ceph-mon[35411]: pgmap v12758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:27 smithi114 ceph-mon[30820]: pgmap v12759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:27 smithi116 ceph-mon[35411]: pgmap v12759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:29 smithi114 ceph-mon[30820]: pgmap v12760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:29 smithi116 ceph-mon[35411]: pgmap v12760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:31 smithi114 ceph-mon[30820]: pgmap v12761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:31 smithi116 ceph-mon[35411]: pgmap v12761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:33 smithi114 ceph-mon[30820]: pgmap v12762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:33 smithi116 ceph-mon[35411]: pgmap v12762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:35 smithi114 ceph-mon[30820]: pgmap v12763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:35 smithi116 ceph-mon[35411]: pgmap v12763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:37 smithi116 ceph-mon[35411]: pgmap v12764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:37 smithi114 ceph-mon[30820]: pgmap v12764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:39 smithi116 ceph-mon[35411]: pgmap v12765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:39 smithi114 ceph-mon[30820]: pgmap v12765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:41 smithi116 ceph-mon[35411]: pgmap v12766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:41 smithi114 ceph-mon[30820]: pgmap v12766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:27:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:43 smithi114 ceph-mon[30820]: pgmap v12767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:43 smithi116 ceph-mon[35411]: pgmap v12767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:27:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:45 smithi114 ceph-mon[30820]: pgmap v12768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:45 smithi116 ceph-mon[35411]: pgmap v12768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:27:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:27:47.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:27:46.900+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:27:47.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:27:47.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:27:47.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:27:46 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:27:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:46 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:27:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:47 smithi114 ceph-mon[30820]: pgmap v12769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:27:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:27:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:27:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:47 smithi116 ceph-mon[35411]: pgmap v12769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:27:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:27:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:27:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:27:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:48 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:27:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:27:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:27:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:27:49.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:48 smithi114 ceph-mon[30820]: pgmap v12770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:27:49.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:48 smithi114 ceph-mon[30820]: pgmap v12771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:27:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:48 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:27:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:27:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:27:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:27:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:27:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:27:49.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:27:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:27:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:27:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:27:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:27:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:27:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:48 smithi116 ceph-mon[35411]: pgmap v12770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:27:49.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:48 smithi116 ceph-mon[35411]: pgmap v12771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:27:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:49 smithi114 ceph-mon[30820]: pgmap v12772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T13:27:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:49 smithi116 ceph-mon[35411]: pgmap v12772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T13:27:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:51 smithi114 ceph-mon[30820]: pgmap v12773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:27:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:51 smithi116 ceph-mon[35411]: pgmap v12773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:27:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:53 smithi114 ceph-mon[30820]: pgmap v12774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:27:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:53 smithi116 ceph-mon[35411]: pgmap v12774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:27:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:55 smithi114 ceph-mon[30820]: pgmap v12775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:27:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:55 smithi116 ceph-mon[35411]: pgmap v12775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:27:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:57 smithi114 ceph-mon[30820]: pgmap v12776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:27:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:57 smithi116 ceph-mon[35411]: pgmap v12776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:28:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:27:59 smithi114 ceph-mon[30820]: pgmap v12777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:27:59 smithi116 ceph-mon[35411]: pgmap v12777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:01 smithi114 ceph-mon[30820]: pgmap v12778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:01 smithi116 ceph-mon[35411]: pgmap v12778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:03 smithi114 ceph-mon[30820]: pgmap v12779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:03 smithi116 ceph-mon[35411]: pgmap v12779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:06 smithi114 ceph-mon[30820]: pgmap v12780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:06 smithi116 ceph-mon[35411]: pgmap v12780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:08 smithi114 ceph-mon[30820]: pgmap v12781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:08 smithi116 ceph-mon[35411]: pgmap v12781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:10 smithi114 ceph-mon[30820]: pgmap v12782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:10 smithi116 ceph-mon[35411]: pgmap v12782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:12 smithi114 ceph-mon[30820]: pgmap v12783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:12 smithi116 ceph-mon[35411]: pgmap v12783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:14 smithi114 ceph-mon[30820]: pgmap v12784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:14 smithi116 ceph-mon[35411]: pgmap v12784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:16 smithi114 ceph-mon[30820]: pgmap v12785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:16 smithi116 ceph-mon[35411]: pgmap v12785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:18 smithi114 ceph-mon[30820]: pgmap v12786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:18 smithi116 ceph-mon[35411]: pgmap v12786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:20 smithi114 ceph-mon[30820]: pgmap v12787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:20 smithi116 ceph-mon[35411]: pgmap v12787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:22 smithi114 ceph-mon[30820]: pgmap v12788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:22 smithi116 ceph-mon[35411]: pgmap v12788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:24 smithi114 ceph-mon[30820]: pgmap v12789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:24 smithi116 ceph-mon[35411]: pgmap v12789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:26 smithi114 ceph-mon[30820]: pgmap v12790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:26 smithi116 ceph-mon[35411]: pgmap v12790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:28 smithi114 ceph-mon[30820]: pgmap v12791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:28 smithi116 ceph-mon[35411]: pgmap v12791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:30 smithi114 ceph-mon[30820]: pgmap v12792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:30 smithi116 ceph-mon[35411]: pgmap v12792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:32 smithi114 ceph-mon[30820]: pgmap v12793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:32 smithi116 ceph-mon[35411]: pgmap v12793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:34 smithi114 ceph-mon[30820]: pgmap v12794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:34 smithi116 ceph-mon[35411]: pgmap v12794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:36 smithi114 ceph-mon[30820]: pgmap v12795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:36 smithi116 ceph-mon[35411]: pgmap v12795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:38 smithi114 ceph-mon[30820]: pgmap v12796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:38 smithi116 ceph-mon[35411]: pgmap v12796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:40 smithi114 ceph-mon[30820]: pgmap v12797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:40 smithi116 ceph-mon[35411]: pgmap v12797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:42 smithi114 ceph-mon[30820]: pgmap v12798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:42 smithi116 ceph-mon[35411]: pgmap v12798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:28:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:44 smithi114 ceph-mon[30820]: pgmap v12799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:44 smithi116 ceph-mon[35411]: pgmap v12799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:46 smithi114 ceph-mon[30820]: pgmap v12800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:46 smithi116 ceph-mon[35411]: pgmap v12800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:28:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:28:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:28:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:48 smithi114 ceph-mon[30820]: pgmap v12801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:48 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:28:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:48 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:28:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:48.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:48.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:48 smithi116 ceph-mon[35411]: pgmap v12801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:28:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:48 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:28:48.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:48 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:28:48.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:48.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:48.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:48.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:28:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:49 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:28:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:49 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:28:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:49 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:28:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:49 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:28:50.196+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:28:50.658 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:28:50.659 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:28:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-mon[30820]: pgmap v12802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:28:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:28:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:28:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:50 smithi116 ceph-mon[35411]: pgmap v12802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:28:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:28:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:28:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:28:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:28:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:28:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:28:51.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:51 smithi114 ceph-mon[30820]: pgmap v12803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T13:28:51.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:51 smithi114 ceph-mon[30820]: pgmap v12804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:28:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:28:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:28:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:28:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:28:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:28:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:28:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:28:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:28:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:28:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:28:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:51 smithi116 ceph-mon[35411]: pgmap v12803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T13:28:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:51 smithi116 ceph-mon[35411]: pgmap v12804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:28:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:53 smithi114 ceph-mon[30820]: pgmap v12805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:28:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:53 smithi116 ceph-mon[35411]: pgmap v12805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:28:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:55 smithi114 ceph-mon[30820]: pgmap v12806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:28:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:55 smithi116 ceph-mon[35411]: pgmap v12806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:28:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:57 smithi116 ceph-mon[35411]: pgmap v12807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:28:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:57 smithi114 ceph-mon[30820]: pgmap v12807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:28:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:28:59 smithi116 ceph-mon[35411]: pgmap v12808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:28:59.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:28:59 smithi114 ceph-mon[30820]: pgmap v12808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:29:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:01 smithi116 ceph-mon[35411]: pgmap v12809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:29:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:01 smithi114 ceph-mon[30820]: pgmap v12809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:29:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:03 smithi114 ceph-mon[30820]: pgmap v12810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:03 smithi116 ceph-mon[35411]: pgmap v12810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:05 smithi114 ceph-mon[30820]: pgmap v12811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:05 smithi116 ceph-mon[35411]: pgmap v12811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:07.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:07 smithi116 ceph-mon[35411]: pgmap v12812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:07 smithi114 ceph-mon[30820]: pgmap v12812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:09 smithi114 ceph-mon[30820]: pgmap v12813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:09 smithi116 ceph-mon[35411]: pgmap v12813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:11 smithi114 ceph-mon[30820]: pgmap v12814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:11 smithi116 ceph-mon[35411]: pgmap v12814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:13 smithi114 ceph-mon[30820]: pgmap v12815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:13 smithi116 ceph-mon[35411]: pgmap v12815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:15 smithi114 ceph-mon[30820]: pgmap v12816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:15 smithi116 ceph-mon[35411]: pgmap v12816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:17 smithi114 ceph-mon[30820]: pgmap v12817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:17 smithi116 ceph-mon[35411]: pgmap v12817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:18 smithi114 ceph-mon[30820]: pgmap v12818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:18 smithi116 ceph-mon[35411]: pgmap v12818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:21 smithi114 ceph-mon[30820]: pgmap v12819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:21 smithi116 ceph-mon[35411]: pgmap v12819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:23 smithi114 ceph-mon[30820]: pgmap v12820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:23 smithi116 ceph-mon[35411]: pgmap v12820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:25 smithi114 ceph-mon[30820]: pgmap v12821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:25 smithi116 ceph-mon[35411]: pgmap v12821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:27 smithi114 ceph-mon[30820]: pgmap v12822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:27 smithi116 ceph-mon[35411]: pgmap v12822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:29 smithi114 ceph-mon[30820]: pgmap v12823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:29 smithi116 ceph-mon[35411]: pgmap v12823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:31 smithi114 ceph-mon[30820]: pgmap v12824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:31 smithi116 ceph-mon[35411]: pgmap v12824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:33 smithi114 ceph-mon[30820]: pgmap v12825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:33 smithi116 ceph-mon[35411]: pgmap v12825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:35 smithi114 ceph-mon[30820]: pgmap v12826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:35 smithi116 ceph-mon[35411]: pgmap v12826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:37 smithi114 ceph-mon[30820]: pgmap v12827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:37 smithi116 ceph-mon[35411]: pgmap v12827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:39 smithi114 ceph-mon[30820]: pgmap v12828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:39 smithi116 ceph-mon[35411]: pgmap v12828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:41 smithi114 ceph-mon[30820]: pgmap v12829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:41 smithi116 ceph-mon[35411]: pgmap v12829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:43 smithi114 ceph-mon[30820]: pgmap v12830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:43 smithi116 ceph-mon[35411]: pgmap v12830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:45 smithi114 ceph-mon[30820]: pgmap v12831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:45 smithi116 ceph-mon[35411]: pgmap v12831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:47 smithi114 ceph-mon[30820]: pgmap v12832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:47 smithi116 ceph-mon[35411]: pgmap v12832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:29:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:49 smithi114 ceph-mon[30820]: pgmap v12833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:49 smithi116 ceph-mon[35411]: pgmap v12833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:29:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:51 smithi114 ceph-mon[30820]: pgmap v12834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:29:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:51 smithi116 ceph-mon[35411]: pgmap v12834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:29:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:29:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-mon[30820]: pgmap v12835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:53.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:29:53.556+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:29:53.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:29:53.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:29:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:29:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:53 smithi116 ceph-mon[35411]: pgmap v12835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:29:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:54 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:29:54.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:54 smithi114 ceph-mon[30820]: pgmap v12836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:29:54.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:54 smithi114 ceph-mon[30820]: pgmap v12837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:29:54.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:54 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:29:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:29:54.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:29:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:29:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:54 smithi116 ceph-mon[35411]: pgmap v12836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:29:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:54 smithi116 ceph-mon[35411]: pgmap v12837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:29:54.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:29:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:56 smithi114 ceph-mon[30820]: pgmap v12838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:29:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:56 smithi116 ceph-mon[35411]: pgmap v12838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:29:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:29:58 smithi114 ceph-mon[30820]: pgmap v12839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:29:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:29:58 smithi116 ceph-mon[35411]: pgmap v12839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:30:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:00 smithi114 ceph-mon[30820]: pgmap v12840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:30:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:30:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:00 smithi116 ceph-mon[35411]: pgmap v12840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:30:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:30:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:02 smithi114 ceph-mon[30820]: pgmap v12841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:30:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:02 smithi116 ceph-mon[35411]: pgmap v12841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:30:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:04 smithi114 ceph-mon[30820]: pgmap v12842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:30:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:04 smithi116 ceph-mon[35411]: pgmap v12842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:30:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:06 smithi114 ceph-mon[30820]: pgmap v12843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:06 smithi116 ceph-mon[35411]: pgmap v12843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:08 smithi114 ceph-mon[30820]: pgmap v12844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:30:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:08 smithi116 ceph-mon[35411]: pgmap v12844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:30:10.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:10 smithi114 ceph-mon[30820]: pgmap v12845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:30:10.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:10 smithi116 ceph-mon[35411]: pgmap v12845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:30:12.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:12 smithi114 ceph-mon[30820]: pgmap v12846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:12.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:12 smithi116 ceph-mon[35411]: pgmap v12846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:14 smithi114 ceph-mon[30820]: pgmap v12847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:14 smithi116 ceph-mon[35411]: pgmap v12847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:16 smithi114 ceph-mon[30820]: pgmap v12848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:16 smithi116 ceph-mon[35411]: pgmap v12848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:18 smithi114 ceph-mon[30820]: pgmap v12849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:18 smithi116 ceph-mon[35411]: pgmap v12849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:20.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:20 smithi114 ceph-mon[30820]: pgmap v12850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:30:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:20 smithi116 ceph-mon[35411]: pgmap v12850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:30:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:22 smithi114 ceph-mon[30820]: pgmap v12851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:22 smithi116 ceph-mon[35411]: pgmap v12851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:24 smithi116 ceph-mon[35411]: pgmap v12852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:24 smithi114 ceph-mon[30820]: pgmap v12852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:26 smithi116 ceph-mon[35411]: pgmap v12853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:26 smithi114 ceph-mon[30820]: pgmap v12853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:28 smithi116 ceph-mon[35411]: pgmap v12854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:28 smithi114 ceph-mon[30820]: pgmap v12854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:30.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:30 smithi114 ceph-mon[30820]: pgmap v12855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:30:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:30 smithi116 ceph-mon[35411]: pgmap v12855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:30:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:32 smithi114 ceph-mon[30820]: pgmap v12856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:32 smithi116 ceph-mon[35411]: pgmap v12856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:34 smithi114 ceph-mon[30820]: pgmap v12857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:34 smithi116 ceph-mon[35411]: pgmap v12857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:36 smithi114 ceph-mon[30820]: pgmap v12858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:36 smithi116 ceph-mon[35411]: pgmap v12858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:38.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:38 smithi116 ceph-mon[35411]: pgmap v12859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:38 smithi114 ceph-mon[30820]: pgmap v12859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:40 smithi114 ceph-mon[30820]: pgmap v12860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:30:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:40 smithi116 ceph-mon[35411]: pgmap v12860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:30:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:42 smithi114 ceph-mon[30820]: pgmap v12861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:42 smithi116 ceph-mon[35411]: pgmap v12861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:44 smithi114 ceph-mon[30820]: pgmap v12862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:44 smithi116 ceph-mon[35411]: pgmap v12862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:46 smithi114 ceph-mon[30820]: pgmap v12863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:46 smithi116 ceph-mon[35411]: pgmap v12863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:48 smithi114 ceph-mon[30820]: pgmap v12864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:48 smithi116 ceph-mon[35411]: pgmap v12864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:50 smithi114 ceph-mon[30820]: pgmap v12865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:30:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:50 smithi116 ceph-mon[35411]: pgmap v12865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:30:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:52 smithi114 ceph-mon[30820]: pgmap v12866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:52 smithi116 ceph-mon[35411]: pgmap v12866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:30:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:53 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:30:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:53 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:30:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:54 smithi114 ceph-mon[30820]: pgmap v12867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:54 smithi116 ceph-mon[35411]: pgmap v12867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:30:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:55 smithi114 ceph-mon[30820]: pgmap v12868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:55 smithi116 ceph-mon[35411]: pgmap v12868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:30:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:30:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:30:57.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:30:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:30:56.916+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:30:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:30:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:30:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:30:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:30:57.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:30:57.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:30:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:30:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:30:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:30:57.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:30:56 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:30:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:30:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:30:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:30:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:57 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:30:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:30:58.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:57 smithi114 ceph-mon[30820]: pgmap v12869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:30:58.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:57 smithi114 ceph-mon[30820]: pgmap v12870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:30:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:57 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:30:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:30:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:30:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:30:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:30:58.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:30:58.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:30:58.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:30:58.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:30:58.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:30:58.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:30:58.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:30:58.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:57 smithi116 ceph-mon[35411]: pgmap v12869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:30:58.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:57 smithi116 ceph-mon[35411]: pgmap v12870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:31:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:30:59 smithi114 ceph-mon[30820]: pgmap v12871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:31:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:30:59 smithi116 ceph-mon[35411]: pgmap v12871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:31:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:01 smithi114 ceph-mon[30820]: pgmap v12872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:31:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:01 smithi116 ceph-mon[35411]: pgmap v12872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:31:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:03 smithi114 ceph-mon[30820]: pgmap v12873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:31:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:03 smithi116 ceph-mon[35411]: pgmap v12873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:31:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:05 smithi114 ceph-mon[30820]: pgmap v12874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:31:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:05 smithi116 ceph-mon[35411]: pgmap v12874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:31:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:07 smithi114 ceph-mon[30820]: pgmap v12875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:31:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:07 smithi116 ceph-mon[35411]: pgmap v12875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:31:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:10 smithi114 ceph-mon[30820]: pgmap v12876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:10 smithi116 ceph-mon[35411]: pgmap v12876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:12 smithi114 ceph-mon[30820]: pgmap v12877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:12 smithi116 ceph-mon[35411]: pgmap v12877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:14 smithi114 ceph-mon[30820]: pgmap v12878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:14 smithi116 ceph-mon[35411]: pgmap v12878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:16 smithi114 ceph-mon[30820]: pgmap v12879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:16 smithi116 ceph-mon[35411]: pgmap v12879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:18 smithi114 ceph-mon[30820]: pgmap v12880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:18 smithi116 ceph-mon[35411]: pgmap v12880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:20 smithi114 ceph-mon[30820]: pgmap v12881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:20 smithi116 ceph-mon[35411]: pgmap v12881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:22 smithi114 ceph-mon[30820]: pgmap v12882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:22 smithi116 ceph-mon[35411]: pgmap v12882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:24 smithi114 ceph-mon[30820]: pgmap v12883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:24 smithi116 ceph-mon[35411]: pgmap v12883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:26 smithi114 ceph-mon[30820]: pgmap v12884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:26 smithi116 ceph-mon[35411]: pgmap v12884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:28 smithi114 ceph-mon[30820]: pgmap v12885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:28 smithi116 ceph-mon[35411]: pgmap v12885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:30 smithi114 ceph-mon[30820]: pgmap v12886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:30 smithi116 ceph-mon[35411]: pgmap v12886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:32 smithi114 ceph-mon[30820]: pgmap v12887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:32 smithi116 ceph-mon[35411]: pgmap v12887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:34 smithi114 ceph-mon[30820]: pgmap v12888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:34 smithi116 ceph-mon[35411]: pgmap v12888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:36 smithi114 ceph-mon[30820]: pgmap v12889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:36 smithi116 ceph-mon[35411]: pgmap v12889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:38 smithi114 ceph-mon[30820]: pgmap v12890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:38 smithi116 ceph-mon[35411]: pgmap v12890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:40 smithi114 ceph-mon[30820]: pgmap v12891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:40 smithi116 ceph-mon[35411]: pgmap v12891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:42 smithi114 ceph-mon[30820]: pgmap v12892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:42 smithi116 ceph-mon[35411]: pgmap v12892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:44 smithi114 ceph-mon[30820]: pgmap v12893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:44 smithi116 ceph-mon[35411]: pgmap v12893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:46 smithi114 ceph-mon[30820]: pgmap v12894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:46 smithi116 ceph-mon[35411]: pgmap v12894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:48 smithi114 ceph-mon[30820]: pgmap v12895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:48 smithi116 ceph-mon[35411]: pgmap v12895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:50 smithi114 ceph-mon[30820]: pgmap v12896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:50 smithi116 ceph-mon[35411]: pgmap v12896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:52 smithi114 ceph-mon[30820]: pgmap v12897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:52 smithi116 ceph-mon[35411]: pgmap v12897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:31:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:54 smithi114 ceph-mon[30820]: pgmap v12898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:54 smithi116 ceph-mon[35411]: pgmap v12898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:31:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:56 smithi114 ceph-mon[30820]: pgmap v12899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:56 smithi116 ceph-mon[35411]: pgmap v12899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:31:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:31:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:31:58.221 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:58 smithi116 ceph-mon[35411]: pgmap v12900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:31:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:58 smithi114 ceph-mon[30820]: pgmap v12900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:59 smithi114 ceph-mon[30820]: pgmap v12901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:31:59 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:59 smithi116 ceph-mon[35411]: pgmap v12901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:00.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:31:59 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:00.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:32:00.277+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:32:00.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:32:00.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:32:00.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:32:00.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:32:00.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:32:00.672 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:32:00.673 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:32:00.673 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:32:00.673 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:32:00.673 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:32:00.673 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:32:00.673 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:32:01.108 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:32:01.109 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:32:01.110 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:32:01.111 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:32:01.111 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-mon[30820]: pgmap v12902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T13:32:01.111 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-mon[30820]: pgmap v12903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:32:01.111 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:00 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:32:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:32:01.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:00 smithi116 ceph-mon[35411]: pgmap v12902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:00 smithi116 ceph-mon[35411]: pgmap v12903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:32:01.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:32:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:03 smithi114 ceph-mon[30820]: pgmap v12904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:03 smithi116 ceph-mon[35411]: pgmap v12904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:05 smithi114 ceph-mon[30820]: pgmap v12905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:05 smithi116 ceph-mon[35411]: pgmap v12905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:07 smithi114 ceph-mon[30820]: pgmap v12906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:07 smithi116 ceph-mon[35411]: pgmap v12906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:09 smithi114 ceph-mon[30820]: pgmap v12907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:09 smithi116 ceph-mon[35411]: pgmap v12907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:11 smithi114 ceph-mon[30820]: pgmap v12908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:32:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:11 smithi116 ceph-mon[35411]: pgmap v12908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:32:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:13 smithi114 ceph-mon[30820]: pgmap v12909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:32:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:13 smithi116 ceph-mon[35411]: pgmap v12909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:32:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:15 smithi114 ceph-mon[30820]: pgmap v12910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:32:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:15 smithi116 ceph-mon[35411]: pgmap v12910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:32:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:17 smithi114 ceph-mon[30820]: pgmap v12911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:17 smithi116 ceph-mon[35411]: pgmap v12911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:19 smithi114 ceph-mon[30820]: pgmap v12912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:32:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:19 smithi116 ceph-mon[35411]: pgmap v12912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:32:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:21 smithi114 ceph-mon[30820]: pgmap v12913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:21 smithi116 ceph-mon[35411]: pgmap v12913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:32:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:23 smithi114 ceph-mon[30820]: pgmap v12914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T13:32:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:23 smithi116 ceph-mon[35411]: pgmap v12914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 5 op/s 2024-09-19T13:32:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:25 smithi114 ceph-mon[30820]: pgmap v12915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:32:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:25 smithi116 ceph-mon[35411]: pgmap v12915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:32:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:27 smithi114 ceph-mon[30820]: pgmap v12916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:27 smithi116 ceph-mon[35411]: pgmap v12916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:29 smithi114 ceph-mon[30820]: pgmap v12917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:29 smithi116 ceph-mon[35411]: pgmap v12917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:31 smithi114 ceph-mon[30820]: pgmap v12918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:32:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:31 smithi116 ceph-mon[35411]: pgmap v12918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:32:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:33 smithi114 ceph-mon[30820]: pgmap v12919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:32:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:33 smithi116 ceph-mon[35411]: pgmap v12919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:32:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:35 smithi114 ceph-mon[30820]: pgmap v12920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:32:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:35 smithi116 ceph-mon[35411]: pgmap v12920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:32:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:37 smithi114 ceph-mon[30820]: pgmap v12921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:37 smithi116 ceph-mon[35411]: pgmap v12921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:39 smithi114 ceph-mon[30820]: pgmap v12922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:39 smithi116 ceph-mon[35411]: pgmap v12922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:41 smithi114 ceph-mon[30820]: pgmap v12923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:32:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:41 smithi116 ceph-mon[35411]: pgmap v12923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:32:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:43 smithi114 ceph-mon[30820]: pgmap v12924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:32:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:43 smithi116 ceph-mon[35411]: pgmap v12924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:32:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:45 smithi116 ceph-mon[35411]: pgmap v12925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:32:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:45 smithi114 ceph-mon[30820]: pgmap v12925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:32:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:47 smithi116 ceph-mon[35411]: pgmap v12926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:47 smithi114 ceph-mon[30820]: pgmap v12926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:49 smithi116 ceph-mon[35411]: pgmap v12927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:49 smithi114 ceph-mon[30820]: pgmap v12927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:51 smithi114 ceph-mon[30820]: pgmap v12928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:32:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:51 smithi116 ceph-mon[35411]: pgmap v12928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:32:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:53 smithi114 ceph-mon[30820]: pgmap v12929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:32:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:53 smithi116 ceph-mon[35411]: pgmap v12929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:32:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:55 smithi114 ceph-mon[30820]: pgmap v12930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:32:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:55 smithi116 ceph-mon[35411]: pgmap v12930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:32:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:57 smithi114 ceph-mon[30820]: pgmap v12931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:57 smithi116 ceph-mon[35411]: pgmap v12931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:32:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:32:59 smithi114 ceph-mon[30820]: pgmap v12932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:32:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:32:59 smithi116 ceph-mon[35411]: pgmap v12932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:33:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:33:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:01 smithi114 ceph-mon[30820]: pgmap v12933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:33:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:01 smithi116 ceph-mon[35411]: pgmap v12933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:33:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-mon[30820]: pgmap v12934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:33:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:33:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:33:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:33:03.572+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:33:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:33:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:33:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:33:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:33:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:33:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:33:03.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:33:03.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:33:03.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:33:03 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:33:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:03 smithi116 ceph-mon[35411]: pgmap v12934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:33:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:33:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:33:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:33:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:04 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:33:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:33:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:33:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:33:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:33:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:33:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:33:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:33:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:33:05.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:33:05.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:04 smithi114 ceph-mon[30820]: pgmap v12935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:33:05.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:04 smithi114 ceph-mon[30820]: pgmap v12936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:33:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:04 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:33:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:33:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:33:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:33:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:33:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:33:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:33:05.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:33:05.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:33:05.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:33:05.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:33:05.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:33:05.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:04 smithi116 ceph-mon[35411]: pgmap v12935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:33:05.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:04 smithi116 ceph-mon[35411]: pgmap v12936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:33:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:06 smithi114 ceph-mon[30820]: pgmap v12937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:33:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:06 smithi116 ceph-mon[35411]: pgmap v12937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:33:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:08 smithi114 ceph-mon[30820]: pgmap v12938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:33:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:08 smithi116 ceph-mon[35411]: pgmap v12938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:33:10.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:10 smithi116 ceph-mon[35411]: pgmap v12939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T13:33:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:10 smithi114 ceph-mon[30820]: pgmap v12939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 5 op/s 2024-09-19T13:33:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:12 smithi114 ceph-mon[30820]: pgmap v12940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:33:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:12 smithi116 ceph-mon[35411]: pgmap v12940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:33:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:14 smithi114 ceph-mon[30820]: pgmap v12941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:33:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:14 smithi116 ceph-mon[35411]: pgmap v12941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:33:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:16 smithi114 ceph-mon[30820]: pgmap v12942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:16 smithi116 ceph-mon[35411]: pgmap v12942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:18 smithi114 ceph-mon[30820]: pgmap v12943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:18 smithi116 ceph-mon[35411]: pgmap v12943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:20 smithi114 ceph-mon[30820]: pgmap v12944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:33:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:20 smithi116 ceph-mon[35411]: pgmap v12944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:33:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:22 smithi114 ceph-mon[30820]: pgmap v12945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:33:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:22 smithi116 ceph-mon[35411]: pgmap v12945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:33:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:23 smithi114 ceph-mon[30820]: pgmap v12946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:23 smithi116 ceph-mon[35411]: pgmap v12946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:26 smithi114 ceph-mon[30820]: pgmap v12947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:26 smithi116 ceph-mon[35411]: pgmap v12947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:28 smithi114 ceph-mon[30820]: pgmap v12948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:28 smithi116 ceph-mon[35411]: pgmap v12948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:30 smithi114 ceph-mon[30820]: pgmap v12949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:33:30.959 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:30 smithi116 ceph-mon[35411]: pgmap v12949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:33:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:32 smithi114 ceph-mon[30820]: pgmap v12950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:33:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:32 smithi116 ceph-mon[35411]: pgmap v12950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:33:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:34 smithi114 ceph-mon[30820]: pgmap v12951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:34 smithi116 ceph-mon[35411]: pgmap v12951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:36 smithi114 ceph-mon[30820]: pgmap v12952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:36 smithi116 ceph-mon[35411]: pgmap v12952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:38 smithi114 ceph-mon[30820]: pgmap v12953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:38 smithi116 ceph-mon[35411]: pgmap v12953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:40 smithi116 ceph-mon[35411]: pgmap v12954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:33:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:40 smithi114 ceph-mon[30820]: pgmap v12954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:33:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:42 smithi116 ceph-mon[35411]: pgmap v12955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:33:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:42 smithi114 ceph-mon[30820]: pgmap v12955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:33:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:44 smithi114 ceph-mon[30820]: pgmap v12956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:44 smithi116 ceph-mon[35411]: pgmap v12956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:46 smithi114 ceph-mon[30820]: pgmap v12957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:46 smithi116 ceph-mon[35411]: pgmap v12957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:48 smithi114 ceph-mon[30820]: pgmap v12958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:48 smithi116 ceph-mon[35411]: pgmap v12958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:50 smithi114 ceph-mon[30820]: pgmap v12959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:33:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:50 smithi116 ceph-mon[35411]: pgmap v12959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:33:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:52 smithi114 ceph-mon[30820]: pgmap v12960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:33:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:52 smithi116 ceph-mon[35411]: pgmap v12960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:33:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:54 smithi114 ceph-mon[30820]: pgmap v12961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:54 smithi116 ceph-mon[35411]: pgmap v12961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:56 smithi114 ceph-mon[30820]: pgmap v12962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:56 smithi116 ceph-mon[35411]: pgmap v12962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:33:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:33:58 smithi114 ceph-mon[30820]: pgmap v12963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:33:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:33:58 smithi116 ceph-mon[35411]: pgmap v12963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:00 smithi114 ceph-mon[30820]: pgmap v12964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:00 smithi116 ceph-mon[35411]: pgmap v12964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:02.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:02 smithi114 ceph-mon[30820]: pgmap v12965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:02 smithi116 ceph-mon[35411]: pgmap v12965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:03 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:34:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:03 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:34:05.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:04 smithi114 ceph-mon[30820]: pgmap v12966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:04 smithi116 ceph-mon[35411]: pgmap v12966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-mon[30820]: pgmap v12967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:07.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:34:06.932+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:34:07.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:34:07.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:34:06 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:34:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:06 smithi116 ceph-mon[35411]: pgmap v12967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:34:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:34:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:34:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:34:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:34:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:34:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:08 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:34:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:34:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:34:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:34:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:34:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:34:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:34:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:34:09.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:34:09.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:34:09.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:34:09.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:34:09.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:34:09.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:34:09.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:34:09.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:34:09.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:34:09.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:34:09.171 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:34:09.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:34:09.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:34:09.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:34:09.172 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:34:09.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:08 smithi114 ceph-mon[30820]: pgmap v12968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:34:09.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:08 smithi114 ceph-mon[30820]: pgmap v12969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:34:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:08 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:34:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:34:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:34:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:34:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:34:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:34:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:34:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:34:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:34:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:34:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:34:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:34:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:34:09.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:34:09.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:34:09.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:34:09.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:34:09.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:34:09.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:34:09.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:34:09.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:34:09.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:34:09.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:34:09.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:08 smithi116 ceph-mon[35411]: pgmap v12968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:34:09.184 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:08 smithi116 ceph-mon[35411]: pgmap v12969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:34:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:10 smithi114 ceph-mon[30820]: pgmap v12970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:34:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:10 smithi116 ceph-mon[35411]: pgmap v12970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:34:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:12 smithi114 ceph-mon[30820]: pgmap v12971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:34:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:12 smithi116 ceph-mon[35411]: pgmap v12971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:34:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:14 smithi114 ceph-mon[30820]: pgmap v12972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:34:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:14 smithi116 ceph-mon[35411]: pgmap v12972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:34:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:16 smithi114 ceph-mon[30820]: pgmap v12973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:34:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:16 smithi116 ceph-mon[35411]: pgmap v12973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:34:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:18 smithi114 ceph-mon[30820]: pgmap v12974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:34:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:18 smithi116 ceph-mon[35411]: pgmap v12974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:34:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:20 smithi114 ceph-mon[30820]: pgmap v12975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:20 smithi116 ceph-mon[35411]: pgmap v12975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:22 smithi114 ceph-mon[30820]: pgmap v12976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:22 smithi116 ceph-mon[35411]: pgmap v12976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:24 smithi114 ceph-mon[30820]: pgmap v12977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:24 smithi116 ceph-mon[35411]: pgmap v12977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:25 smithi114 ceph-mon[30820]: pgmap v12978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:25 smithi116 ceph-mon[35411]: pgmap v12978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:28 smithi114 ceph-mon[30820]: pgmap v12979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:28 smithi116 ceph-mon[35411]: pgmap v12979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:30 smithi114 ceph-mon[30820]: pgmap v12980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:30 smithi116 ceph-mon[35411]: pgmap v12980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:32 smithi114 ceph-mon[30820]: pgmap v12981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:32 smithi116 ceph-mon[35411]: pgmap v12981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:34 smithi114 ceph-mon[30820]: pgmap v12982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:34 smithi116 ceph-mon[35411]: pgmap v12982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:36 smithi114 ceph-mon[30820]: pgmap v12983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:36 smithi116 ceph-mon[35411]: pgmap v12983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:38 smithi114 ceph-mon[30820]: pgmap v12984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:38 smithi116 ceph-mon[35411]: pgmap v12984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:40 smithi114 ceph-mon[30820]: pgmap v12985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:40 smithi116 ceph-mon[35411]: pgmap v12985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:42 smithi114 ceph-mon[30820]: pgmap v12986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:42 smithi116 ceph-mon[35411]: pgmap v12986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:44 smithi114 ceph-mon[30820]: pgmap v12987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:44 smithi116 ceph-mon[35411]: pgmap v12987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:46 smithi114 ceph-mon[30820]: pgmap v12988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:46 smithi116 ceph-mon[35411]: pgmap v12988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:48 smithi114 ceph-mon[30820]: pgmap v12989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:48 smithi116 ceph-mon[35411]: pgmap v12989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:50 smithi114 ceph-mon[30820]: pgmap v12990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:50 smithi116 ceph-mon[35411]: pgmap v12990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:52 smithi114 ceph-mon[30820]: pgmap v12991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:52 smithi116 ceph-mon[35411]: pgmap v12991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:34:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:54 smithi114 ceph-mon[30820]: pgmap v12992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:54 smithi116 ceph-mon[35411]: pgmap v12992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:34:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:56 smithi114 ceph-mon[30820]: pgmap v12993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:56 smithi116 ceph-mon[35411]: pgmap v12993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:34:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:34:58 smithi114 ceph-mon[30820]: pgmap v12994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:34:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:34:58 smithi116 ceph-mon[35411]: pgmap v12994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:00 smithi114 ceph-mon[30820]: pgmap v12995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:00 smithi116 ceph-mon[35411]: pgmap v12995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:02 smithi114 ceph-mon[30820]: pgmap v12996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:35:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:02 smithi116 ceph-mon[35411]: pgmap v12996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:35:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:04 smithi114 ceph-mon[30820]: pgmap v12997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:04 smithi116 ceph-mon[35411]: pgmap v12997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:06 smithi114 ceph-mon[30820]: pgmap v12998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:06 smithi116 ceph-mon[35411]: pgmap v12998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:35:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:35:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:08 smithi114 ceph-mon[30820]: pgmap v12999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:08 smithi116 ceph-mon[35411]: pgmap v12999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-mon[30820]: pgmap v13000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:35:10.293+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:35:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:35:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:35:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:35:10.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:35:10.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:35:10.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:35:10.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:35:10.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:35:10.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:35:10.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:35:10.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:35:10.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:35:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:35:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:10 smithi116 ceph-mon[35411]: pgmap v13000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:11 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:35:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:35:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:35:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:11 smithi114 ceph-mon[30820]: pgmap v13001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:11 smithi114 ceph-mon[30820]: pgmap v13002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:35:11.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:35:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:11 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:35:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:35:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:35:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:35:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:35:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:35:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:35:11.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:35:11.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:35:11.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:35:11.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:35:11.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:11 smithi116 ceph-mon[35411]: pgmap v13001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T13:35:11.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:11 smithi116 ceph-mon[35411]: pgmap v13002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:35:11.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:35:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:13 smithi114 ceph-mon[30820]: pgmap v13003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:35:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:13 smithi116 ceph-mon[35411]: pgmap v13003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:35:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:15 smithi114 ceph-mon[30820]: pgmap v13004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:35:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:15 smithi116 ceph-mon[35411]: pgmap v13004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:35:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:17 smithi114 ceph-mon[30820]: pgmap v13005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:35:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:17 smithi116 ceph-mon[35411]: pgmap v13005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:35:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:19 smithi114 ceph-mon[30820]: pgmap v13006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:35:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:19 smithi116 ceph-mon[35411]: pgmap v13006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:35:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:21 smithi114 ceph-mon[30820]: pgmap v13007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:35:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:21 smithi116 ceph-mon[35411]: pgmap v13007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:35:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:23 smithi114 ceph-mon[30820]: pgmap v13008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:35:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:23 smithi116 ceph-mon[35411]: pgmap v13008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:35:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:25 smithi114 ceph-mon[30820]: pgmap v13009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:25 smithi116 ceph-mon[35411]: pgmap v13009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:27 smithi114 ceph-mon[30820]: pgmap v13010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:27 smithi116 ceph-mon[35411]: pgmap v13010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:29 smithi114 ceph-mon[30820]: pgmap v13011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:29 smithi116 ceph-mon[35411]: pgmap v13011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:31 smithi114 ceph-mon[30820]: pgmap v13012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:35:31.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:31 smithi116 ceph-mon[35411]: pgmap v13012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:35:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:33 smithi114 ceph-mon[30820]: pgmap v13013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:35:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:33 smithi116 ceph-mon[35411]: pgmap v13013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:35:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:35 smithi114 ceph-mon[30820]: pgmap v13014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:35 smithi116 ceph-mon[35411]: pgmap v13014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:37 smithi114 ceph-mon[30820]: pgmap v13015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:37 smithi116 ceph-mon[35411]: pgmap v13015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:39 smithi114 ceph-mon[30820]: pgmap v13016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:39 smithi116 ceph-mon[35411]: pgmap v13016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:41 smithi114 ceph-mon[30820]: pgmap v13017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:35:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:41 smithi116 ceph-mon[35411]: pgmap v13017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:35:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:43 smithi114 ceph-mon[30820]: pgmap v13018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:35:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:43 smithi116 ceph-mon[35411]: pgmap v13018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:35:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:45 smithi116 ceph-mon[35411]: pgmap v13019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:45 smithi114 ceph-mon[30820]: pgmap v13019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:47 smithi116 ceph-mon[35411]: pgmap v13020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:47 smithi114 ceph-mon[30820]: pgmap v13020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:49 smithi114 ceph-mon[30820]: pgmap v13021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:49 smithi116 ceph-mon[35411]: pgmap v13021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:35:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:51 smithi114 ceph-mon[30820]: pgmap v13022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:35:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:51 smithi116 ceph-mon[35411]: pgmap v13022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:35:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:53 smithi114 ceph-mon[30820]: pgmap v13023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:35:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:53 smithi116 ceph-mon[35411]: pgmap v13023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:35:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:55 smithi114 ceph-mon[30820]: pgmap v13024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:55 smithi116 ceph-mon[35411]: pgmap v13024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:35:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:57 smithi114 ceph-mon[30820]: pgmap v13025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:35:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:57 smithi116 ceph-mon[35411]: pgmap v13025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:00.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:35:59 smithi114 ceph-mon[30820]: pgmap v13026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:35:59 smithi116 ceph-mon[35411]: pgmap v13026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:01 smithi114 ceph-mon[30820]: pgmap v13027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:36:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:01 smithi116 ceph-mon[35411]: pgmap v13027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:36:03.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:03 smithi114 ceph-mon[30820]: pgmap v13028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:36:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:03 smithi116 ceph-mon[35411]: pgmap v13028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:36:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:05 smithi114 ceph-mon[30820]: pgmap v13029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:36:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:05 smithi116 ceph-mon[35411]: pgmap v13029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:36:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:07 smithi114 ceph-mon[30820]: pgmap v13030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:07 smithi116 ceph-mon[35411]: pgmap v13030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:09 smithi114 ceph-mon[30820]: pgmap v13031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:09 smithi116 ceph-mon[35411]: pgmap v13031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:11.080 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:36:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:36:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:11 smithi114 ceph-mon[30820]: pgmap v13032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:36:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:11 smithi116 ceph-mon[35411]: pgmap v13032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:36:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-mon[30820]: pgmap v13033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:36:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:36:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:36:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:36:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:36:13.652+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:36:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:36:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:36:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:36:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:36:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:36:13.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:36:13.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:36:13.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:36:13.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:36:13.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:36:13 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:36:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:13 smithi116 ceph-mon[35411]: pgmap v13033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:36:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:36:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:36:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:36:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:14 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:36:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:36:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:36:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:36:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:36:15.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:14 smithi114 ceph-mon[30820]: pgmap v13034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:36:15.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:14 smithi114 ceph-mon[30820]: pgmap v13035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:14 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:36:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:36:15.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:14 smithi116 ceph-mon[35411]: pgmap v13034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:36:15.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:14 smithi116 ceph-mon[35411]: pgmap v13035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:36:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:16 smithi114 ceph-mon[30820]: pgmap v13036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:36:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:16 smithi116 ceph-mon[35411]: pgmap v13036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:36:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:18 smithi114 ceph-mon[30820]: pgmap v13037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:36:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:18 smithi116 ceph-mon[35411]: pgmap v13037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:36:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:20 smithi114 ceph-mon[30820]: pgmap v13038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:36:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:20 smithi116 ceph-mon[35411]: pgmap v13038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:36:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:22 smithi114 ceph-mon[30820]: pgmap v13039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:36:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:22 smithi116 ceph-mon[35411]: pgmap v13039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:36:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:24 smithi114 ceph-mon[30820]: pgmap v13040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:36:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:24 smithi116 ceph-mon[35411]: pgmap v13040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:36:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:26 smithi114 ceph-mon[30820]: pgmap v13041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:36:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:26 smithi116 ceph-mon[35411]: pgmap v13041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:36:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:28 smithi114 ceph-mon[30820]: pgmap v13042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:28 smithi116 ceph-mon[35411]: pgmap v13042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:30 smithi114 ceph-mon[30820]: pgmap v13043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:36:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:30 smithi116 ceph-mon[35411]: pgmap v13043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:36:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:32 smithi114 ceph-mon[30820]: pgmap v13044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:32 smithi116 ceph-mon[35411]: pgmap v13044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:34 smithi114 ceph-mon[30820]: pgmap v13045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:34 smithi116 ceph-mon[35411]: pgmap v13045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:36 smithi114 ceph-mon[30820]: pgmap v13046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:36:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:36 smithi116 ceph-mon[35411]: pgmap v13046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:36:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:38 smithi114 ceph-mon[30820]: pgmap v13047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:38 smithi116 ceph-mon[35411]: pgmap v13047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:40 smithi114 ceph-mon[30820]: pgmap v13048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:36:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:40 smithi116 ceph-mon[35411]: pgmap v13048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:36:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:41 smithi114 ceph-mon[30820]: pgmap v13049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:41 smithi116 ceph-mon[35411]: pgmap v13049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:44 smithi114 ceph-mon[30820]: pgmap v13050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:44 smithi116 ceph-mon[35411]: pgmap v13050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:46 smithi114 ceph-mon[30820]: pgmap v13051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:36:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:46 smithi116 ceph-mon[35411]: pgmap v13051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:36:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:48 smithi114 ceph-mon[30820]: pgmap v13052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:48 smithi116 ceph-mon[35411]: pgmap v13052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:51.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:50 smithi114 ceph-mon[30820]: pgmap v13053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:36:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:50 smithi116 ceph-mon[35411]: pgmap v13053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:36:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:52 smithi114 ceph-mon[30820]: pgmap v13054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:52 smithi116 ceph-mon[35411]: pgmap v13054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:36:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:54 smithi114 ceph-mon[30820]: pgmap v13055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:54 smithi116 ceph-mon[35411]: pgmap v13055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:56 smithi114 ceph-mon[30820]: pgmap v13056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:36:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:56 smithi116 ceph-mon[35411]: pgmap v13056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:36:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:36:58 smithi114 ceph-mon[30820]: pgmap v13057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:36:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:36:58 smithi116 ceph-mon[35411]: pgmap v13057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:00 smithi114 ceph-mon[30820]: pgmap v13058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:00 smithi116 ceph-mon[35411]: pgmap v13058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:02 smithi114 ceph-mon[30820]: pgmap v13059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:02 smithi116 ceph-mon[35411]: pgmap v13059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:04 smithi114 ceph-mon[30820]: pgmap v13060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:04 smithi116 ceph-mon[35411]: pgmap v13060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:06 smithi114 ceph-mon[30820]: pgmap v13061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:06 smithi116 ceph-mon[35411]: pgmap v13061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:08 smithi114 ceph-mon[30820]: pgmap v13062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:08 smithi116 ceph-mon[35411]: pgmap v13062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:10 smithi114 ceph-mon[30820]: pgmap v13063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:10 smithi116 ceph-mon[35411]: pgmap v13063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:12 smithi114 ceph-mon[30820]: pgmap v13064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:12 smithi116 ceph-mon[35411]: pgmap v13064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:14 smithi114 ceph-mon[30820]: pgmap v13065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:37:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:14 smithi116 ceph-mon[35411]: pgmap v13065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:37:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:16 smithi114 ceph-mon[30820]: pgmap v13066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:37:17.013+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:37:17.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:37:17.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:37:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:16 smithi116 ceph-mon[35411]: pgmap v13066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:37:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:37:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:37:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:37:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:37:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:37:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:18 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:37:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:37:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:37:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:18 smithi114 ceph-mon[30820]: pgmap v13067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:37:19.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:18 smithi114 ceph-mon[30820]: pgmap v13068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:37:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:18 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:37:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:37:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:37:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:37:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:37:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:37:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:37:19.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:37:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:18 smithi116 ceph-mon[35411]: pgmap v13067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:37:19.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:18 smithi116 ceph-mon[35411]: pgmap v13068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:37:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:20 smithi114 ceph-mon[30820]: pgmap v13069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:37:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:20 smithi116 ceph-mon[35411]: pgmap v13069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:37:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:21 smithi114 ceph-mon[30820]: pgmap v13070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:37:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:21 smithi116 ceph-mon[35411]: pgmap v13070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:37:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:24 smithi114 ceph-mon[30820]: pgmap v13071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:37:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:24 smithi116 ceph-mon[35411]: pgmap v13071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:37:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:26 smithi114 ceph-mon[30820]: pgmap v13072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:37:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:26 smithi116 ceph-mon[35411]: pgmap v13072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:37:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:28 smithi114 ceph-mon[30820]: pgmap v13073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:37:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:28 smithi116 ceph-mon[35411]: pgmap v13073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:37:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:30 smithi114 ceph-mon[30820]: pgmap v13074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:30 smithi116 ceph-mon[35411]: pgmap v13074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:32 smithi114 ceph-mon[30820]: pgmap v13075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:32 smithi116 ceph-mon[35411]: pgmap v13075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:34 smithi114 ceph-mon[30820]: pgmap v13076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:34 smithi116 ceph-mon[35411]: pgmap v13076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:36 smithi114 ceph-mon[30820]: pgmap v13077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:36 smithi116 ceph-mon[35411]: pgmap v13077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:38 smithi114 ceph-mon[30820]: pgmap v13078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:38 smithi116 ceph-mon[35411]: pgmap v13078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:40 smithi114 ceph-mon[30820]: pgmap v13079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:40 smithi116 ceph-mon[35411]: pgmap v13079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:42 smithi114 ceph-mon[30820]: pgmap v13080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:42 smithi116 ceph-mon[35411]: pgmap v13080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:44 smithi114 ceph-mon[30820]: pgmap v13081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:44 smithi116 ceph-mon[35411]: pgmap v13081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:46 smithi114 ceph-mon[30820]: pgmap v13082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:46 smithi116 ceph-mon[35411]: pgmap v13082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:48 smithi114 ceph-mon[30820]: pgmap v13083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:48 smithi116 ceph-mon[35411]: pgmap v13083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:50 smithi114 ceph-mon[30820]: pgmap v13084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:50 smithi116 ceph-mon[35411]: pgmap v13084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:52 smithi114 ceph-mon[30820]: pgmap v13085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:52 smithi116 ceph-mon[35411]: pgmap v13085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:37:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:54 smithi114 ceph-mon[30820]: pgmap v13086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:54 smithi116 ceph-mon[35411]: pgmap v13086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:37:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:56 smithi114 ceph-mon[30820]: pgmap v13087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:56 smithi116 ceph-mon[35411]: pgmap v13087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:37:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:37:58 smithi114 ceph-mon[30820]: pgmap v13088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:37:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:37:58 smithi116 ceph-mon[35411]: pgmap v13088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:00 smithi114 ceph-mon[30820]: pgmap v13089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:00 smithi116 ceph-mon[35411]: pgmap v13089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:02 smithi114 ceph-mon[30820]: pgmap v13090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:38:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:02 smithi116 ceph-mon[35411]: pgmap v13090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:38:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:04 smithi114 ceph-mon[30820]: pgmap v13091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:04 smithi116 ceph-mon[35411]: pgmap v13091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:06 smithi114 ceph-mon[30820]: pgmap v13092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:06 smithi116 ceph-mon[35411]: pgmap v13092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:08 smithi114 ceph-mon[30820]: pgmap v13093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:08 smithi116 ceph-mon[35411]: pgmap v13093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:10 smithi114 ceph-mon[30820]: pgmap v13094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:10 smithi116 ceph-mon[35411]: pgmap v13094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:12 smithi114 ceph-mon[30820]: pgmap v13095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:38:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:12 smithi116 ceph-mon[35411]: pgmap v13095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:38:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:14 smithi114 ceph-mon[30820]: pgmap v13096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:14 smithi116 ceph-mon[35411]: pgmap v13096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:16 smithi116 ceph-mon[35411]: pgmap v13097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:16 smithi114 ceph-mon[30820]: pgmap v13097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:38:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:38:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:18 smithi116 ceph-mon[35411]: pgmap v13098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:18 smithi114 ceph-mon[30820]: pgmap v13098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-mon[30820]: pgmap v13099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:38:20.372+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:38:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:38:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:38:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:38:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:38:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:38:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:38:20.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:38:20.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:38:20.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:38:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:38:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:20 smithi116 ceph-mon[35411]: pgmap v13099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:38:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:38:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:38:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:38:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:21 smithi114 ceph-mon[30820]: pgmap v13100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:21 smithi114 ceph-mon[30820]: pgmap v13101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:38:21.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:38:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:38:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:38:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:38:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:21 smithi116 ceph-mon[35411]: pgmap v13100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 7 op/s 2024-09-19T13:38:21.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:21 smithi116 ceph-mon[35411]: pgmap v13101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:38:21.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:38:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:23 smithi114 ceph-mon[30820]: pgmap v13102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:38:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:23 smithi116 ceph-mon[35411]: pgmap v13102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:38:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:25 smithi114 ceph-mon[30820]: pgmap v13103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:38:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:25 smithi116 ceph-mon[35411]: pgmap v13103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:38:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:27 smithi114 ceph-mon[30820]: pgmap v13104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:38:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:27 smithi116 ceph-mon[35411]: pgmap v13104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:38:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:29 smithi114 ceph-mon[30820]: pgmap v13105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:38:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:29 smithi116 ceph-mon[35411]: pgmap v13105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:38:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:31 smithi114 ceph-mon[30820]: pgmap v13106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:38:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:31 smithi116 ceph-mon[35411]: pgmap v13106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:38:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:33 smithi114 ceph-mon[30820]: pgmap v13107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:38:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:33 smithi116 ceph-mon[35411]: pgmap v13107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:38:35.719 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:35 smithi114 ceph-mon[30820]: pgmap v13108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:35 smithi116 ceph-mon[35411]: pgmap v13108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:37 smithi114 ceph-mon[30820]: pgmap v13109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:37 smithi116 ceph-mon[35411]: pgmap v13109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:39 smithi114 ceph-mon[30820]: pgmap v13110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:39 smithi116 ceph-mon[35411]: pgmap v13110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:41 smithi114 ceph-mon[30820]: pgmap v13111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:38:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:41 smithi116 ceph-mon[35411]: pgmap v13111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:38:43.731 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:43 smithi116 ceph-mon[35411]: pgmap v13112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:38:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:43 smithi114 ceph-mon[30820]: pgmap v13112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:38:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:45 smithi114 ceph-mon[30820]: pgmap v13113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:45 smithi116 ceph-mon[35411]: pgmap v13113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:47 smithi114 ceph-mon[30820]: pgmap v13114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:47 smithi116 ceph-mon[35411]: pgmap v13114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:49 smithi114 ceph-mon[30820]: pgmap v13115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:49 smithi116 ceph-mon[35411]: pgmap v13115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:51 smithi114 ceph-mon[30820]: pgmap v13116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:38:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:51 smithi116 ceph-mon[35411]: pgmap v13116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:38:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:53 smithi114 ceph-mon[30820]: pgmap v13117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:38:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:53 smithi116 ceph-mon[35411]: pgmap v13117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:38:55.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:55 smithi114 ceph-mon[30820]: pgmap v13118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:55 smithi116 ceph-mon[35411]: pgmap v13118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:38:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:57 smithi114 ceph-mon[30820]: pgmap v13119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:57 smithi116 ceph-mon[35411]: pgmap v13119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:38:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:38:59 smithi114 ceph-mon[30820]: pgmap v13120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:38:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:38:59 smithi116 ceph-mon[35411]: pgmap v13120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:01 smithi114 ceph-mon[30820]: pgmap v13121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:39:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:01 smithi116 ceph-mon[35411]: pgmap v13121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:39:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:03 smithi114 ceph-mon[30820]: pgmap v13122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:39:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:03 smithi116 ceph-mon[35411]: pgmap v13122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:39:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:05 smithi114 ceph-mon[30820]: pgmap v13123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:39:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:05 smithi116 ceph-mon[35411]: pgmap v13123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:39:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:07 smithi114 ceph-mon[30820]: pgmap v13124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:39:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:07 smithi116 ceph-mon[35411]: pgmap v13124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:39:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:09 smithi114 ceph-mon[30820]: pgmap v13125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:09 smithi116 ceph-mon[35411]: pgmap v13125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:11 smithi114 ceph-mon[30820]: pgmap v13126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:39:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:11 smithi116 ceph-mon[35411]: pgmap v13126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:39:13.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:13 smithi116 ceph-mon[35411]: pgmap v13127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:39:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:13 smithi114 ceph-mon[30820]: pgmap v13127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:39:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:15 smithi114 ceph-mon[30820]: pgmap v13128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:39:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:15 smithi116 ceph-mon[35411]: pgmap v13128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:39:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:17 smithi114 ceph-mon[30820]: pgmap v13129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:39:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:17 smithi116 ceph-mon[35411]: pgmap v13129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:39:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:19 smithi114 ceph-mon[30820]: pgmap v13130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:19 smithi116 ceph-mon[35411]: pgmap v13130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:39:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:39:22.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: pgmap v13131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:39:22.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.841 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.842 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.842 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.842 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.842 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.842 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: pgmap v13131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:22.843 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:39:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:22 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:39:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:22 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:39:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:22 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:39:23.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:22 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:39:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:22 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:39:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:22 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:39:23.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:22 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:39:23.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:22 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:39:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-mon[30820]: pgmap v13132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:39:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:39:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:39:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:39:23.733+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:39:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:39:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:39:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:39:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:39:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:39:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:39:24.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:39:24.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:39:24.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:39:24.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:39:23 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:39:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:23 smithi116 ceph-mon[35411]: pgmap v13132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:39:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:39:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:23 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:39:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:24 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:39:25.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:39:25.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:39:25.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:39:25.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:39:25.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:39:25.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:39:25.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:24 smithi114 ceph-mon[30820]: pgmap v13133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:39:25.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:39:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:24 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:39:25.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:39:25.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:39:25.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:24 smithi116 ceph-mon[35411]: pgmap v13133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:39:25.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:39:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:26 smithi114 ceph-mon[30820]: pgmap v13134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:39:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:26 smithi116 ceph-mon[35411]: pgmap v13134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:39:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:28 smithi114 ceph-mon[30820]: pgmap v13135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:39:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:28 smithi116 ceph-mon[35411]: pgmap v13135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:39:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:30 smithi114 ceph-mon[30820]: pgmap v13136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:39:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:30 smithi116 ceph-mon[35411]: pgmap v13136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:39:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:32 smithi114 ceph-mon[30820]: pgmap v13137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T13:39:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:32 smithi116 ceph-mon[35411]: pgmap v13137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T13:39:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:34 smithi114 ceph-mon[30820]: pgmap v13138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:39:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:34 smithi116 ceph-mon[35411]: pgmap v13138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:39:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:36 smithi114 ceph-mon[30820]: pgmap v13139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:39:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:36 smithi116 ceph-mon[35411]: pgmap v13139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:39:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:38 smithi114 ceph-mon[30820]: pgmap v13140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:38 smithi116 ceph-mon[35411]: pgmap v13140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:40 smithi114 ceph-mon[30820]: pgmap v13141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:39:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:40 smithi116 ceph-mon[35411]: pgmap v13141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:39:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:42 smithi114 ceph-mon[30820]: pgmap v13142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:39:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:42 smithi116 ceph-mon[35411]: pgmap v13142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:39:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:44 smithi114 ceph-mon[30820]: pgmap v13143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:44 smithi116 ceph-mon[35411]: pgmap v13143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:46 smithi114 ceph-mon[30820]: pgmap v13144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:39:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:46 smithi116 ceph-mon[35411]: pgmap v13144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:39:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:48 smithi114 ceph-mon[30820]: pgmap v13145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:48 smithi116 ceph-mon[35411]: pgmap v13145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:39:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:50 smithi114 ceph-mon[30820]: pgmap v13146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:39:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:50 smithi116 ceph-mon[35411]: pgmap v13146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:39:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:52 smithi114 ceph-mon[30820]: pgmap v13147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:39:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:52 smithi116 ceph-mon[35411]: pgmap v13147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:39:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:54 smithi114 ceph-mon[30820]: pgmap v13148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:39:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:54 smithi116 ceph-mon[35411]: pgmap v13148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:39:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:56 smithi114 ceph-mon[30820]: pgmap v13149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:39:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:56 smithi116 ceph-mon[35411]: pgmap v13149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:39:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:39:58 smithi114 ceph-mon[30820]: pgmap v13150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:39:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:39:58 smithi116 ceph-mon[35411]: pgmap v13150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:40:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:00 smithi114 ceph-mon[30820]: pgmap v13151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:40:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:40:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:00 smithi116 ceph-mon[35411]: pgmap v13151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:40:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:40:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:02 smithi114 ceph-mon[30820]: pgmap v13152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:02 smithi116 ceph-mon[35411]: pgmap v13152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:04 smithi114 ceph-mon[30820]: pgmap v13153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:04 smithi116 ceph-mon[35411]: pgmap v13153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:05 smithi114 ceph-mon[30820]: pgmap v13154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:05 smithi116 ceph-mon[35411]: pgmap v13154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:08 smithi114 ceph-mon[30820]: pgmap v13155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:08 smithi116 ceph-mon[35411]: pgmap v13155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:10 smithi114 ceph-mon[30820]: pgmap v13156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:40:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:10 smithi116 ceph-mon[35411]: pgmap v13156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:40:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:12 smithi114 ceph-mon[30820]: pgmap v13157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:12 smithi116 ceph-mon[35411]: pgmap v13157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:14 smithi114 ceph-mon[30820]: pgmap v13158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:40:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:14 smithi116 ceph-mon[35411]: pgmap v13158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:40:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:16 smithi114 ceph-mon[30820]: pgmap v13159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:16 smithi116 ceph-mon[35411]: pgmap v13159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:18 smithi114 ceph-mon[30820]: pgmap v13160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:40:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:18 smithi116 ceph-mon[35411]: pgmap v13160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:40:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:19 smithi114 ceph-mon[30820]: pgmap v13161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:40:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:19 smithi116 ceph-mon[35411]: pgmap v13161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:40:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:22 smithi114 ceph-mon[30820]: pgmap v13162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:22 smithi116 ceph-mon[35411]: pgmap v13162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:24 smithi114 ceph-mon[30820]: pgmap v13163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:40:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:24 smithi116 ceph-mon[35411]: pgmap v13163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:40:27.095 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:26 smithi114 ceph-mon[30820]: pgmap v13164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:26 smithi116 ceph-mon[35411]: pgmap v13164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:40:27.092+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:40:27.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:40:27.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:40:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:40:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:40:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:40:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:40:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:40:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:40:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:40:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:40:29.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:28 smithi114 ceph-mon[30820]: pgmap v13165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:40:29.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:28 smithi114 ceph-mon[30820]: pgmap v13166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:40:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:40:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:40:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:40:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:40:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:40:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:40:29.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:40:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:40:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:40:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:40:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:28 smithi116 ceph-mon[35411]: pgmap v13165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:40:29.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:28 smithi116 ceph-mon[35411]: pgmap v13166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:40:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:29 smithi114 ceph-mon[30820]: pgmap v13167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:40:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:29 smithi116 ceph-mon[35411]: pgmap v13167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:40:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:32 smithi114 ceph-mon[30820]: pgmap v13168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:40:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:32 smithi116 ceph-mon[35411]: pgmap v13168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:40:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:34 smithi114 ceph-mon[30820]: pgmap v13169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:40:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:34 smithi116 ceph-mon[35411]: pgmap v13169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:40:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:36 smithi114 ceph-mon[30820]: pgmap v13170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:40:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:36 smithi116 ceph-mon[35411]: pgmap v13170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:40:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:38 smithi114 ceph-mon[30820]: pgmap v13171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:40:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:38 smithi116 ceph-mon[35411]: pgmap v13171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:40:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:40 smithi116 ceph-mon[35411]: pgmap v13172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:40 smithi114 ceph-mon[30820]: pgmap v13172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:42 smithi116 ceph-mon[35411]: pgmap v13173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:42 smithi114 ceph-mon[30820]: pgmap v13173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:44 smithi116 ceph-mon[35411]: pgmap v13174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:44 smithi114 ceph-mon[30820]: pgmap v13174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:46 smithi114 ceph-mon[30820]: pgmap v13175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:40:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:46 smithi116 ceph-mon[35411]: pgmap v13175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:40:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:48 smithi114 ceph-mon[30820]: pgmap v13176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:48 smithi116 ceph-mon[35411]: pgmap v13176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:50 smithi114 ceph-mon[30820]: pgmap v13177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:50 smithi116 ceph-mon[35411]: pgmap v13177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:52 smithi114 ceph-mon[30820]: pgmap v13178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:52 smithi116 ceph-mon[35411]: pgmap v13178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:40:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:54 smithi114 ceph-mon[30820]: pgmap v13179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:54 smithi116 ceph-mon[35411]: pgmap v13179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:40:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:56 smithi114 ceph-mon[30820]: pgmap v13180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:40:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:56 smithi116 ceph-mon[35411]: pgmap v13180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:40:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:40:58 smithi114 ceph-mon[30820]: pgmap v13181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:40:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:40:58 smithi116 ceph-mon[35411]: pgmap v13181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:00 smithi114 ceph-mon[30820]: pgmap v13182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:00 smithi116 ceph-mon[35411]: pgmap v13182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:02 smithi114 ceph-mon[30820]: pgmap v13183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:41:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:02 smithi116 ceph-mon[35411]: pgmap v13183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:41:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:04 smithi114 ceph-mon[30820]: pgmap v13184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:04 smithi116 ceph-mon[35411]: pgmap v13184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:06 smithi114 ceph-mon[30820]: pgmap v13185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:06 smithi116 ceph-mon[35411]: pgmap v13185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:08 smithi114 ceph-mon[30820]: pgmap v13186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:08 smithi116 ceph-mon[35411]: pgmap v13186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:10 smithi114 ceph-mon[30820]: pgmap v13187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:10 smithi116 ceph-mon[35411]: pgmap v13187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:12 smithi114 ceph-mon[30820]: pgmap v13188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:41:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:12 smithi116 ceph-mon[35411]: pgmap v13188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:41:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:14 smithi114 ceph-mon[30820]: pgmap v13189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:14 smithi116 ceph-mon[35411]: pgmap v13189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:16 smithi114 ceph-mon[30820]: pgmap v13190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:16 smithi116 ceph-mon[35411]: pgmap v13190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:18 smithi114 ceph-mon[30820]: pgmap v13191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:18 smithi116 ceph-mon[35411]: pgmap v13191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:20 smithi114 ceph-mon[30820]: pgmap v13192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:20 smithi116 ceph-mon[35411]: pgmap v13192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:22 smithi114 ceph-mon[30820]: pgmap v13193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:41:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:22 smithi116 ceph-mon[35411]: pgmap v13193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:41:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:24 smithi114 ceph-mon[30820]: pgmap v13194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:24 smithi116 ceph-mon[35411]: pgmap v13194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:26 smithi114 ceph-mon[30820]: pgmap v13195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:26 smithi116 ceph-mon[35411]: pgmap v13195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:27 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:41:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:27 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:41:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:28 smithi114 ceph-mon[30820]: pgmap v13196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:28 smithi116 ceph-mon[35411]: pgmap v13196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:30.784 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:41:30.452+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:41:30.784 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:41:30.784 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:41:30.784 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:41:30.784 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:41:30.784 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:41:30.784 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:41:30.785 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:41:30.785 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:41:30.785 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:41:30.785 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:41:30.785 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:41:30.785 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:41:30.785 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:41:30.785 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:41:30.786 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:41:30.786 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:41:30.786 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:41:30.786 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:41:30.786 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:41:30.786 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:41:30.786 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:41:30.786 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:41:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-mon[30820]: pgmap v13197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:41:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:41:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:30 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:41:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:30 smithi116 ceph-mon[35411]: pgmap v13197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:41:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:41:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:30 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:41:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:31 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:41:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:41:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:41:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:41:32.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:41:32.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:41:32.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:41:32.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:41:32.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:41:32.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:41:32.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:31 smithi114 ceph-mon[30820]: pgmap v13198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T13:41:32.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:31 smithi114 ceph-mon[30820]: pgmap v13199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:31 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:41:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:41:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:41:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:41:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:41:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:41:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:41:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:41:32.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:41:32.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:41:32.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:41:32.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:31 smithi116 ceph-mon[35411]: pgmap v13198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T13:41:32.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:31 smithi116 ceph-mon[35411]: pgmap v13199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:33 smithi114 ceph-mon[30820]: pgmap v13200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:33 smithi116 ceph-mon[35411]: pgmap v13200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:35 smithi114 ceph-mon[30820]: pgmap v13201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:35 smithi116 ceph-mon[35411]: pgmap v13201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:37 smithi114 ceph-mon[30820]: pgmap v13202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:37 smithi116 ceph-mon[35411]: pgmap v13202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:39 smithi114 ceph-mon[30820]: pgmap v13203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:39 smithi116 ceph-mon[35411]: pgmap v13203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:41:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:41 smithi114 ceph-mon[30820]: pgmap v13204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:41:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:41 smithi116 ceph-mon[35411]: pgmap v13204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:41:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:43 smithi114 ceph-mon[30820]: pgmap v13205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:43 smithi116 ceph-mon[35411]: pgmap v13205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:45 smithi114 ceph-mon[30820]: pgmap v13206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:45 smithi116 ceph-mon[35411]: pgmap v13206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:47 smithi114 ceph-mon[30820]: pgmap v13207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:47 smithi116 ceph-mon[35411]: pgmap v13207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:49 smithi114 ceph-mon[30820]: pgmap v13208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:49 smithi116 ceph-mon[35411]: pgmap v13208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:51 smithi114 ceph-mon[30820]: pgmap v13209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:41:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:51 smithi116 ceph-mon[35411]: pgmap v13209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:41:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:53 smithi114 ceph-mon[30820]: pgmap v13210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:53 smithi116 ceph-mon[35411]: pgmap v13210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:41:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:55 smithi114 ceph-mon[30820]: pgmap v13211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:55 smithi116 ceph-mon[35411]: pgmap v13211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:41:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:57 smithi114 ceph-mon[30820]: pgmap v13212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:41:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:57 smithi116 ceph-mon[35411]: pgmap v13212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:41:59 smithi114 ceph-mon[30820]: pgmap v13213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:41:59 smithi116 ceph-mon[35411]: pgmap v13213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:01 smithi114 ceph-mon[30820]: pgmap v13214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:01 smithi116 ceph-mon[35411]: pgmap v13214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:03 smithi114 ceph-mon[30820]: pgmap v13215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:03 smithi116 ceph-mon[35411]: pgmap v13215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:04 smithi114 ceph-mon[30820]: pgmap v13216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:42:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:04 smithi116 ceph-mon[35411]: pgmap v13216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:42:07.870 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:07 smithi114 ceph-mon[30820]: pgmap v13217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:07 smithi116 ceph-mon[35411]: pgmap v13217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:09 smithi114 ceph-mon[30820]: pgmap v13218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:09 smithi116 ceph-mon[35411]: pgmap v13218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:11 smithi114 ceph-mon[30820]: pgmap v13219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:11 smithi116 ceph-mon[35411]: pgmap v13219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:13 smithi114 ceph-mon[30820]: pgmap v13220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:13 smithi116 ceph-mon[35411]: pgmap v13220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:15 smithi114 ceph-mon[30820]: pgmap v13221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:42:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:15 smithi116 ceph-mon[35411]: pgmap v13221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:42:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:17 smithi114 ceph-mon[30820]: pgmap v13222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:17 smithi116 ceph-mon[35411]: pgmap v13222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:19 smithi114 ceph-mon[30820]: pgmap v13223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:19 smithi116 ceph-mon[35411]: pgmap v13223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:21 smithi114 ceph-mon[30820]: pgmap v13224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:21 smithi116 ceph-mon[35411]: pgmap v13224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:23 smithi114 ceph-mon[30820]: pgmap v13225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:23 smithi116 ceph-mon[35411]: pgmap v13225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:25 smithi114 ceph-mon[30820]: pgmap v13226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:42:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:25 smithi116 ceph-mon[35411]: pgmap v13226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:42:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:27 smithi114 ceph-mon[30820]: pgmap v13227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:27 smithi116 ceph-mon[35411]: pgmap v13227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:29 smithi114 ceph-mon[30820]: pgmap v13228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:42:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:29 smithi116 ceph-mon[35411]: pgmap v13228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:42:31.833 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:31 smithi114 ceph-mon[30820]: pgmap v13229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:31.834 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:42:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:31 smithi116 ceph-mon[35411]: pgmap v13229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:42:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-mon[30820]: pgmap v13230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:42:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:33 smithi116 ceph-mon[35411]: pgmap v13230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:42:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:34 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:42:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:42:34.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:34 smithi114 ceph-mon[30820]: pgmap v13231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:34 smithi114 ceph-mon[30820]: pgmap v13232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:34.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:42:33.812+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:42:34.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:42:34.171 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:42:33 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:42:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:34 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:42:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:42:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:42:34.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:34 smithi116 ceph-mon[35411]: pgmap v13231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:34 smithi116 ceph-mon[35411]: pgmap v13232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T13:42:34.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:42:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:35 smithi114 ceph-mon[30820]: pgmap v13233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:42:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:35 smithi116 ceph-mon[35411]: pgmap v13233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:42:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:37 smithi114 ceph-mon[30820]: pgmap v13234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:42:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:37 smithi116 ceph-mon[35411]: pgmap v13234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:42:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:39 smithi114 ceph-mon[30820]: pgmap v13235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:42:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:39 smithi116 ceph-mon[35411]: pgmap v13235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:42:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:41 smithi114 ceph-mon[30820]: pgmap v13236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:42:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:41 smithi116 ceph-mon[35411]: pgmap v13236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:42:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:43 smithi114 ceph-mon[30820]: pgmap v13237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:42:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:43 smithi116 ceph-mon[35411]: pgmap v13237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:42:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:45 smithi114 ceph-mon[30820]: pgmap v13238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:45 smithi116 ceph-mon[35411]: pgmap v13238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:47 smithi114 ceph-mon[30820]: pgmap v13239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:47 smithi116 ceph-mon[35411]: pgmap v13239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:49 smithi114 ceph-mon[30820]: pgmap v13240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:42:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:49 smithi116 ceph-mon[35411]: pgmap v13240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:42:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:51 smithi114 ceph-mon[30820]: pgmap v13241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:51 smithi116 ceph-mon[35411]: pgmap v13241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:42:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:53 smithi114 ceph-mon[30820]: pgmap v13242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:53 smithi116 ceph-mon[35411]: pgmap v13242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:55 smithi114 ceph-mon[30820]: pgmap v13243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:55 smithi116 ceph-mon[35411]: pgmap v13243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:42:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:57 smithi114 ceph-mon[30820]: pgmap v13244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:42:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:57 smithi116 ceph-mon[35411]: pgmap v13244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:42:59 smithi114 ceph-mon[30820]: pgmap v13245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:42:59 smithi116 ceph-mon[35411]: pgmap v13245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:01 smithi114 ceph-mon[30820]: pgmap v13246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:01 smithi116 ceph-mon[35411]: pgmap v13246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:03 smithi114 ceph-mon[30820]: pgmap v13247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:03 smithi116 ceph-mon[35411]: pgmap v13247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:05 smithi114 ceph-mon[30820]: pgmap v13248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:05 smithi116 ceph-mon[35411]: pgmap v13248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:07 smithi114 ceph-mon[30820]: pgmap v13249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:07 smithi116 ceph-mon[35411]: pgmap v13249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:09 smithi114 ceph-mon[30820]: pgmap v13250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:09 smithi116 ceph-mon[35411]: pgmap v13250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:11 smithi114 ceph-mon[30820]: pgmap v13251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:11 smithi116 ceph-mon[35411]: pgmap v13251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:13 smithi114 ceph-mon[30820]: pgmap v13252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:13 smithi116 ceph-mon[35411]: pgmap v13252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:15 smithi114 ceph-mon[30820]: pgmap v13253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:15 smithi116 ceph-mon[35411]: pgmap v13253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:17 smithi114 ceph-mon[30820]: pgmap v13254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:17 smithi116 ceph-mon[35411]: pgmap v13254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:19 smithi114 ceph-mon[30820]: pgmap v13255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:19 smithi116 ceph-mon[35411]: pgmap v13255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:21 smithi114 ceph-mon[30820]: pgmap v13256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:21 smithi116 ceph-mon[35411]: pgmap v13256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:23 smithi114 ceph-mon[30820]: pgmap v13257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:23 smithi116 ceph-mon[35411]: pgmap v13257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:25 smithi114 ceph-mon[30820]: pgmap v13258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:25 smithi116 ceph-mon[35411]: pgmap v13258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:27 smithi114 ceph-mon[30820]: pgmap v13259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:27 smithi116 ceph-mon[35411]: pgmap v13259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:29 smithi114 ceph-mon[30820]: pgmap v13260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:29 smithi116 ceph-mon[35411]: pgmap v13260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:31 smithi114 ceph-mon[30820]: pgmap v13261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:31 smithi116 ceph-mon[35411]: pgmap v13261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:33 smithi114 ceph-mon[30820]: pgmap v13262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:33 smithi116 ceph-mon[35411]: pgmap v13262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:43:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:43:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:35 smithi116 ceph-mon[35411]: pgmap v13263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:35 smithi114 ceph-mon[30820]: pgmap v13263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:43:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:43:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:43:37.172+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:43:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:43:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:43:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:43:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:43:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:43:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:43:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:38 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:43:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:43:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:43:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:43:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:43:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:43:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:43:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:43:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:43:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:43:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:38 smithi114 ceph-mon[30820]: pgmap v13264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:43:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:38 smithi114 ceph-mon[30820]: pgmap v13265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:43:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:38 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:43:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:43:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:38 smithi116 ceph-mon[35411]: pgmap v13264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:43:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:38 smithi116 ceph-mon[35411]: pgmap v13265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:43:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:40 smithi114 ceph-mon[30820]: pgmap v13266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:43:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:40 smithi116 ceph-mon[35411]: pgmap v13266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:43:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:42 smithi114 ceph-mon[30820]: pgmap v13267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:43:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:42 smithi116 ceph-mon[35411]: pgmap v13267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:43:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:44 smithi114 ceph-mon[30820]: pgmap v13268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:43:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:44 smithi116 ceph-mon[35411]: pgmap v13268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:43:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:46 smithi114 ceph-mon[30820]: pgmap v13269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:43:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:46 smithi116 ceph-mon[35411]: pgmap v13269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:43:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:48 smithi114 ceph-mon[30820]: pgmap v13270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:43:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:48 smithi116 ceph-mon[35411]: pgmap v13270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:43:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:50 smithi114 ceph-mon[30820]: pgmap v13271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:50 smithi116 ceph-mon[35411]: pgmap v13271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:52 smithi114 ceph-mon[30820]: pgmap v13272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:52 smithi116 ceph-mon[35411]: pgmap v13272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:43:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:54 smithi114 ceph-mon[30820]: pgmap v13273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:54 smithi116 ceph-mon[35411]: pgmap v13273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:43:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:56 smithi114 ceph-mon[30820]: pgmap v13274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:56 smithi116 ceph-mon[35411]: pgmap v13274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:43:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:43:58 smithi114 ceph-mon[30820]: pgmap v13275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:43:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:43:58 smithi116 ceph-mon[35411]: pgmap v13275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:00 smithi114 ceph-mon[30820]: pgmap v13276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:00 smithi116 ceph-mon[35411]: pgmap v13276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:02 smithi114 ceph-mon[30820]: pgmap v13277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:44:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:02 smithi116 ceph-mon[35411]: pgmap v13277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:44:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:04 smithi114 ceph-mon[30820]: pgmap v13278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:04 smithi116 ceph-mon[35411]: pgmap v13278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:06 smithi114 ceph-mon[30820]: pgmap v13279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:06 smithi116 ceph-mon[35411]: pgmap v13279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:08 smithi114 ceph-mon[30820]: pgmap v13280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:08 smithi116 ceph-mon[35411]: pgmap v13280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:10 smithi114 ceph-mon[30820]: pgmap v13281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:10 smithi116 ceph-mon[35411]: pgmap v13281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:12 smithi114 ceph-mon[30820]: pgmap v13282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:44:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:12 smithi116 ceph-mon[35411]: pgmap v13282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:44:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:14 smithi114 ceph-mon[30820]: pgmap v13283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:14 smithi116 ceph-mon[35411]: pgmap v13283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:16 smithi114 ceph-mon[30820]: pgmap v13284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:16 smithi116 ceph-mon[35411]: pgmap v13284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:18 smithi114 ceph-mon[30820]: pgmap v13285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:18 smithi116 ceph-mon[35411]: pgmap v13285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:20 smithi114 ceph-mon[30820]: pgmap v13286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:20 smithi116 ceph-mon[35411]: pgmap v13286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:22 smithi114 ceph-mon[30820]: pgmap v13287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:44:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:22 smithi116 ceph-mon[35411]: pgmap v13287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:44:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:24 smithi114 ceph-mon[30820]: pgmap v13288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:24 smithi116 ceph-mon[35411]: pgmap v13288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:26 smithi114 ceph-mon[30820]: pgmap v13289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:26 smithi116 ceph-mon[35411]: pgmap v13289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:28 smithi114 ceph-mon[30820]: pgmap v13290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:28 smithi116 ceph-mon[35411]: pgmap v13290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:30 smithi114 ceph-mon[30820]: pgmap v13291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:30 smithi116 ceph-mon[35411]: pgmap v13291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:32 smithi114 ceph-mon[30820]: pgmap v13292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:44:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:32 smithi116 ceph-mon[35411]: pgmap v13292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:44:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:34 smithi114 ceph-mon[30820]: pgmap v13293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:34 smithi116 ceph-mon[35411]: pgmap v13293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:36 smithi114 ceph-mon[30820]: pgmap v13294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:36 smithi116 ceph-mon[35411]: pgmap v13294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:44:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:44:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:38 smithi114 ceph-mon[30820]: pgmap v13295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:38 smithi116 ceph-mon[35411]: pgmap v13295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:40.872 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:44:40.532+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:44:40.872 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:44:40.872 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:44:40.872 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:44:40.872 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:44:40.872 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:44:40.872 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:44:40.872 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:44:40.873 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:44:40.874 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:44:40.874 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:44:40.874 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:44:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-mon[30820]: pgmap v13296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:44:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:44:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:44:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:40 smithi116 ceph-mon[35411]: pgmap v13296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:44:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:44:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:44:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:44:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:44:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:44:42.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:41 smithi114 ceph-mon[30820]: pgmap v13297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T13:44:42.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:41 smithi114 ceph-mon[30820]: pgmap v13298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:44:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:44:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:44:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:44:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:44:42.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:44:42.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:44:42.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:44:42.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:44:42.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:44:42.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:44:42.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:44:42.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:41 smithi116 ceph-mon[35411]: pgmap v13297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T13:44:42.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:41 smithi116 ceph-mon[35411]: pgmap v13298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:43 smithi114 ceph-mon[30820]: pgmap v13299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:43 smithi116 ceph-mon[35411]: pgmap v13299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:44 smithi114 ceph-mon[30820]: pgmap v13300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:44 smithi116 ceph-mon[35411]: pgmap v13300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:47.882 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:47 smithi116 ceph-mon[35411]: pgmap v13301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:47 smithi114 ceph-mon[30820]: pgmap v13301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:49 smithi114 ceph-mon[30820]: pgmap v13302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:49 smithi116 ceph-mon[35411]: pgmap v13302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:44:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:51 smithi114 ceph-mon[30820]: pgmap v13303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:44:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:51 smithi116 ceph-mon[35411]: pgmap v13303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:44:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:53 smithi114 ceph-mon[30820]: pgmap v13304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:53 smithi116 ceph-mon[35411]: pgmap v13304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:55 smithi114 ceph-mon[30820]: pgmap v13305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:55 smithi116 ceph-mon[35411]: pgmap v13305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:44:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:57 smithi114 ceph-mon[30820]: pgmap v13306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:57 smithi116 ceph-mon[35411]: pgmap v13306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:44:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:44:59 smithi114 ceph-mon[30820]: pgmap v13307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:44:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:44:59 smithi116 ceph-mon[35411]: pgmap v13307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:01 smithi114 ceph-mon[30820]: pgmap v13308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:01.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:01 smithi116 ceph-mon[35411]: pgmap v13308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:03 smithi114 ceph-mon[30820]: pgmap v13309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:03 smithi116 ceph-mon[35411]: pgmap v13309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:05 smithi114 ceph-mon[30820]: pgmap v13310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:45:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:05 smithi116 ceph-mon[35411]: pgmap v13310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:45:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:07 smithi114 ceph-mon[30820]: pgmap v13311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:45:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:07 smithi116 ceph-mon[35411]: pgmap v13311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:45:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:09 smithi114 ceph-mon[30820]: pgmap v13312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:09 smithi116 ceph-mon[35411]: pgmap v13312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:11 smithi114 ceph-mon[30820]: pgmap v13313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:11 smithi116 ceph-mon[35411]: pgmap v13313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:13 smithi114 ceph-mon[30820]: pgmap v13314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:13 smithi116 ceph-mon[35411]: pgmap v13314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:15 smithi114 ceph-mon[30820]: pgmap v13315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:45:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:15 smithi116 ceph-mon[35411]: pgmap v13315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:45:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:17 smithi114 ceph-mon[30820]: pgmap v13316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:45:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:17 smithi116 ceph-mon[35411]: pgmap v13316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:45:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:19 smithi114 ceph-mon[30820]: pgmap v13317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:19 smithi116 ceph-mon[35411]: pgmap v13317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:21 smithi114 ceph-mon[30820]: pgmap v13318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:21 smithi116 ceph-mon[35411]: pgmap v13318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:23 smithi114 ceph-mon[30820]: pgmap v13319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:23 smithi116 ceph-mon[35411]: pgmap v13319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:25 smithi114 ceph-mon[30820]: pgmap v13320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:45:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:25 smithi116 ceph-mon[35411]: pgmap v13320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:45:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:27 smithi114 ceph-mon[30820]: pgmap v13321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:45:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:27 smithi116 ceph-mon[35411]: pgmap v13321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:45:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:29 smithi114 ceph-mon[30820]: pgmap v13322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:29 smithi116 ceph-mon[35411]: pgmap v13322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:31 smithi114 ceph-mon[30820]: pgmap v13323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:32.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:31 smithi116 ceph-mon[35411]: pgmap v13323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:33 smithi114 ceph-mon[30820]: pgmap v13324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:33 smithi116 ceph-mon[35411]: pgmap v13324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:35 smithi114 ceph-mon[30820]: pgmap v13325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:45:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:35 smithi116 ceph-mon[35411]: pgmap v13325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:45:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:37 smithi114 ceph-mon[30820]: pgmap v13326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:45:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:37 smithi116 ceph-mon[35411]: pgmap v13326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:45:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:39 smithi114 ceph-mon[30820]: pgmap v13327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:39 smithi116 ceph-mon[35411]: pgmap v13327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:45:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:45:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:41 smithi114 ceph-mon[30820]: pgmap v13328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:41 smithi116 ceph-mon[35411]: pgmap v13328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-mon[30820]: pgmap v13329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:45:43.892+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:45:44.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:45:44.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:45:44.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:45:44.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:45:43 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:45:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:43 smithi116 ceph-mon[35411]: pgmap v13329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:45:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:45:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:45:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:44 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:45:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:45:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:45:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:44 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:45:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:45 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:45:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:45:46.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:45 smithi114 ceph-mon[30820]: pgmap v13330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:45:46.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:45 smithi114 ceph-mon[30820]: pgmap v13331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:45:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:45 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:45:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:45:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:45:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:45:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:45:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:45:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:45:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:45:46.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:45:46.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:45:46.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:45:46.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:45 smithi116 ceph-mon[35411]: pgmap v13330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:45:46.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:45 smithi116 ceph-mon[35411]: pgmap v13331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:45:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:47 smithi114 ceph-mon[30820]: pgmap v13332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:45:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:47 smithi116 ceph-mon[35411]: pgmap v13332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:45:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:49 smithi114 ceph-mon[30820]: pgmap v13333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:45:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:49 smithi116 ceph-mon[35411]: pgmap v13333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:45:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:51 smithi114 ceph-mon[30820]: pgmap v13334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:45:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:51 smithi116 ceph-mon[35411]: pgmap v13334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:45:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:53 smithi114 ceph-mon[30820]: pgmap v13335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:45:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:53 smithi116 ceph-mon[35411]: pgmap v13335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:45:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:55 smithi114 ceph-mon[30820]: pgmap v13336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:45:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:55 smithi116 ceph-mon[35411]: pgmap v13336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:45:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:57 smithi114 ceph-mon[30820]: pgmap v13337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:45:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:57 smithi116 ceph-mon[35411]: pgmap v13337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:45:59 smithi114 ceph-mon[30820]: pgmap v13338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:45:59 smithi116 ceph-mon[35411]: pgmap v13338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:01 smithi114 ceph-mon[30820]: pgmap v13339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:01 smithi116 ceph-mon[35411]: pgmap v13339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:03 smithi114 ceph-mon[30820]: pgmap v13340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:03 smithi116 ceph-mon[35411]: pgmap v13340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:04 smithi114 ceph-mon[30820]: pgmap v13341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:04 smithi116 ceph-mon[35411]: pgmap v13341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:06 smithi114 ceph-mon[30820]: pgmap v13342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:06 smithi116 ceph-mon[35411]: pgmap v13342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:09.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:08 smithi114 ceph-mon[30820]: pgmap v13343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:08 smithi116 ceph-mon[35411]: pgmap v13343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:10 smithi114 ceph-mon[30820]: pgmap v13344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:10 smithi116 ceph-mon[35411]: pgmap v13344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:12 smithi114 ceph-mon[30820]: pgmap v13345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:12 smithi116 ceph-mon[35411]: pgmap v13345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:14 smithi114 ceph-mon[30820]: pgmap v13346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:14 smithi116 ceph-mon[35411]: pgmap v13346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:17.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:16 smithi116 ceph-mon[35411]: pgmap v13347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:16 smithi114 ceph-mon[30820]: pgmap v13347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:18 smithi114 ceph-mon[30820]: pgmap v13348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:18 smithi116 ceph-mon[35411]: pgmap v13348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:20 smithi114 ceph-mon[30820]: pgmap v13349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:20 smithi116 ceph-mon[35411]: pgmap v13349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:22 smithi114 ceph-mon[30820]: pgmap v13350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:22 smithi116 ceph-mon[35411]: pgmap v13350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:24 smithi114 ceph-mon[30820]: pgmap v13351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:24 smithi116 ceph-mon[35411]: pgmap v13351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:26 smithi114 ceph-mon[30820]: pgmap v13352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:26 smithi116 ceph-mon[35411]: pgmap v13352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:28 smithi114 ceph-mon[30820]: pgmap v13353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:28 smithi116 ceph-mon[35411]: pgmap v13353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:30 smithi114 ceph-mon[30820]: pgmap v13354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:30 smithi116 ceph-mon[35411]: pgmap v13354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:32 smithi114 ceph-mon[30820]: pgmap v13355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:32 smithi116 ceph-mon[35411]: pgmap v13355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:34 smithi114 ceph-mon[30820]: pgmap v13356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:34 smithi116 ceph-mon[35411]: pgmap v13356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:36 smithi114 ceph-mon[30820]: pgmap v13357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:36 smithi116 ceph-mon[35411]: pgmap v13357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:39 smithi114 ceph-mon[30820]: pgmap v13358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:39 smithi116 ceph-mon[35411]: pgmap v13358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:41 smithi114 ceph-mon[30820]: pgmap v13359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:41 smithi116 ceph-mon[35411]: pgmap v13359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:46:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:43 smithi114 ceph-mon[30820]: pgmap v13360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:43 smithi116 ceph-mon[35411]: pgmap v13360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:46:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:45 smithi114 ceph-mon[30820]: pgmap v13361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:46:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:45 smithi116 ceph-mon[35411]: pgmap v13361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:46:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:46:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-mon[30820]: pgmap v13362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:46:47.253+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:46:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:46:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:46:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:46:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:46:47.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:46:47.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:46:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:46:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:46:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:46:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:46:47.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:46:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:46:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:47 smithi116 ceph-mon[35411]: pgmap v13362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:46:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:48 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:46:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:46:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:46:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:46:48.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:46:48.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:46:48.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:46:48.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:46:48.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:48 smithi114 ceph-mon[30820]: pgmap v13363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:46:48.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:48 smithi114 ceph-mon[30820]: pgmap v13364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:46:48.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:46:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:48 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:46:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:46:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:46:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:48 smithi116 ceph-mon[35411]: pgmap v13363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:48 smithi116 ceph-mon[35411]: pgmap v13364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:46:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:46:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:50 smithi114 ceph-mon[30820]: pgmap v13365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:46:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:50 smithi116 ceph-mon[35411]: pgmap v13365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:46:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:52 smithi114 ceph-mon[30820]: pgmap v13366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:46:52.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:52 smithi116 ceph-mon[35411]: pgmap v13366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:46:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:54 smithi114 ceph-mon[30820]: pgmap v13367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:46:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:54 smithi116 ceph-mon[35411]: pgmap v13367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:46:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:56 smithi114 ceph-mon[30820]: pgmap v13368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:46:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:56 smithi116 ceph-mon[35411]: pgmap v13368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:46:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:46:58 smithi114 ceph-mon[30820]: pgmap v13369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:46:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:46:58 smithi116 ceph-mon[35411]: pgmap v13369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:47:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:00 smithi114 ceph-mon[30820]: pgmap v13370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:00 smithi116 ceph-mon[35411]: pgmap v13370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:02 smithi114 ceph-mon[30820]: pgmap v13371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:02 smithi116 ceph-mon[35411]: pgmap v13371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:04 smithi114 ceph-mon[30820]: pgmap v13372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:04 smithi116 ceph-mon[35411]: pgmap v13372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:06 smithi114 ceph-mon[30820]: pgmap v13373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:06 smithi116 ceph-mon[35411]: pgmap v13373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:08 smithi114 ceph-mon[30820]: pgmap v13374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:08 smithi116 ceph-mon[35411]: pgmap v13374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:10 smithi114 ceph-mon[30820]: pgmap v13375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:10 smithi116 ceph-mon[35411]: pgmap v13375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:12 smithi114 ceph-mon[30820]: pgmap v13376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:12 smithi116 ceph-mon[35411]: pgmap v13376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:14 smithi114 ceph-mon[30820]: pgmap v13377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:14 smithi116 ceph-mon[35411]: pgmap v13377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:16 smithi114 ceph-mon[30820]: pgmap v13378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:16 smithi116 ceph-mon[35411]: pgmap v13378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:18 smithi114 ceph-mon[30820]: pgmap v13379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:47:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:18 smithi116 ceph-mon[35411]: pgmap v13379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:47:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:20 smithi114 ceph-mon[30820]: pgmap v13380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:20 smithi116 ceph-mon[35411]: pgmap v13380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:22 smithi114 ceph-mon[30820]: pgmap v13381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:22 smithi116 ceph-mon[35411]: pgmap v13381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:24 smithi114 ceph-mon[30820]: pgmap v13382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:24 smithi116 ceph-mon[35411]: pgmap v13382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:26 smithi114 ceph-mon[30820]: pgmap v13383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:26 smithi116 ceph-mon[35411]: pgmap v13383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:28 smithi114 ceph-mon[30820]: pgmap v13384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:47:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:28 smithi116 ceph-mon[35411]: pgmap v13384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:47:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:30 smithi114 ceph-mon[30820]: pgmap v13385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:30 smithi116 ceph-mon[35411]: pgmap v13385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:32 smithi114 ceph-mon[30820]: pgmap v13386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:32 smithi116 ceph-mon[35411]: pgmap v13386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:34 smithi114 ceph-mon[30820]: pgmap v13387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:34 smithi116 ceph-mon[35411]: pgmap v13387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:36 smithi114 ceph-mon[30820]: pgmap v13388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:36 smithi116 ceph-mon[35411]: pgmap v13388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:38 smithi114 ceph-mon[30820]: pgmap v13389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:47:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:38 smithi116 ceph-mon[35411]: pgmap v13389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:47:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:40 smithi114 ceph-mon[30820]: pgmap v13390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:40 smithi116 ceph-mon[35411]: pgmap v13390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:42 smithi114 ceph-mon[30820]: pgmap v13391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:42 smithi116 ceph-mon[35411]: pgmap v13391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:47:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:44 smithi116 ceph-mon[35411]: pgmap v13392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:44 smithi114 ceph-mon[30820]: pgmap v13392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:47:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:46 smithi116 ceph-mon[35411]: pgmap v13393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:46 smithi114 ceph-mon[30820]: pgmap v13393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:47:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:47:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:47:48.831 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:48 smithi116 ceph-mon[35411]: pgmap v13394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:47:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:48 smithi114 ceph-mon[30820]: pgmap v13394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:47:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-mon[30820]: pgmap v13395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:47:50.612+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:47:50.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:47:50.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:47:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:47:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:50 smithi116 ceph-mon[35411]: pgmap v13395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:47:51.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:47:51.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:47:51.920 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:47:51.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:51 smithi114 ceph-mon[30820]: pgmap v13396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:47:51.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:51 smithi114 ceph-mon[30820]: pgmap v13397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:47:51.923 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:47:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:47:51.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:51 smithi116 ceph-mon[35411]: pgmap v13396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:51 smithi116 ceph-mon[35411]: pgmap v13397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:47:51.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:47:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:53 smithi116 ceph-mon[35411]: pgmap v13398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:47:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:53 smithi114 ceph-mon[30820]: pgmap v13398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:47:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:55 smithi116 ceph-mon[35411]: pgmap v13399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:47:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:55 smithi114 ceph-mon[30820]: pgmap v13399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T13:47:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:57 smithi114 ceph-mon[30820]: pgmap v13400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:47:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:57 smithi116 ceph-mon[35411]: pgmap v13400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:48:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:47:59 smithi114 ceph-mon[30820]: pgmap v13401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:48:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:47:59 smithi116 ceph-mon[35411]: pgmap v13401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:48:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:01 smithi114 ceph-mon[30820]: pgmap v13402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:48:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:01 smithi116 ceph-mon[35411]: pgmap v13402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:48:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:03 smithi114 ceph-mon[30820]: pgmap v13403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:03 smithi116 ceph-mon[35411]: pgmap v13403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:05 smithi114 ceph-mon[30820]: pgmap v13404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:05 smithi116 ceph-mon[35411]: pgmap v13404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:07 smithi114 ceph-mon[30820]: pgmap v13405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:07 smithi116 ceph-mon[35411]: pgmap v13405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:09 smithi114 ceph-mon[30820]: pgmap v13406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:09 smithi116 ceph-mon[35411]: pgmap v13406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:11.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:11 smithi114 ceph-mon[30820]: pgmap v13407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:11 smithi116 ceph-mon[35411]: pgmap v13407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:13 smithi114 ceph-mon[30820]: pgmap v13408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:13 smithi116 ceph-mon[35411]: pgmap v13408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:15 smithi114 ceph-mon[30820]: pgmap v13409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:15 smithi116 ceph-mon[35411]: pgmap v13409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:17 smithi114 ceph-mon[30820]: pgmap v13410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:17 smithi116 ceph-mon[35411]: pgmap v13410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:19 smithi114 ceph-mon[30820]: pgmap v13411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:19 smithi116 ceph-mon[35411]: pgmap v13411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:21 smithi114 ceph-mon[30820]: pgmap v13412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:21 smithi116 ceph-mon[35411]: pgmap v13412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:23 smithi114 ceph-mon[30820]: pgmap v13413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:23 smithi116 ceph-mon[35411]: pgmap v13413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:25 smithi114 ceph-mon[30820]: pgmap v13414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:25 smithi116 ceph-mon[35411]: pgmap v13414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:27 smithi114 ceph-mon[30820]: pgmap v13415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:27 smithi116 ceph-mon[35411]: pgmap v13415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:29 smithi114 ceph-mon[30820]: pgmap v13416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:29 smithi116 ceph-mon[35411]: pgmap v13416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:31 smithi114 ceph-mon[30820]: pgmap v13417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:31 smithi116 ceph-mon[35411]: pgmap v13417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:33 smithi114 ceph-mon[30820]: pgmap v13418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:33 smithi116 ceph-mon[35411]: pgmap v13418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:35 smithi114 ceph-mon[30820]: pgmap v13419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:35 smithi116 ceph-mon[35411]: pgmap v13419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:37 smithi114 ceph-mon[30820]: pgmap v13420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:37 smithi116 ceph-mon[35411]: pgmap v13420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:39 smithi114 ceph-mon[30820]: pgmap v13421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:39 smithi116 ceph-mon[35411]: pgmap v13421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:41 smithi114 ceph-mon[30820]: pgmap v13422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:41 smithi116 ceph-mon[35411]: pgmap v13422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:43 smithi114 ceph-mon[30820]: pgmap v13423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:43 smithi116 ceph-mon[35411]: pgmap v13423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:45 smithi114 ceph-mon[30820]: pgmap v13424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:45 smithi116 ceph-mon[35411]: pgmap v13424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:48:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:47 smithi114 ceph-mon[30820]: pgmap v13425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:47 smithi116 ceph-mon[35411]: pgmap v13425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:48:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:49 smithi114 ceph-mon[30820]: pgmap v13426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:49 smithi116 ceph-mon[35411]: pgmap v13426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:50 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:48:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:50 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:48:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:51 smithi114 ceph-mon[30820]: pgmap v13427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:51 smithi116 ceph-mon[35411]: pgmap v13427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:48:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-mon[30820]: pgmap v13428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:48:53.972+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:48:54.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:48:54.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:48:54.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:48:53 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:48:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:53 smithi116 ceph-mon[35411]: pgmap v13428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:48:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:48:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:48:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:48:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:48:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:48:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:48:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:55 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:48:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:48:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:48:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:48:56.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:48:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:48:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:48:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:48:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:48:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:55 smithi114 ceph-mon[30820]: pgmap v13429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:48:56.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:55 smithi114 ceph-mon[30820]: pgmap v13430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:48:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:55 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:48:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:48:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:48:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:48:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:48:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:48:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:48:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:48:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:48:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:55 smithi116 ceph-mon[35411]: pgmap v13429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:48:56.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:55 smithi116 ceph-mon[35411]: pgmap v13430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:48:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:57 smithi114 ceph-mon[30820]: pgmap v13431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:48:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:57 smithi116 ceph-mon[35411]: pgmap v13431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:49:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:48:59 smithi114 ceph-mon[30820]: pgmap v13432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:49:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:48:59 smithi116 ceph-mon[35411]: pgmap v13432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:49:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:01 smithi114 ceph-mon[30820]: pgmap v13433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:49:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:01 smithi116 ceph-mon[35411]: pgmap v13433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:49:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:03 smithi114 ceph-mon[30820]: pgmap v13434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:49:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:03 smithi116 ceph-mon[35411]: pgmap v13434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:49:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:05 smithi114 ceph-mon[30820]: pgmap v13435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:49:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:05 smithi116 ceph-mon[35411]: pgmap v13435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:49:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:07 smithi114 ceph-mon[30820]: pgmap v13436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:07 smithi116 ceph-mon[35411]: pgmap v13436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:09 smithi114 ceph-mon[30820]: pgmap v13437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:09 smithi116 ceph-mon[35411]: pgmap v13437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:11 smithi114 ceph-mon[30820]: pgmap v13438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:11 smithi116 ceph-mon[35411]: pgmap v13438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:13 smithi114 ceph-mon[30820]: pgmap v13439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:13 smithi116 ceph-mon[35411]: pgmap v13439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:15 smithi114 ceph-mon[30820]: pgmap v13440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:15 smithi116 ceph-mon[35411]: pgmap v13440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:17 smithi114 ceph-mon[30820]: pgmap v13441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:17 smithi116 ceph-mon[35411]: pgmap v13441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:19 smithi114 ceph-mon[30820]: pgmap v13442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:19 smithi116 ceph-mon[35411]: pgmap v13442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:21 smithi114 ceph-mon[30820]: pgmap v13443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:21 smithi116 ceph-mon[35411]: pgmap v13443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:23 smithi114 ceph-mon[30820]: pgmap v13444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:23 smithi116 ceph-mon[35411]: pgmap v13444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:25 smithi114 ceph-mon[30820]: pgmap v13445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:25 smithi116 ceph-mon[35411]: pgmap v13445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:27 smithi114 ceph-mon[30820]: pgmap v13446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:27 smithi116 ceph-mon[35411]: pgmap v13446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:29 smithi114 ceph-mon[30820]: pgmap v13447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:29 smithi116 ceph-mon[35411]: pgmap v13447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:31 smithi114 ceph-mon[30820]: pgmap v13448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:31 smithi116 ceph-mon[35411]: pgmap v13448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:33 smithi114 ceph-mon[30820]: pgmap v13449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:33 smithi116 ceph-mon[35411]: pgmap v13449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:35 smithi114 ceph-mon[30820]: pgmap v13450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:35 smithi116 ceph-mon[35411]: pgmap v13450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:37 smithi114 ceph-mon[30820]: pgmap v13451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:37 smithi116 ceph-mon[35411]: pgmap v13451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:39 smithi114 ceph-mon[30820]: pgmap v13452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:39 smithi116 ceph-mon[35411]: pgmap v13452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:41 smithi114 ceph-mon[30820]: pgmap v13453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:41 smithi116 ceph-mon[35411]: pgmap v13453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:43 smithi114 ceph-mon[30820]: pgmap v13454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:43 smithi116 ceph-mon[35411]: pgmap v13454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:45 smithi114 ceph-mon[30820]: pgmap v13455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:45 smithi116 ceph-mon[35411]: pgmap v13455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:47 smithi114 ceph-mon[30820]: pgmap v13456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:47 smithi116 ceph-mon[35411]: pgmap v13456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:49 smithi114 ceph-mon[30820]: pgmap v13457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:49 smithi116 ceph-mon[35411]: pgmap v13457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:51 smithi114 ceph-mon[30820]: pgmap v13458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:51 smithi116 ceph-mon[35411]: pgmap v13458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:49:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:53 smithi116 ceph-mon[35411]: pgmap v13459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:53 smithi114 ceph-mon[30820]: pgmap v13459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:49:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:49:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:54 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:49:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:55.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:54 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: pgmap v13460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:49:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:49:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:49:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:49:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:56.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: pgmap v13460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:49:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T13:49:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T13:49:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:49:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:49:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T13:49:57.335 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:56 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:49:57.335 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:56 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:56 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T13:49:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:56 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T13:49:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:49:57.332+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:49:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:49:57.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:49:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:49:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:49:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:49:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:49:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:49:57.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:49:57.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:49:57.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:49:57.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:49:57.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:49:57.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:49:57.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:49:57.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:49:57.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:49:57.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:49:57.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:49:57.671 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:49:57.672 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:49:57.672 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:49:57.672 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:49:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-mon[30820]: pgmap v13461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:49:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:49:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:49:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:57 smithi116 ceph-mon[35411]: pgmap v13461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:49:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:49:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:49:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:49:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:58 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:49:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:49:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:58 smithi114 ceph-mon[30820]: pgmap v13462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:49:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:58 smithi114 ceph-mon[30820]: pgmap v13463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:58 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:49:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:49:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:49:59.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:49:59.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:58 smithi116 ceph-mon[35411]: pgmap v13462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:49:59.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:58 smithi116 ceph-mon[35411]: pgmap v13463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:50:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:49:59 smithi114 ceph-mon[30820]: pgmap v13464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:50:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:49:59 smithi116 ceph-mon[35411]: pgmap v13464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:50:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:50:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T13:50:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:01 smithi114 ceph-mon[30820]: pgmap v13465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:50:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:01 smithi116 ceph-mon[35411]: pgmap v13465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:50:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:04 smithi114 ceph-mon[30820]: pgmap v13466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:50:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:04 smithi116 ceph-mon[35411]: pgmap v13466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:50:07.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:06 smithi114 ceph-mon[30820]: pgmap v13467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:50:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:06 smithi116 ceph-mon[35411]: pgmap v13467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:50:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:08 smithi114 ceph-mon[30820]: pgmap v13468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:50:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:08 smithi116 ceph-mon[35411]: pgmap v13468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:50:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:10 smithi114 ceph-mon[30820]: pgmap v13469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:10 smithi116 ceph-mon[35411]: pgmap v13469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:12 smithi114 ceph-mon[30820]: pgmap v13470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:12 smithi116 ceph-mon[35411]: pgmap v13470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:14 smithi114 ceph-mon[30820]: pgmap v13471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:14 smithi116 ceph-mon[35411]: pgmap v13471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:16 smithi114 ceph-mon[30820]: pgmap v13472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:50:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:16 smithi116 ceph-mon[35411]: pgmap v13472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:50:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:18 smithi114 ceph-mon[30820]: pgmap v13473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:18 smithi116 ceph-mon[35411]: pgmap v13473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:20 smithi114 ceph-mon[30820]: pgmap v13474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:20 smithi116 ceph-mon[35411]: pgmap v13474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:22 smithi114 ceph-mon[30820]: pgmap v13475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:22 smithi116 ceph-mon[35411]: pgmap v13475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:24 smithi114 ceph-mon[30820]: pgmap v13476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:24 smithi116 ceph-mon[35411]: pgmap v13476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:26 smithi114 ceph-mon[30820]: pgmap v13477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:50:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:26 smithi116 ceph-mon[35411]: pgmap v13477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:50:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:28 smithi114 ceph-mon[30820]: pgmap v13478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:28 smithi116 ceph-mon[35411]: pgmap v13478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:30 smithi114 ceph-mon[30820]: pgmap v13479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:30 smithi116 ceph-mon[35411]: pgmap v13479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:32 smithi114 ceph-mon[30820]: pgmap v13480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:32 smithi116 ceph-mon[35411]: pgmap v13480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:34 smithi114 ceph-mon[30820]: pgmap v13481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:34 smithi116 ceph-mon[35411]: pgmap v13481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:36 smithi114 ceph-mon[30820]: pgmap v13482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:50:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:36 smithi116 ceph-mon[35411]: pgmap v13482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:50:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:38 smithi114 ceph-mon[30820]: pgmap v13483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:38 smithi116 ceph-mon[35411]: pgmap v13483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:40 smithi114 ceph-mon[30820]: pgmap v13484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:40 smithi116 ceph-mon[35411]: pgmap v13484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:42 smithi114 ceph-mon[30820]: pgmap v13485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:42 smithi116 ceph-mon[35411]: pgmap v13485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:44 smithi114 ceph-mon[30820]: pgmap v13486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:44 smithi116 ceph-mon[35411]: pgmap v13486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:46 smithi114 ceph-mon[30820]: pgmap v13487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:50:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:46 smithi116 ceph-mon[35411]: pgmap v13487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T13:50:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:48 smithi116 ceph-mon[35411]: pgmap v13488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:48 smithi114 ceph-mon[30820]: pgmap v13488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:50 smithi116 ceph-mon[35411]: pgmap v13489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:50 smithi114 ceph-mon[30820]: pgmap v13489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:50:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:52 smithi116 ceph-mon[35411]: pgmap v13490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:52 smithi114 ceph-mon[30820]: pgmap v13490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:50:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:54 smithi114 ceph-mon[30820]: pgmap v13491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:54 smithi116 ceph-mon[35411]: pgmap v13491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:50:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:56 smithi114 ceph-mon[30820]: pgmap v13492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:50:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:56 smithi116 ceph-mon[35411]: pgmap v13492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:50:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:50:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:50:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:58 smithi114 ceph-mon[30820]: pgmap v13493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:50:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:50:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:50:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:58 smithi116 ceph-mon[35411]: pgmap v13493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T13:50:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:50:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:50:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:51:00.948 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:51:00.693+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:51:00.949 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:51:00.950 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:51:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-mon[30820]: pgmap v13494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:51:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:51:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:51:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:00 smithi116 ceph-mon[35411]: pgmap v13494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:51:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:51:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:51:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:01 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:51:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:51:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:51:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:51:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:51:02.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:51:02.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:51:02.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:51:02.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:51:02.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:51:02.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:01 smithi114 ceph-mon[30820]: pgmap v13495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:51:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:01 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:51:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:51:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:51:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:51:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:51:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:51:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:51:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:51:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:51:02.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:01 smithi116 ceph-mon[35411]: pgmap v13495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:51:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:02 smithi114 ceph-mon[30820]: pgmap v13496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:51:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:02 smithi116 ceph-mon[35411]: pgmap v13496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:51:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:05 smithi114 ceph-mon[30820]: pgmap v13497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:51:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:05 smithi116 ceph-mon[35411]: pgmap v13497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:51:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:07 smithi114 ceph-mon[30820]: pgmap v13498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T13:51:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:07 smithi116 ceph-mon[35411]: pgmap v13498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 5 op/s 2024-09-19T13:51:10.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:09 smithi114 ceph-mon[30820]: pgmap v13499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:51:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:09 smithi116 ceph-mon[35411]: pgmap v13499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:51:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:11 smithi114 ceph-mon[30820]: pgmap v13500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:51:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:11 smithi116 ceph-mon[35411]: pgmap v13500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:51:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:13 smithi114 ceph-mon[30820]: pgmap v13501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:13 smithi116 ceph-mon[35411]: pgmap v13501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:15 smithi114 ceph-mon[30820]: pgmap v13502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:15 smithi116 ceph-mon[35411]: pgmap v13502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:17 smithi114 ceph-mon[30820]: pgmap v13503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:17 smithi116 ceph-mon[35411]: pgmap v13503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:19 smithi114 ceph-mon[30820]: pgmap v13504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:19 smithi116 ceph-mon[35411]: pgmap v13504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:21 smithi114 ceph-mon[30820]: pgmap v13505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:51:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:21 smithi116 ceph-mon[35411]: pgmap v13505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:51:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:23 smithi114 ceph-mon[30820]: pgmap v13506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:23 smithi116 ceph-mon[35411]: pgmap v13506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:25 smithi114 ceph-mon[30820]: pgmap v13507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:25 smithi116 ceph-mon[35411]: pgmap v13507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:27 smithi114 ceph-mon[30820]: pgmap v13508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:27 smithi116 ceph-mon[35411]: pgmap v13508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:29 smithi114 ceph-mon[30820]: pgmap v13509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:29 smithi116 ceph-mon[35411]: pgmap v13509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:31 smithi114 ceph-mon[30820]: pgmap v13510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:51:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:31 smithi116 ceph-mon[35411]: pgmap v13510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:51:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:33 smithi114 ceph-mon[30820]: pgmap v13511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:33 smithi116 ceph-mon[35411]: pgmap v13511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:35 smithi114 ceph-mon[30820]: pgmap v13512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:35 smithi116 ceph-mon[35411]: pgmap v13512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:37 smithi114 ceph-mon[30820]: pgmap v13513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:37 smithi116 ceph-mon[35411]: pgmap v13513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:39 smithi114 ceph-mon[30820]: pgmap v13514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:39 smithi116 ceph-mon[35411]: pgmap v13514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:41 smithi114 ceph-mon[30820]: pgmap v13515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:51:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:41 smithi116 ceph-mon[35411]: pgmap v13515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:51:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:43 smithi114 ceph-mon[30820]: pgmap v13516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:43 smithi116 ceph-mon[35411]: pgmap v13516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:51:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:45 smithi114 ceph-mon[30820]: pgmap v13517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:45 smithi116 ceph-mon[35411]: pgmap v13517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:51:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:47 smithi114 ceph-mon[30820]: pgmap v13518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:47 smithi116 ceph-mon[35411]: pgmap v13518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:49 smithi114 ceph-mon[30820]: pgmap v13519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:51:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:49 smithi116 ceph-mon[35411]: pgmap v13519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:51:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:51 smithi114 ceph-mon[30820]: pgmap v13520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:51:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:51 smithi116 ceph-mon[35411]: pgmap v13520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:51:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:53 smithi114 ceph-mon[30820]: pgmap v13521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:51:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:53 smithi116 ceph-mon[35411]: pgmap v13521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:51:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:55 smithi114 ceph-mon[30820]: pgmap v13522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:51:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:55 smithi116 ceph-mon[35411]: pgmap v13522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:51:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:57 smithi114 ceph-mon[30820]: pgmap v13523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:51:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:57 smithi116 ceph-mon[35411]: pgmap v13523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:51:59 smithi114 ceph-mon[30820]: pgmap v13524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:51:59 smithi116 ceph-mon[35411]: pgmap v13524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:00 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:52:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:00 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:52:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:01 smithi114 ceph-mon[30820]: pgmap v13525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:01 smithi116 ceph-mon[35411]: pgmap v13525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:03 smithi114 ceph-mon[30820]: pgmap v13526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:04.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:52:04.053+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:52:04.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:52:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:52:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:52:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:52:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:52:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:52:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:52:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:52:04.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:52:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:03 smithi116 ceph-mon[35411]: pgmap v13526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:52:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:52:05.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:52:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:52:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:05 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:52:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:52:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:52:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:52:06.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:52:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:52:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:52:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:52:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:52:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:52:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:05 smithi114 ceph-mon[30820]: pgmap v13527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:52:06.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:05 smithi114 ceph-mon[30820]: pgmap v13528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:05 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:52:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:52:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:52:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:52:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:52:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:05 smithi116 ceph-mon[35411]: pgmap v13527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:52:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:05 smithi116 ceph-mon[35411]: pgmap v13528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:52:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:07 smithi114 ceph-mon[30820]: pgmap v13529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:52:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:07 smithi116 ceph-mon[35411]: pgmap v13529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:52:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:09 smithi114 ceph-mon[30820]: pgmap v13530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:52:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:09 smithi116 ceph-mon[35411]: pgmap v13530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:52:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:11 smithi114 ceph-mon[30820]: pgmap v13531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:52:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:11 smithi116 ceph-mon[35411]: pgmap v13531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:52:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:13 smithi114 ceph-mon[30820]: pgmap v13532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:52:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:13 smithi116 ceph-mon[35411]: pgmap v13532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:52:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:15 smithi114 ceph-mon[30820]: pgmap v13533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:52:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:15 smithi116 ceph-mon[35411]: pgmap v13533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:52:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:17 smithi114 ceph-mon[30820]: pgmap v13534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:17 smithi116 ceph-mon[35411]: pgmap v13534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:19 smithi116 ceph-mon[35411]: pgmap v13535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:19 smithi114 ceph-mon[30820]: pgmap v13535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:22.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:21 smithi116 ceph-mon[35411]: pgmap v13536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:52:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:21 smithi114 ceph-mon[30820]: pgmap v13536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:52:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:23 smithi116 ceph-mon[35411]: pgmap v13537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:23 smithi114 ceph-mon[30820]: pgmap v13537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:25 smithi114 ceph-mon[30820]: pgmap v13538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:25 smithi116 ceph-mon[35411]: pgmap v13538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:27 smithi114 ceph-mon[30820]: pgmap v13539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:27 smithi116 ceph-mon[35411]: pgmap v13539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:29 smithi114 ceph-mon[30820]: pgmap v13540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:29 smithi116 ceph-mon[35411]: pgmap v13540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:32.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:31 smithi114 ceph-mon[30820]: pgmap v13541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:52:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:31 smithi116 ceph-mon[35411]: pgmap v13541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:52:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:33 smithi114 ceph-mon[30820]: pgmap v13542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:33 smithi116 ceph-mon[35411]: pgmap v13542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:35 smithi114 ceph-mon[30820]: pgmap v13543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:35 smithi116 ceph-mon[35411]: pgmap v13543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:37 smithi114 ceph-mon[30820]: pgmap v13544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:37 smithi116 ceph-mon[35411]: pgmap v13544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:39 smithi114 ceph-mon[30820]: pgmap v13545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:39 smithi116 ceph-mon[35411]: pgmap v13545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:41 smithi114 ceph-mon[30820]: pgmap v13546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:52:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:41 smithi116 ceph-mon[35411]: pgmap v13546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:52:44.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:43 smithi114 ceph-mon[30820]: pgmap v13547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:43 smithi116 ceph-mon[35411]: pgmap v13547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:44 smithi114 ceph-mon[30820]: pgmap v13548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:44 smithi116 ceph-mon[35411]: pgmap v13548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:47 smithi114 ceph-mon[30820]: pgmap v13549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:47 smithi116 ceph-mon[35411]: pgmap v13549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:49 smithi114 ceph-mon[30820]: pgmap v13550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:52:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:49 smithi116 ceph-mon[35411]: pgmap v13550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:52:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:51 smithi114 ceph-mon[30820]: pgmap v13551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:52:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:51 smithi116 ceph-mon[35411]: pgmap v13551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:52:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:53 smithi114 ceph-mon[30820]: pgmap v13552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:53 smithi116 ceph-mon[35411]: pgmap v13552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:52:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:55 smithi114 ceph-mon[30820]: pgmap v13553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:52:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:55 smithi116 ceph-mon[35411]: pgmap v13553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:52:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:57 smithi114 ceph-mon[30820]: pgmap v13554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:57 smithi116 ceph-mon[35411]: pgmap v13554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:52:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:52:59 smithi114 ceph-mon[30820]: pgmap v13555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:52:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:52:59 smithi116 ceph-mon[35411]: pgmap v13555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:01 smithi114 ceph-mon[30820]: pgmap v13556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:01 smithi116 ceph-mon[35411]: pgmap v13556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:03 smithi114 ceph-mon[30820]: pgmap v13557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:03 smithi116 ceph-mon[35411]: pgmap v13557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:05 smithi114 ceph-mon[30820]: pgmap v13558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:53:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:05 smithi116 ceph-mon[35411]: pgmap v13558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:53:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:06 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:06.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:06 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:07.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:53:07.412+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:53:07.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:53:07.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:53:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-mon[30820]: pgmap v13559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:53:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:53:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:07 smithi116 ceph-mon[35411]: pgmap v13559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:53:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:53:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:53:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:08 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:53:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:53:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:53:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:53:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:53:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:53:09.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:53:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:53:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:53:09.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:08 smithi114 ceph-mon[30820]: pgmap v13560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:53:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:08 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:53:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:53:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:53:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:53:09.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:53:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:53:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:53:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:53:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:53:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:53:09.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:08 smithi116 ceph-mon[35411]: pgmap v13560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:53:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:10 smithi114 ceph-mon[30820]: pgmap v13561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:53:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:10 smithi116 ceph-mon[35411]: pgmap v13561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:53:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:12 smithi114 ceph-mon[30820]: pgmap v13562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:53:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:12 smithi116 ceph-mon[35411]: pgmap v13562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:53:14.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:14 smithi114 ceph-mon[30820]: pgmap v13563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:53:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:14 smithi116 ceph-mon[35411]: pgmap v13563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:53:17.105 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:16 smithi114 ceph-mon[30820]: pgmap v13564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T13:53:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:16 smithi116 ceph-mon[35411]: pgmap v13564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 7 op/s 2024-09-19T13:53:19.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:18 smithi114 ceph-mon[30820]: pgmap v13565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:53:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:18 smithi116 ceph-mon[35411]: pgmap v13565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:53:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:20 smithi114 ceph-mon[30820]: pgmap v13566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:20 smithi116 ceph-mon[35411]: pgmap v13566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:22.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:22 smithi116 ceph-mon[35411]: pgmap v13567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:22 smithi114 ceph-mon[30820]: pgmap v13567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:24 smithi114 ceph-mon[30820]: pgmap v13568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:24 smithi116 ceph-mon[35411]: pgmap v13568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:26 smithi114 ceph-mon[30820]: pgmap v13569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:26 smithi116 ceph-mon[35411]: pgmap v13569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:28 smithi114 ceph-mon[30820]: pgmap v13570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:28 smithi116 ceph-mon[35411]: pgmap v13570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:30 smithi114 ceph-mon[30820]: pgmap v13571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:30 smithi116 ceph-mon[35411]: pgmap v13571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:32 smithi114 ceph-mon[30820]: pgmap v13572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:32 smithi116 ceph-mon[35411]: pgmap v13572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:34 smithi114 ceph-mon[30820]: pgmap v13573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:34 smithi116 ceph-mon[35411]: pgmap v13573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:36 smithi114 ceph-mon[30820]: pgmap v13574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:36 smithi116 ceph-mon[35411]: pgmap v13574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:38 smithi114 ceph-mon[30820]: pgmap v13575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:38 smithi116 ceph-mon[35411]: pgmap v13575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:40 smithi114 ceph-mon[30820]: pgmap v13576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:40 smithi116 ceph-mon[35411]: pgmap v13576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:42 smithi114 ceph-mon[30820]: pgmap v13577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:42 smithi116 ceph-mon[35411]: pgmap v13577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:44 smithi114 ceph-mon[30820]: pgmap v13578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:44 smithi116 ceph-mon[35411]: pgmap v13578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:46 smithi114 ceph-mon[30820]: pgmap v13579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:46 smithi116 ceph-mon[35411]: pgmap v13579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:48 smithi114 ceph-mon[30820]: pgmap v13580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:48 smithi116 ceph-mon[35411]: pgmap v13580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:50 smithi114 ceph-mon[30820]: pgmap v13581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:50 smithi116 ceph-mon[35411]: pgmap v13581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:53:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:52 smithi114 ceph-mon[30820]: pgmap v13582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:52 smithi116 ceph-mon[35411]: pgmap v13582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:53:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:54 smithi114 ceph-mon[30820]: pgmap v13583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:54 smithi116 ceph-mon[35411]: pgmap v13583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:56 smithi114 ceph-mon[30820]: pgmap v13584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:56 smithi116 ceph-mon[35411]: pgmap v13584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:53:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:53:58 smithi114 ceph-mon[30820]: pgmap v13585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:53:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:53:58 smithi116 ceph-mon[35411]: pgmap v13585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:00 smithi114 ceph-mon[30820]: pgmap v13586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:00 smithi116 ceph-mon[35411]: pgmap v13586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:02 smithi114 ceph-mon[30820]: pgmap v13587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:02 smithi116 ceph-mon[35411]: pgmap v13587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:04 smithi114 ceph-mon[30820]: pgmap v13588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:04 smithi116 ceph-mon[35411]: pgmap v13588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:06 smithi114 ceph-mon[30820]: pgmap v13589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:54:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:06 smithi116 ceph-mon[35411]: pgmap v13589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:54:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:07 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:54:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:07 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:54:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:08 smithi114 ceph-mon[30820]: pgmap v13590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:08 smithi116 ceph-mon[35411]: pgmap v13590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-mon[30820]: pgmap v13591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:54:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:54:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:11.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:54:10.708+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:54:11.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:54:11.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:54:11.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:54:11.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:54:11.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:54:11.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:54:11.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:54:11.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:54:11.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:54:11.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:54:11.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:54:10 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:54:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:10 smithi116 ceph-mon[35411]: pgmap v13591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:54:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:54:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:54:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:11 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:54:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:54:12.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:11 smithi114 ceph-mon[30820]: pgmap v13592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:54:12.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:11 smithi114 ceph-mon[30820]: pgmap v13593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:11 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:54:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:54:12.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:11 smithi116 ceph-mon[35411]: pgmap v13592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T13:54:12.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:11 smithi116 ceph-mon[35411]: pgmap v13593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:54:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:13 smithi114 ceph-mon[30820]: pgmap v13594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:54:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:13 smithi116 ceph-mon[35411]: pgmap v13594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:54:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:15 smithi114 ceph-mon[30820]: pgmap v13595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:54:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:15 smithi116 ceph-mon[35411]: pgmap v13595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:54:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:17 smithi114 ceph-mon[30820]: pgmap v13596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:54:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:17 smithi116 ceph-mon[35411]: pgmap v13596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:54:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:19 smithi114 ceph-mon[30820]: pgmap v13597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:54:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:19 smithi116 ceph-mon[35411]: pgmap v13597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:54:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:21 smithi114 ceph-mon[30820]: pgmap v13598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:54:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:21 smithi116 ceph-mon[35411]: pgmap v13598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:54:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:23 smithi114 ceph-mon[30820]: pgmap v13599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:23 smithi116 ceph-mon[35411]: pgmap v13599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:25 smithi114 ceph-mon[30820]: pgmap v13600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:25 smithi116 ceph-mon[35411]: pgmap v13600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:27 smithi114 ceph-mon[30820]: pgmap v13601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:27 smithi116 ceph-mon[35411]: pgmap v13601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:29 smithi114 ceph-mon[30820]: pgmap v13602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:29 smithi116 ceph-mon[35411]: pgmap v13602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:31 smithi114 ceph-mon[30820]: pgmap v13603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:54:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:31 smithi116 ceph-mon[35411]: pgmap v13603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:54:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:33 smithi114 ceph-mon[30820]: pgmap v13604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:33 smithi116 ceph-mon[35411]: pgmap v13604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:35 smithi114 ceph-mon[30820]: pgmap v13605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:35 smithi116 ceph-mon[35411]: pgmap v13605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:37 smithi114 ceph-mon[30820]: pgmap v13606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:37 smithi116 ceph-mon[35411]: pgmap v13606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:39 smithi114 ceph-mon[30820]: pgmap v13607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:39 smithi116 ceph-mon[35411]: pgmap v13607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:41 smithi114 ceph-mon[30820]: pgmap v13608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:54:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:41 smithi116 ceph-mon[35411]: pgmap v13608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:54:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:43 smithi116 ceph-mon[35411]: pgmap v13609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:43 smithi114 ceph-mon[30820]: pgmap v13609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:45 smithi116 ceph-mon[35411]: pgmap v13610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:46.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:45 smithi114 ceph-mon[30820]: pgmap v13610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:47 smithi116 ceph-mon[35411]: pgmap v13611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:47 smithi114 ceph-mon[30820]: pgmap v13611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:49 smithi114 ceph-mon[30820]: pgmap v13612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:49 smithi116 ceph-mon[35411]: pgmap v13612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:51 smithi114 ceph-mon[30820]: pgmap v13613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:54:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:51 smithi116 ceph-mon[35411]: pgmap v13613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:54:54.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:53 smithi116 ceph-mon[35411]: pgmap v13614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:53 smithi114 ceph-mon[30820]: pgmap v13614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:54:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:55 smithi114 ceph-mon[30820]: pgmap v13615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:55 smithi116 ceph-mon[35411]: pgmap v13615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:54:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:57 smithi114 ceph-mon[30820]: pgmap v13616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:54:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:57 smithi116 ceph-mon[35411]: pgmap v13616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:54:59 smithi114 ceph-mon[30820]: pgmap v13617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:54:59 smithi116 ceph-mon[35411]: pgmap v13617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:01 smithi114 ceph-mon[30820]: pgmap v13618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:01 smithi116 ceph-mon[35411]: pgmap v13618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:03 smithi114 ceph-mon[30820]: pgmap v13619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:55:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:03 smithi116 ceph-mon[35411]: pgmap v13619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:55:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:05 smithi114 ceph-mon[30820]: pgmap v13620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:55:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:05 smithi116 ceph-mon[35411]: pgmap v13620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T13:55:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:07 smithi114 ceph-mon[30820]: pgmap v13621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:07 smithi116 ceph-mon[35411]: pgmap v13621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:09 smithi114 ceph-mon[30820]: pgmap v13622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:55:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:09 smithi116 ceph-mon[35411]: pgmap v13622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T13:55:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:10 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:55:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:10 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:55:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:11 smithi114 ceph-mon[30820]: pgmap v13623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:11 smithi116 ceph-mon[35411]: pgmap v13623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:13 smithi114 ceph-mon[30820]: pgmap v13624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:55:14.068+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:55:14.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:55:14.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:55:14.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:55:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:13 smithi116 ceph-mon[35411]: pgmap v13624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:55:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:55:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:55:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:55:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:55:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:55:16.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:15 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:55:16.319 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:55:16.319 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:55:16.320 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:15 smithi114 ceph-mon[30820]: pgmap v13625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:55:16.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:15 smithi114 ceph-mon[30820]: pgmap v13626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:55:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:15 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:55:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:55:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:55:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:55:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:55:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:55:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:55:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:55:16.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:55:16.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:15 smithi116 ceph-mon[35411]: pgmap v13625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:55:16.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:15 smithi116 ceph-mon[35411]: pgmap v13626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:55:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:17 smithi114 ceph-mon[30820]: pgmap v13627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:55:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:17 smithi116 ceph-mon[35411]: pgmap v13627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:55:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:19 smithi114 ceph-mon[30820]: pgmap v13628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:55:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:19 smithi116 ceph-mon[35411]: pgmap v13628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:55:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:21 smithi114 ceph-mon[30820]: pgmap v13629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:55:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:21 smithi116 ceph-mon[35411]: pgmap v13629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:55:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:23 smithi114 ceph-mon[30820]: pgmap v13630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:55:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:23 smithi116 ceph-mon[35411]: pgmap v13630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:55:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:25 smithi114 ceph-mon[30820]: pgmap v13631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:55:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:25 smithi116 ceph-mon[35411]: pgmap v13631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:55:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:27 smithi114 ceph-mon[30820]: pgmap v13632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:27 smithi116 ceph-mon[35411]: pgmap v13632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:29 smithi114 ceph-mon[30820]: pgmap v13633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:29 smithi116 ceph-mon[35411]: pgmap v13633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:31 smithi114 ceph-mon[30820]: pgmap v13634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:55:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:31 smithi116 ceph-mon[35411]: pgmap v13634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:55:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:33 smithi114 ceph-mon[30820]: pgmap v13635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:33 smithi116 ceph-mon[35411]: pgmap v13635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:35 smithi114 ceph-mon[30820]: pgmap v13636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:35 smithi116 ceph-mon[35411]: pgmap v13636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:37 smithi114 ceph-mon[30820]: pgmap v13637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:37 smithi116 ceph-mon[35411]: pgmap v13637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:39 smithi114 ceph-mon[30820]: pgmap v13638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:39 smithi116 ceph-mon[35411]: pgmap v13638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:41 smithi116 ceph-mon[35411]: pgmap v13639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:55:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:41 smithi114 ceph-mon[30820]: pgmap v13639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:55:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:43 smithi116 ceph-mon[35411]: pgmap v13640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:43 smithi114 ceph-mon[30820]: pgmap v13640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:45 smithi116 ceph-mon[35411]: pgmap v13641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:45 smithi114 ceph-mon[30820]: pgmap v13641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:47.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:47 smithi114 ceph-mon[30820]: pgmap v13642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:47 smithi116 ceph-mon[35411]: pgmap v13642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:49 smithi114 ceph-mon[30820]: pgmap v13643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:49 smithi116 ceph-mon[35411]: pgmap v13643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:51 smithi114 ceph-mon[30820]: pgmap v13644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:55:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:51 smithi116 ceph-mon[35411]: pgmap v13644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:55:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:53 smithi114 ceph-mon[30820]: pgmap v13645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:53.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:53 smithi116 ceph-mon[35411]: pgmap v13645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:55:55.481 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:55 smithi116 ceph-mon[35411]: pgmap v13646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:55 smithi114 ceph-mon[30820]: pgmap v13646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:57 smithi114 ceph-mon[30820]: pgmap v13647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:57 smithi116 ceph-mon[35411]: pgmap v13647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:55:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:55:59 smithi114 ceph-mon[30820]: pgmap v13648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:55:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:55:59 smithi116 ceph-mon[35411]: pgmap v13648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:01 smithi114 ceph-mon[30820]: pgmap v13649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:56:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:01 smithi116 ceph-mon[35411]: pgmap v13649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:56:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:03 smithi114 ceph-mon[30820]: pgmap v13650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:03 smithi116 ceph-mon[35411]: pgmap v13650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:05 smithi114 ceph-mon[30820]: pgmap v13651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:05 smithi116 ceph-mon[35411]: pgmap v13651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:07 smithi114 ceph-mon[30820]: pgmap v13652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:07 smithi116 ceph-mon[35411]: pgmap v13652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:09 smithi114 ceph-mon[30820]: pgmap v13653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:09 smithi116 ceph-mon[35411]: pgmap v13653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:11 smithi114 ceph-mon[30820]: pgmap v13654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:56:11.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:11 smithi116 ceph-mon[35411]: pgmap v13654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:56:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:13 smithi114 ceph-mon[30820]: pgmap v13655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:13 smithi116 ceph-mon[35411]: pgmap v13655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:56:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:56:15.522 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:15 smithi114 ceph-mon[30820]: pgmap v13656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:15 smithi116 ceph-mon[35411]: pgmap v13656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:17.362 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-mon[30820]: pgmap v13657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:17.517 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:17 smithi116 ceph-mon[35411]: pgmap v13657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:17.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:56:17.428+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:56:17.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:56:17.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:56:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:18 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:56:19.267 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:19.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:18 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:56:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:56:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:56:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:56:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:56:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:56:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:56:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:56:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:56:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:20.230 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:19 smithi114 ceph-mon[30820]: pgmap v13658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:20.230 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:20.230 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:20.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:56:20.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:56:20.231 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:19 smithi116 ceph-mon[35411]: pgmap v13658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:56:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:56:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:56:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:20 smithi114 ceph-mon[30820]: pgmap v13659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:56:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:20 smithi114 ceph-mon[30820]: pgmap v13660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:20 smithi116 ceph-mon[35411]: pgmap v13659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T13:56:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:20 smithi116 ceph-mon[35411]: pgmap v13660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:22 smithi114 ceph-mon[30820]: pgmap v13661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:22 smithi116 ceph-mon[35411]: pgmap v13661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:25.232 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:24 smithi116 ceph-mon[35411]: pgmap v13662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:24 smithi114 ceph-mon[30820]: pgmap v13662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:27.421 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:26 smithi114 ceph-mon[30820]: pgmap v13663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:26 smithi116 ceph-mon[35411]: pgmap v13663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:28 smithi114 ceph-mon[30820]: pgmap v13664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:28 smithi116 ceph-mon[35411]: pgmap v13664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T13:56:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:30 smithi114 ceph-mon[30820]: pgmap v13665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:56:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:30 smithi116 ceph-mon[35411]: pgmap v13665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:56:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:32 smithi114 ceph-mon[30820]: pgmap v13666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:32 smithi116 ceph-mon[35411]: pgmap v13666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:34 smithi114 ceph-mon[30820]: pgmap v13667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:34 smithi116 ceph-mon[35411]: pgmap v13667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:36 smithi114 ceph-mon[30820]: pgmap v13668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:36 smithi116 ceph-mon[35411]: pgmap v13668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:38 smithi114 ceph-mon[30820]: pgmap v13669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:38 smithi116 ceph-mon[35411]: pgmap v13669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:40 smithi114 ceph-mon[30820]: pgmap v13670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:56:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:40 smithi116 ceph-mon[35411]: pgmap v13670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:56:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:42 smithi114 ceph-mon[30820]: pgmap v13671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:42 smithi116 ceph-mon[35411]: pgmap v13671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:44 smithi114 ceph-mon[30820]: pgmap v13672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:44 smithi116 ceph-mon[35411]: pgmap v13672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:46 smithi114 ceph-mon[30820]: pgmap v13673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:46 smithi116 ceph-mon[35411]: pgmap v13673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:48 smithi114 ceph-mon[30820]: pgmap v13674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:48 smithi116 ceph-mon[35411]: pgmap v13674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:51 smithi114 ceph-mon[30820]: pgmap v13675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:56:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:51 smithi116 ceph-mon[35411]: pgmap v13675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:56:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:53 smithi114 ceph-mon[30820]: pgmap v13676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:53 smithi116 ceph-mon[35411]: pgmap v13676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:56:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:54 smithi114 ceph-mon[30820]: pgmap v13677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:54 smithi116 ceph-mon[35411]: pgmap v13677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:57.132 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:56 smithi116 ceph-mon[35411]: pgmap v13678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:56 smithi114 ceph-mon[30820]: pgmap v13678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:56:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:56:58 smithi114 ceph-mon[30820]: pgmap v13679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:56:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:56:58 smithi116 ceph-mon[35411]: pgmap v13679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:00 smithi114 ceph-mon[30820]: pgmap v13680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:00 smithi116 ceph-mon[35411]: pgmap v13680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:02 smithi114 ceph-mon[30820]: pgmap v13681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:02 smithi116 ceph-mon[35411]: pgmap v13681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:04 smithi114 ceph-mon[30820]: pgmap v13682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:04 smithi116 ceph-mon[35411]: pgmap v13682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:06 smithi116 ceph-mon[35411]: pgmap v13683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:57:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:06 smithi114 ceph-mon[30820]: pgmap v13683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:57:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:08 smithi114 ceph-mon[30820]: pgmap v13684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:08 smithi116 ceph-mon[35411]: pgmap v13684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:10 smithi114 ceph-mon[30820]: pgmap v13685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:10 smithi116 ceph-mon[35411]: pgmap v13685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:12 smithi114 ceph-mon[30820]: pgmap v13686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:12 smithi116 ceph-mon[35411]: pgmap v13686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:14 smithi114 ceph-mon[30820]: pgmap v13687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:14 smithi116 ceph-mon[35411]: pgmap v13687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:17.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:16 smithi114 ceph-mon[30820]: pgmap v13688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:57:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:16 smithi116 ceph-mon[35411]: pgmap v13688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:57:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:18 smithi114 ceph-mon[30820]: pgmap v13689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:18 smithi116 ceph-mon[35411]: pgmap v13689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:19 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:57:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:19 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:57:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-mon[30820]: pgmap v13690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:57:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:57:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:57:21.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:57:20.852+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:57:21.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:57:21.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:57:21.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:57:21.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:57:21.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:57:21.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:57:21.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:57:21.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:57:21.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:57:21.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:57:20 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:57:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:20 smithi116 ceph-mon[35411]: pgmap v13690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:57:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:57:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:57:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:21 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:57:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:57:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:57:22.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:21 smithi114 ceph-mon[30820]: pgmap v13691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 6 op/s 2024-09-19T13:57:22.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:21 smithi114 ceph-mon[30820]: pgmap v13692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:57:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:21 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:57:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:57:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:57:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:57:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:57:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:57:22.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:57:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:57:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:57:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:57:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:21 smithi116 ceph-mon[35411]: pgmap v13691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 6 op/s 2024-09-19T13:57:22.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:21 smithi116 ceph-mon[35411]: pgmap v13692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:57:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:23 smithi114 ceph-mon[30820]: pgmap v13693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:57:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:23 smithi116 ceph-mon[35411]: pgmap v13693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:57:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:25 smithi114 ceph-mon[30820]: pgmap v13694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:57:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:25 smithi116 ceph-mon[35411]: pgmap v13694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:57:28.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:27 smithi116 ceph-mon[35411]: pgmap v13695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:57:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:27 smithi114 ceph-mon[30820]: pgmap v13695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T13:57:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:29 smithi114 ceph-mon[30820]: pgmap v13696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T13:57:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:29 smithi116 ceph-mon[35411]: pgmap v13696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 4 op/s 2024-09-19T13:57:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:31 smithi114 ceph-mon[30820]: pgmap v13697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:57:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:31 smithi116 ceph-mon[35411]: pgmap v13697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:57:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:33 smithi114 ceph-mon[30820]: pgmap v13698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:33 smithi116 ceph-mon[35411]: pgmap v13698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:35 smithi114 ceph-mon[30820]: pgmap v13699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:35 smithi116 ceph-mon[35411]: pgmap v13699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:37 smithi114 ceph-mon[30820]: pgmap v13700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:37 smithi116 ceph-mon[35411]: pgmap v13700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:39 smithi114 ceph-mon[30820]: pgmap v13701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:39 smithi116 ceph-mon[35411]: pgmap v13701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:42 smithi114 ceph-mon[30820]: pgmap v13702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:57:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:42 smithi116 ceph-mon[35411]: pgmap v13702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:57:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:44 smithi114 ceph-mon[30820]: pgmap v13703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:44.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:44 smithi116 ceph-mon[35411]: pgmap v13703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:46 smithi114 ceph-mon[30820]: pgmap v13704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:46 smithi116 ceph-mon[35411]: pgmap v13704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:48 smithi114 ceph-mon[30820]: pgmap v13705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:48 smithi116 ceph-mon[35411]: pgmap v13705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:50 smithi114 ceph-mon[30820]: pgmap v13706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:50 smithi116 ceph-mon[35411]: pgmap v13706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:51 smithi114 ceph-mon[30820]: pgmap v13707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:57:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:51 smithi116 ceph-mon[35411]: pgmap v13707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:57:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:53 smithi116 ceph-mon[35411]: pgmap v13708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:53 smithi114 ceph-mon[30820]: pgmap v13708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:57:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:55 smithi116 ceph-mon[35411]: pgmap v13709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:55 smithi114 ceph-mon[30820]: pgmap v13709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:57:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:57 smithi114 ceph-mon[30820]: pgmap v13710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:57:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:57 smithi116 ceph-mon[35411]: pgmap v13710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:00.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:57:59 smithi116 ceph-mon[35411]: pgmap v13711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:57:59 smithi114 ceph-mon[30820]: pgmap v13711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:01 smithi114 ceph-mon[30820]: pgmap v13712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:02.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:01 smithi116 ceph-mon[35411]: pgmap v13712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:03 smithi114 ceph-mon[30820]: pgmap v13713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:03 smithi116 ceph-mon[35411]: pgmap v13713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:05 smithi114 ceph-mon[30820]: pgmap v13714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:58:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:05 smithi116 ceph-mon[35411]: pgmap v13714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:58:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:07 smithi114 ceph-mon[30820]: pgmap v13715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:07 smithi116 ceph-mon[35411]: pgmap v13715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:09 smithi114 ceph-mon[30820]: pgmap v13716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:09 smithi116 ceph-mon[35411]: pgmap v13716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:11 smithi114 ceph-mon[30820]: pgmap v13717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:11 smithi116 ceph-mon[35411]: pgmap v13717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:13 smithi114 ceph-mon[30820]: pgmap v13718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:13 smithi116 ceph-mon[35411]: pgmap v13718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:15 smithi114 ceph-mon[30820]: pgmap v13719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:58:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:15 smithi116 ceph-mon[35411]: pgmap v13719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:58:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:17 smithi114 ceph-mon[30820]: pgmap v13720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:17 smithi116 ceph-mon[35411]: pgmap v13720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:19 smithi114 ceph-mon[30820]: pgmap v13721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:19 smithi116 ceph-mon[35411]: pgmap v13721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:20 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:58:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:20 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:58:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:21 smithi114 ceph-mon[30820]: pgmap v13722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:21 smithi116 ceph-mon[35411]: pgmap v13722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:23 smithi114 ceph-mon[30820]: pgmap v13723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:58:24.148+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:58:24.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:58:24.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:58:24.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:58:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:23 smithi116 ceph-mon[35411]: pgmap v13723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:58:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:58:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:58:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:58:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:58:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:58:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:25 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:58:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:58:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:58:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:58:26.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:58:26.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:58:26.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:58:26.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:58:26.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:58:26.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:58:26.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:25 smithi114 ceph-mon[30820]: pgmap v13724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:58:26.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:25 smithi114 ceph-mon[30820]: pgmap v13725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:58:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:25 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:58:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:58:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:58:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:58:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:58:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:58:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:58:26.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:58:26.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:58:26.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:25 smithi116 ceph-mon[35411]: pgmap v13724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T13:58:26.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:25 smithi116 ceph-mon[35411]: pgmap v13725: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T13:58:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:27 smithi114 ceph-mon[30820]: pgmap v13726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:58:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:27 smithi116 ceph-mon[35411]: pgmap v13726: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:58:30.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:30 smithi114 ceph-mon[30820]: pgmap v13727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:58:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:30 smithi116 ceph-mon[35411]: pgmap v13727: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:58:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:32 smithi114 ceph-mon[30820]: pgmap v13728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:58:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:32 smithi116 ceph-mon[35411]: pgmap v13728: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T13:58:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:34 smithi114 ceph-mon[30820]: pgmap v13729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:58:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:34 smithi116 ceph-mon[35411]: pgmap v13729: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:58:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:36 smithi114 ceph-mon[30820]: pgmap v13730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:58:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:36 smithi116 ceph-mon[35411]: pgmap v13730: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:58:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:38 smithi114 ceph-mon[30820]: pgmap v13731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:38 smithi116 ceph-mon[35411]: pgmap v13731: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:40 smithi114 ceph-mon[30820]: pgmap v13732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:40 smithi116 ceph-mon[35411]: pgmap v13732: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:42 smithi114 ceph-mon[30820]: pgmap v13733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:58:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:42 smithi116 ceph-mon[35411]: pgmap v13733: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:58:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:43 smithi114 ceph-mon[30820]: pgmap v13734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:43 smithi116 ceph-mon[35411]: pgmap v13734: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:45 smithi114 ceph-mon[30820]: pgmap v13735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:45 smithi116 ceph-mon[35411]: pgmap v13735: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:47 smithi114 ceph-mon[30820]: pgmap v13736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:47 smithi116 ceph-mon[35411]: pgmap v13736: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:49 smithi114 ceph-mon[30820]: pgmap v13737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:49 smithi116 ceph-mon[35411]: pgmap v13737: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:51 smithi114 ceph-mon[30820]: pgmap v13738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:58:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:51 smithi116 ceph-mon[35411]: pgmap v13738: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:58:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:53 smithi114 ceph-mon[30820]: pgmap v13739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:53 smithi116 ceph-mon[35411]: pgmap v13739: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:58:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:55 smithi114 ceph-mon[30820]: pgmap v13740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:55 smithi116 ceph-mon[35411]: pgmap v13740: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:57 smithi114 ceph-mon[30820]: pgmap v13741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:57 smithi116 ceph-mon[35411]: pgmap v13741: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:58:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:58:59 smithi114 ceph-mon[30820]: pgmap v13742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:58:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:58:59 smithi116 ceph-mon[35411]: pgmap v13742: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:01 smithi114 ceph-mon[30820]: pgmap v13743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:01 smithi116 ceph-mon[35411]: pgmap v13743: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:03 smithi114 ceph-mon[30820]: pgmap v13744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:03 smithi116 ceph-mon[35411]: pgmap v13744: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:05 smithi114 ceph-mon[30820]: pgmap v13745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:05 smithi116 ceph-mon[35411]: pgmap v13745: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:07 smithi114 ceph-mon[30820]: pgmap v13746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:07 smithi116 ceph-mon[35411]: pgmap v13746: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:09 smithi114 ceph-mon[30820]: pgmap v13747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:09 smithi116 ceph-mon[35411]: pgmap v13747: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:11 smithi114 ceph-mon[30820]: pgmap v13748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:11 smithi116 ceph-mon[35411]: pgmap v13748: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:13 smithi114 ceph-mon[30820]: pgmap v13749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:13 smithi116 ceph-mon[35411]: pgmap v13749: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:15 smithi114 ceph-mon[30820]: pgmap v13750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:15 smithi116 ceph-mon[35411]: pgmap v13750: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:17 smithi114 ceph-mon[30820]: pgmap v13751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:17 smithi116 ceph-mon[35411]: pgmap v13751: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:19 smithi114 ceph-mon[30820]: pgmap v13752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:19 smithi116 ceph-mon[35411]: pgmap v13752: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:21 smithi114 ceph-mon[30820]: pgmap v13753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:21 smithi116 ceph-mon[35411]: pgmap v13753: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:23 smithi114 ceph-mon[30820]: pgmap v13754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:23 smithi116 ceph-mon[35411]: pgmap v13754: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:59:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T13:59:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:25 smithi114 ceph-mon[30820]: pgmap v13755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:25 smithi116 ceph-mon[35411]: pgmap v13755: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-mon[30820]: pgmap v13756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T13:59:27.508+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:59:27.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:59:27.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 13:59:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:59:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:27 smithi116 ceph-mon[35411]: pgmap v13756: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:59:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T13:59:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T13:59:28.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:28 smithi114 ceph-mon[30820]: pgmap v13757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:28 smithi114 ceph-mon[30820]: pgmap v13758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:28.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:59:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:59:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T13:59:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T13:59:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T13:59:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T13:59:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T13:59:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T13:59:28.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T13:59:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T13:59:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T13:59:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T13:59:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T13:59:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T13:59:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:28 smithi116 ceph-mon[35411]: pgmap v13757: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T13:59:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:28 smithi116 ceph-mon[35411]: pgmap v13758: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:28.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T13:59:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:30 smithi114 ceph-mon[30820]: pgmap v13759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:30 smithi116 ceph-mon[35411]: pgmap v13759: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:32 smithi114 ceph-mon[30820]: pgmap v13760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:32 smithi116 ceph-mon[35411]: pgmap v13760: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:34 smithi114 ceph-mon[30820]: pgmap v13761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:34 smithi116 ceph-mon[35411]: pgmap v13761: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:36 smithi114 ceph-mon[30820]: pgmap v13762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:36 smithi116 ceph-mon[35411]: pgmap v13762: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T13:59:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:38 smithi114 ceph-mon[30820]: pgmap v13763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:59:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:38 smithi116 ceph-mon[35411]: pgmap v13763: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T13:59:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:40 smithi114 ceph-mon[30820]: pgmap v13764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:40 smithi116 ceph-mon[35411]: pgmap v13764: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:42 smithi114 ceph-mon[30820]: pgmap v13765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:42 smithi116 ceph-mon[35411]: pgmap v13765: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:44 smithi114 ceph-mon[30820]: pgmap v13766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:44 smithi116 ceph-mon[35411]: pgmap v13766: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:46 smithi114 ceph-mon[30820]: pgmap v13767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:46 smithi116 ceph-mon[35411]: pgmap v13767: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:48 smithi114 ceph-mon[30820]: pgmap v13768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:48 smithi116 ceph-mon[35411]: pgmap v13768: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:50 smithi114 ceph-mon[30820]: pgmap v13769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:50 smithi116 ceph-mon[35411]: pgmap v13769: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T13:59:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:52 smithi114 ceph-mon[30820]: pgmap v13770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:52 smithi116 ceph-mon[35411]: pgmap v13770: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T13:59:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:54 smithi114 ceph-mon[30820]: pgmap v13771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:54 smithi116 ceph-mon[35411]: pgmap v13771: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:56.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:56 smithi114 ceph-mon[30820]: pgmap v13772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:56 smithi116 ceph-mon[35411]: pgmap v13772: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T13:59:58.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 13:59:58 smithi114 ceph-mon[30820]: pgmap v13773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T13:59:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 13:59:58 smithi116 ceph-mon[35411]: pgmap v13773: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:00 smithi114 ceph-mon[30820]: pgmap v13774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:00.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T14:00:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:00 smithi116 ceph-mon[35411]: pgmap v13774: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T14:00:02.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:02 smithi114 ceph-mon[30820]: pgmap v13775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:02.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:02 smithi116 ceph-mon[35411]: pgmap v13775: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:04 smithi114 ceph-mon[30820]: pgmap v13776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:04 smithi116 ceph-mon[35411]: pgmap v13776: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:06.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:06 smithi114 ceph-mon[30820]: pgmap v13777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:00:06.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:06 smithi116 ceph-mon[35411]: pgmap v13777: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:00:08.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:08 smithi114 ceph-mon[30820]: pgmap v13778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:08.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:08 smithi116 ceph-mon[35411]: pgmap v13778: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:10 smithi114 ceph-mon[30820]: pgmap v13779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:10 smithi116 ceph-mon[35411]: pgmap v13779: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:12 smithi114 ceph-mon[30820]: pgmap v13780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:12 smithi116 ceph-mon[35411]: pgmap v13780: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:14 smithi114 ceph-mon[30820]: pgmap v13781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:14 smithi116 ceph-mon[35411]: pgmap v13781: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:16 smithi114 ceph-mon[30820]: pgmap v13782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:00:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:16 smithi116 ceph-mon[35411]: pgmap v13782: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:00:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:18 smithi114 ceph-mon[30820]: pgmap v13783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:18 smithi116 ceph-mon[35411]: pgmap v13783: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:20 smithi114 ceph-mon[30820]: pgmap v13784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:20 smithi116 ceph-mon[35411]: pgmap v13784: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:22 smithi114 ceph-mon[30820]: pgmap v13785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:22 smithi116 ceph-mon[35411]: pgmap v13785: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:24 smithi114 ceph-mon[30820]: pgmap v13786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:24 smithi116 ceph-mon[35411]: pgmap v13786: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:26 smithi114 ceph-mon[30820]: pgmap v13787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:00:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:26 smithi116 ceph-mon[35411]: pgmap v13787: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:00:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:00:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:00:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: pgmap v13788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:29 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: pgmap v13788: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T14:00:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T14:00:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:29.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:29 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:00:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T14:00:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T14:00:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:30 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T14:00:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:30 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T14:00:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:31 smithi114 ceph-mon[30820]: pgmap v13789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:00:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:00:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:00:30.868+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:00:31.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:00:31.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:00:30 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:00:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:31 smithi116 ceph-mon[35411]: pgmap v13789: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:00:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:00:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:00:32.331 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:32 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:00:32.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:00:32.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:00:32.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:00:32.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:00:32.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:00:32.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:00:32.331 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:00:32.332 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:00:32.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:00:32.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:00:32.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:00:32.333 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:00:32.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:32 smithi116 ceph-mon[35411]: pgmap v13790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:00:32.333 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:32 smithi116 ceph-mon[35411]: pgmap v13791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:00:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:32 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:00:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:00:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:00:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:00:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:00:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:00:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:00:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:00:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:00:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:00:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:00:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:32 smithi114 ceph-mon[30820]: pgmap v13790: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:00:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:32 smithi114 ceph-mon[30820]: pgmap v13791: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:00:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:34 smithi114 ceph-mon[30820]: pgmap v13792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:00:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:34 smithi116 ceph-mon[35411]: pgmap v13792: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:00:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:36 smithi114 ceph-mon[30820]: pgmap v13793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:00:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:36 smithi116 ceph-mon[35411]: pgmap v13793: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:00:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:38 smithi114 ceph-mon[30820]: pgmap v13794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:00:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:38 smithi116 ceph-mon[35411]: pgmap v13794: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:00:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:40 smithi114 ceph-mon[30820]: pgmap v13795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:00:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:40 smithi116 ceph-mon[35411]: pgmap v13795: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:00:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:42 smithi114 ceph-mon[30820]: pgmap v13796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:00:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:42 smithi116 ceph-mon[35411]: pgmap v13796: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:00:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:44 smithi114 ceph-mon[30820]: pgmap v13797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:44.432 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:44 smithi116 ceph-mon[35411]: pgmap v13797: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:46 smithi114 ceph-mon[30820]: pgmap v13798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:46 smithi116 ceph-mon[35411]: pgmap v13798: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:48 smithi114 ceph-mon[30820]: pgmap v13799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:48 smithi116 ceph-mon[35411]: pgmap v13799: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:50 smithi114 ceph-mon[30820]: pgmap v13800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:50 smithi116 ceph-mon[35411]: pgmap v13800: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:51.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:51 smithi114 ceph-mon[30820]: pgmap v13801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:00:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:51 smithi116 ceph-mon[35411]: pgmap v13801: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:00:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:53 smithi114 ceph-mon[30820]: pgmap v13802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:53 smithi116 ceph-mon[35411]: pgmap v13802: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:00:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:55 smithi114 ceph-mon[30820]: pgmap v13803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:55 smithi116 ceph-mon[35411]: pgmap v13803: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:00:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:57 smithi114 ceph-mon[30820]: pgmap v13804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:00:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:57 smithi116 ceph-mon[35411]: pgmap v13804: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:00:59 smithi114 ceph-mon[30820]: pgmap v13805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:00:59 smithi116 ceph-mon[35411]: pgmap v13805: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:01 smithi114 ceph-mon[30820]: pgmap v13806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:01:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:01 smithi116 ceph-mon[35411]: pgmap v13806: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:01:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:03 smithi114 ceph-mon[30820]: pgmap v13807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:03 smithi116 ceph-mon[35411]: pgmap v13807: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:05 smithi114 ceph-mon[30820]: pgmap v13808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:05 smithi116 ceph-mon[35411]: pgmap v13808: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:07 smithi114 ceph-mon[30820]: pgmap v13809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:07 smithi116 ceph-mon[35411]: pgmap v13809: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:09 smithi114 ceph-mon[30820]: pgmap v13810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:09 smithi116 ceph-mon[35411]: pgmap v13810: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:11 smithi114 ceph-mon[30820]: pgmap v13811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:01:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:11 smithi116 ceph-mon[35411]: pgmap v13811: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:01:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:13 smithi114 ceph-mon[30820]: pgmap v13812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:13 smithi116 ceph-mon[35411]: pgmap v13812: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:15 smithi114 ceph-mon[30820]: pgmap v13813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:15 smithi116 ceph-mon[35411]: pgmap v13813: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:18 smithi114 ceph-mon[30820]: pgmap v13814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:18 smithi116 ceph-mon[35411]: pgmap v13814: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:20 smithi114 ceph-mon[30820]: pgmap v13815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:20 smithi116 ceph-mon[35411]: pgmap v13815: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:22 smithi114 ceph-mon[30820]: pgmap v13816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:01:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:22 smithi116 ceph-mon[35411]: pgmap v13816: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:01:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:24 smithi114 ceph-mon[30820]: pgmap v13817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:24 smithi116 ceph-mon[35411]: pgmap v13817: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:26 smithi114 ceph-mon[30820]: pgmap v13818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:26 smithi116 ceph-mon[35411]: pgmap v13818: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:28 smithi114 ceph-mon[30820]: pgmap v13819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:28 smithi116 ceph-mon[35411]: pgmap v13819: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:30 smithi114 ceph-mon[30820]: pgmap v13820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:30 smithi116 ceph-mon[35411]: pgmap v13820: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:31 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:01:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:31 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:01:32.330 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:32 smithi116 ceph-mon[35411]: pgmap v13821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:01:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:32 smithi114 ceph-mon[30820]: pgmap v13821: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:01:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-mon[30820]: pgmap v13822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:34.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:01:34.228+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:01:34.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:01:34.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:01:34.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:01:34.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:01:34.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:01:34.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:01:34.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:01:34.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:01:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:01:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:34 smithi116 ceph-mon[35411]: pgmap v13822: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:01:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:01:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:01:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:01:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:01:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:01:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:36 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:01:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:01:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:01:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:01:36.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:01:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:01:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:01:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:01:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:01:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:01:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:01:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:36 smithi114 ceph-mon[30820]: pgmap v13823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:01:36.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:36 smithi114 ceph-mon[30820]: pgmap v13824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:01:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:36 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:01:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:01:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:01:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:01:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:01:36.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:01:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:01:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:01:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:01:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:01:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:36 smithi116 ceph-mon[35411]: pgmap v13823: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:01:36.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:36 smithi116 ceph-mon[35411]: pgmap v13824: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:01:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:38 smithi114 ceph-mon[30820]: pgmap v13825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:01:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:38 smithi116 ceph-mon[35411]: pgmap v13825: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:01:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:40 smithi114 ceph-mon[30820]: pgmap v13826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:01:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:40 smithi116 ceph-mon[35411]: pgmap v13826: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:01:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:42 smithi114 ceph-mon[30820]: pgmap v13827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:01:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:42 smithi116 ceph-mon[35411]: pgmap v13827: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:01:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:44 smithi114 ceph-mon[30820]: pgmap v13828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:01:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:44 smithi116 ceph-mon[35411]: pgmap v13828: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:01:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:45 smithi114 ceph-mon[30820]: pgmap v13829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:01:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:45 smithi116 ceph-mon[35411]: pgmap v13829: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:01:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:47 smithi114 ceph-mon[30820]: pgmap v13830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:47 smithi116 ceph-mon[35411]: pgmap v13830: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:49 smithi114 ceph-mon[30820]: pgmap v13831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:49 smithi116 ceph-mon[35411]: pgmap v13831: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:51 smithi114 ceph-mon[30820]: pgmap v13832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:51 smithi116 ceph-mon[35411]: pgmap v13832: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:53 smithi114 ceph-mon[30820]: pgmap v13833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:53 smithi116 ceph-mon[35411]: pgmap v13833: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:55 smithi114 ceph-mon[30820]: pgmap v13834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:55 smithi116 ceph-mon[35411]: pgmap v13834: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:01:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:57 smithi114 ceph-mon[30820]: pgmap v13835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:57 smithi116 ceph-mon[35411]: pgmap v13835: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:01:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:01:59 smithi114 ceph-mon[30820]: pgmap v13836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:01:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:01:59 smithi116 ceph-mon[35411]: pgmap v13836: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:02:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:01 smithi114 ceph-mon[30820]: pgmap v13837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:01 smithi116 ceph-mon[35411]: pgmap v13837: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:03 smithi114 ceph-mon[30820]: pgmap v13838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:03 smithi116 ceph-mon[35411]: pgmap v13838: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:05 smithi114 ceph-mon[30820]: pgmap v13839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:05 smithi116 ceph-mon[35411]: pgmap v13839: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:07 smithi114 ceph-mon[30820]: pgmap v13840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:07 smithi116 ceph-mon[35411]: pgmap v13840: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:09 smithi114 ceph-mon[30820]: pgmap v13841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:02:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:09 smithi116 ceph-mon[35411]: pgmap v13841: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:02:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:11 smithi114 ceph-mon[30820]: pgmap v13842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:11 smithi116 ceph-mon[35411]: pgmap v13842: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:13 smithi114 ceph-mon[30820]: pgmap v13843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:13 smithi116 ceph-mon[35411]: pgmap v13843: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:15.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:15 smithi114 ceph-mon[30820]: pgmap v13844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:15 smithi116 ceph-mon[35411]: pgmap v13844: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:17 smithi114 ceph-mon[30820]: pgmap v13845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:17 smithi116 ceph-mon[35411]: pgmap v13845: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:19 smithi114 ceph-mon[30820]: pgmap v13846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:02:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:19 smithi116 ceph-mon[35411]: pgmap v13846: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:02:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:21 smithi114 ceph-mon[30820]: pgmap v13847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:21 smithi116 ceph-mon[35411]: pgmap v13847: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:23 smithi114 ceph-mon[30820]: pgmap v13848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:23 smithi116 ceph-mon[35411]: pgmap v13848: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:25 smithi114 ceph-mon[30820]: pgmap v13849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:25 smithi116 ceph-mon[35411]: pgmap v13849: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:27 smithi114 ceph-mon[30820]: pgmap v13850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:27 smithi116 ceph-mon[35411]: pgmap v13850: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:29 smithi114 ceph-mon[30820]: pgmap v13851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:02:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:29 smithi116 ceph-mon[35411]: pgmap v13851: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:02:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:31 smithi114 ceph-mon[30820]: pgmap v13852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:31 smithi116 ceph-mon[35411]: pgmap v13852: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:33 smithi114 ceph-mon[30820]: pgmap v13853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:33 smithi116 ceph-mon[35411]: pgmap v13853: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:02:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:02:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:35 smithi114 ceph-mon[30820]: pgmap v13854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:35 smithi116 ceph-mon[35411]: pgmap v13854: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-mon[30820]: pgmap v13855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:37.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:02:37.588+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:02:37.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:02:37.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:02:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:02:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:37 smithi116 ceph-mon[35411]: pgmap v13855: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:38 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:02:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:02:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:02:38.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:38 smithi114 ceph-mon[30820]: pgmap v13856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:38 smithi114 ceph-mon[30820]: pgmap v13857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:02:38.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:02:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:38 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:02:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:02:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:02:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:02:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:02:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:02:38.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:38 smithi116 ceph-mon[35411]: pgmap v13856: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:38 smithi116 ceph-mon[35411]: pgmap v13857: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:02:38.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:02:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:40 smithi114 ceph-mon[30820]: pgmap v13858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:02:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:40 smithi116 ceph-mon[35411]: pgmap v13858: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:02:42.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:42 smithi114 ceph-mon[30820]: pgmap v13859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:02:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:42 smithi116 ceph-mon[35411]: pgmap v13859: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:02:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:44 smithi114 ceph-mon[30820]: pgmap v13860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:02:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:44 smithi116 ceph-mon[35411]: pgmap v13860: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:02:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:46 smithi114 ceph-mon[30820]: pgmap v13861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:02:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:46 smithi116 ceph-mon[35411]: pgmap v13861: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:02:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:48 smithi114 ceph-mon[30820]: pgmap v13862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:02:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:48 smithi116 ceph-mon[35411]: pgmap v13862: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:02:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:50 smithi116 ceph-mon[35411]: pgmap v13863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:50 smithi114 ceph-mon[30820]: pgmap v13863: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:02:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:52 smithi116 ceph-mon[35411]: pgmap v13864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:52 smithi114 ceph-mon[30820]: pgmap v13864: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:02:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:54 smithi116 ceph-mon[35411]: pgmap v13865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:02:54.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:54 smithi114 ceph-mon[30820]: pgmap v13865: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:02:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:56 smithi114 ceph-mon[30820]: pgmap v13866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:02:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:56 smithi116 ceph-mon[35411]: pgmap v13866: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:02:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:02:58 smithi114 ceph-mon[30820]: pgmap v13867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:02:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:02:58 smithi116 ceph-mon[35411]: pgmap v13867: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:03:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:00 smithi114 ceph-mon[30820]: pgmap v13868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:03:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:00 smithi116 ceph-mon[35411]: pgmap v13868: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:03:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:02 smithi114 ceph-mon[30820]: pgmap v13869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 6 op/s 2024-09-19T14:03:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:02 smithi116 ceph-mon[35411]: pgmap v13869: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 6 op/s 2024-09-19T14:03:04.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:04 smithi116 ceph-mon[35411]: pgmap v13870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:03:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:04 smithi114 ceph-mon[30820]: pgmap v13870: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:03:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:06 smithi114 ceph-mon[30820]: pgmap v13871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:03:07.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:06 smithi116 ceph-mon[35411]: pgmap v13871: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:03:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:08 smithi114 ceph-mon[30820]: pgmap v13872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:03:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:08 smithi116 ceph-mon[35411]: pgmap v13872: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:03:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:10 smithi114 ceph-mon[30820]: pgmap v13873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:03:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:10 smithi116 ceph-mon[35411]: pgmap v13873: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:03:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:12 smithi114 ceph-mon[30820]: pgmap v13874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:03:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:12 smithi116 ceph-mon[35411]: pgmap v13874: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:03:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:14 smithi114 ceph-mon[30820]: pgmap v13875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:14 smithi116 ceph-mon[35411]: pgmap v13875: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:16 smithi114 ceph-mon[30820]: pgmap v13876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:03:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:16 smithi116 ceph-mon[35411]: pgmap v13876: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:03:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:18 smithi114 ceph-mon[30820]: pgmap v13877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:18 smithi116 ceph-mon[35411]: pgmap v13877: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:20 smithi114 ceph-mon[30820]: pgmap v13878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:03:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:20 smithi116 ceph-mon[35411]: pgmap v13878: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:03:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:22 smithi114 ceph-mon[30820]: pgmap v13879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:03:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:22 smithi116 ceph-mon[35411]: pgmap v13879: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:03:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:24 smithi114 ceph-mon[30820]: pgmap v13880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:24 smithi116 ceph-mon[35411]: pgmap v13880: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:26 smithi114 ceph-mon[30820]: pgmap v13881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:03:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:26 smithi116 ceph-mon[35411]: pgmap v13881: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:03:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:28 smithi114 ceph-mon[30820]: pgmap v13882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:28 smithi116 ceph-mon[35411]: pgmap v13882: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:30 smithi114 ceph-mon[30820]: pgmap v13883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:03:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:30 smithi116 ceph-mon[35411]: pgmap v13883: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:03:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:32 smithi114 ceph-mon[30820]: pgmap v13884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:03:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:32 smithi116 ceph-mon[35411]: pgmap v13884: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:03:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:34 smithi114 ceph-mon[30820]: pgmap v13885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:34 smithi116 ceph-mon[35411]: pgmap v13885: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:36 smithi114 ceph-mon[30820]: pgmap v13886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:03:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:36 smithi116 ceph-mon[35411]: pgmap v13886: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:03:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:37 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:03:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:37 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:03:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:38 smithi114 ceph-mon[30820]: pgmap v13887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:38 smithi116 ceph-mon[35411]: pgmap v13887: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:40 smithi116 ceph-mon[35411]: pgmap v13888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:03:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:40.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:40 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-mon[30820]: pgmap v13888: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:03:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:41 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:03:41.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:03:41.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:41 smithi114 ceph-mon[30820]: pgmap v13889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:41 smithi114 ceph-mon[30820]: pgmap v13890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:41 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:03:40.948+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:03:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:03:41.421 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:03:41.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:03:41.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:03:41.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:03:41.422 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:03:40 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:03:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:41 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:03:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:03:41.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:03:41.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:03:41.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:03:41.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:03:41.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:03:41.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:41 smithi116 ceph-mon[35411]: pgmap v13889: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:03:41.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:41 smithi116 ceph-mon[35411]: pgmap v13890: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:03:41.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:41 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:03:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:44 smithi114 ceph-mon[30820]: pgmap v13891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:03:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:44 smithi116 ceph-mon[35411]: pgmap v13891: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:03:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:46 smithi114 ceph-mon[30820]: pgmap v13892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:03:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:46 smithi116 ceph-mon[35411]: pgmap v13892: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:03:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:48 smithi114 ceph-mon[30820]: pgmap v13893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:03:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:48 smithi116 ceph-mon[35411]: pgmap v13893: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:03:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:50 smithi114 ceph-mon[30820]: pgmap v13894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:03:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:50 smithi116 ceph-mon[35411]: pgmap v13894: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:03:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:52 smithi114 ceph-mon[30820]: pgmap v13895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:03:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:52 smithi116 ceph-mon[35411]: pgmap v13895: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:03:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:54 smithi114 ceph-mon[30820]: pgmap v13896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:54 smithi116 ceph-mon[35411]: pgmap v13896: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:03:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:56 smithi114 ceph-mon[30820]: pgmap v13897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:03:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:56 smithi116 ceph-mon[35411]: pgmap v13897: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:03:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:03:58 smithi114 ceph-mon[30820]: pgmap v13898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:03:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:03:58 smithi116 ceph-mon[35411]: pgmap v13898: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:00 smithi114 ceph-mon[30820]: pgmap v13899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:00 smithi116 ceph-mon[35411]: pgmap v13899: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:02 smithi114 ceph-mon[30820]: pgmap v13900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:02 smithi116 ceph-mon[35411]: pgmap v13900: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:04 smithi114 ceph-mon[30820]: pgmap v13901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:04 smithi116 ceph-mon[35411]: pgmap v13901: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:06 smithi114 ceph-mon[30820]: pgmap v13902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:04:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:06 smithi116 ceph-mon[35411]: pgmap v13902: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:04:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:08 smithi114 ceph-mon[30820]: pgmap v13903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:08 smithi116 ceph-mon[35411]: pgmap v13903: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:10 smithi114 ceph-mon[30820]: pgmap v13904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:10 smithi116 ceph-mon[35411]: pgmap v13904: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:12 smithi114 ceph-mon[30820]: pgmap v13905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:12 smithi116 ceph-mon[35411]: pgmap v13905: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:14 smithi114 ceph-mon[30820]: pgmap v13906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:14 smithi116 ceph-mon[35411]: pgmap v13906: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:16 smithi114 ceph-mon[30820]: pgmap v13907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:04:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:16 smithi116 ceph-mon[35411]: pgmap v13907: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:04:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:18 smithi114 ceph-mon[30820]: pgmap v13908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:18 smithi116 ceph-mon[35411]: pgmap v13908: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:20 smithi114 ceph-mon[30820]: pgmap v13909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:20 smithi116 ceph-mon[35411]: pgmap v13909: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:22 smithi114 ceph-mon[30820]: pgmap v13910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:22 smithi116 ceph-mon[35411]: pgmap v13910: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:24 smithi114 ceph-mon[30820]: pgmap v13911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:24 smithi116 ceph-mon[35411]: pgmap v13911: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:26 smithi114 ceph-mon[30820]: pgmap v13912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:04:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:26 smithi116 ceph-mon[35411]: pgmap v13912: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:04:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:27 smithi114 ceph-mon[30820]: pgmap v13913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:27 smithi116 ceph-mon[35411]: pgmap v13913: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:30 smithi114 ceph-mon[30820]: pgmap v13914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:30 smithi116 ceph-mon[35411]: pgmap v13914: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:32 smithi114 ceph-mon[30820]: pgmap v13915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:32 smithi116 ceph-mon[35411]: pgmap v13915: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:34 smithi114 ceph-mon[30820]: pgmap v13916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:34 smithi116 ceph-mon[35411]: pgmap v13916: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:36 smithi114 ceph-mon[30820]: pgmap v13917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:04:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:36 smithi116 ceph-mon[35411]: pgmap v13917: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:04:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:38 smithi114 ceph-mon[30820]: pgmap v13918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:38 smithi116 ceph-mon[35411]: pgmap v13918: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:40 smithi114 ceph-mon[30820]: pgmap v13919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:40 smithi116 ceph-mon[35411]: pgmap v13919: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:42 smithi114 ceph-mon[30820]: pgmap v13920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:04:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:42.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:42.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:42.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:42 smithi116 ceph-mon[35411]: pgmap v13920: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:04:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:04:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-mon[30820]: pgmap v13921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:04:44.245+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:04:44.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:04:44.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:04:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:04:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:44 smithi116 ceph-mon[35411]: pgmap v13921: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:04:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:04:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:04:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:04:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:04:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:04:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:46 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:04:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:04:46.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:46 smithi114 ceph-mon[30820]: pgmap v13922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T14:04:46.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:46 smithi114 ceph-mon[30820]: pgmap v13923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:04:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:46 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:04:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:04:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:04:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:04:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:04:46.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:04:46.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:04:46.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:04:46.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:04:46.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:04:46.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:46 smithi116 ceph-mon[35411]: pgmap v13922: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T14:04:46.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:46 smithi116 ceph-mon[35411]: pgmap v13923: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:04:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:48 smithi114 ceph-mon[30820]: pgmap v13924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:04:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:48 smithi116 ceph-mon[35411]: pgmap v13924: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:04:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:50 smithi114 ceph-mon[30820]: pgmap v13925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:04:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:50 smithi116 ceph-mon[35411]: pgmap v13925: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:04:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:52 smithi114 ceph-mon[30820]: pgmap v13926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:04:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:52 smithi116 ceph-mon[35411]: pgmap v13926: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:04:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:54 smithi114 ceph-mon[30820]: pgmap v13927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:04:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:54 smithi116 ceph-mon[35411]: pgmap v13927: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:04:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:56 smithi114 ceph-mon[30820]: pgmap v13928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:04:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:56 smithi116 ceph-mon[35411]: pgmap v13928: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:04:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:04:58 smithi114 ceph-mon[30820]: pgmap v13929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:04:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:04:58 smithi116 ceph-mon[35411]: pgmap v13929: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:00 smithi114 ceph-mon[30820]: pgmap v13930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:00 smithi116 ceph-mon[35411]: pgmap v13930: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:02 smithi114 ceph-mon[30820]: pgmap v13931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:02 smithi116 ceph-mon[35411]: pgmap v13931: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:04 smithi114 ceph-mon[30820]: pgmap v13932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:04 smithi116 ceph-mon[35411]: pgmap v13932: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:06 smithi114 ceph-mon[30820]: pgmap v13933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:06 smithi116 ceph-mon[35411]: pgmap v13933: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:08 smithi114 ceph-mon[30820]: pgmap v13934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:08 smithi116 ceph-mon[35411]: pgmap v13934: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:10 smithi114 ceph-mon[30820]: pgmap v13935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:10 smithi116 ceph-mon[35411]: pgmap v13935: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:12 smithi114 ceph-mon[30820]: pgmap v13936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:12 smithi116 ceph-mon[35411]: pgmap v13936: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:14 smithi114 ceph-mon[30820]: pgmap v13937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:14 smithi116 ceph-mon[35411]: pgmap v13937: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:16 smithi114 ceph-mon[30820]: pgmap v13938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:16 smithi116 ceph-mon[35411]: pgmap v13938: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:18 smithi114 ceph-mon[30820]: pgmap v13939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:18 smithi116 ceph-mon[35411]: pgmap v13939: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:20 smithi114 ceph-mon[30820]: pgmap v13940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:20 smithi116 ceph-mon[35411]: pgmap v13940: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:22 smithi114 ceph-mon[30820]: pgmap v13941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:22 smithi116 ceph-mon[35411]: pgmap v13941: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:24 smithi114 ceph-mon[30820]: pgmap v13942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:24 smithi116 ceph-mon[35411]: pgmap v13942: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:26 smithi114 ceph-mon[30820]: pgmap v13943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:26 smithi116 ceph-mon[35411]: pgmap v13943: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:28 smithi114 ceph-mon[30820]: pgmap v13944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:28 smithi116 ceph-mon[35411]: pgmap v13944: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:30 smithi114 ceph-mon[30820]: pgmap v13945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:30 smithi116 ceph-mon[35411]: pgmap v13945: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:32 smithi114 ceph-mon[30820]: pgmap v13946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:32 smithi116 ceph-mon[35411]: pgmap v13946: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:34 smithi114 ceph-mon[30820]: pgmap v13947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:34 smithi116 ceph-mon[35411]: pgmap v13947: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:36 smithi114 ceph-mon[30820]: pgmap v13948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:36 smithi116 ceph-mon[35411]: pgmap v13948: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:38 smithi114 ceph-mon[30820]: pgmap v13949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:38 smithi116 ceph-mon[35411]: pgmap v13949: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:40 smithi114 ceph-mon[30820]: pgmap v13950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:40 smithi116 ceph-mon[35411]: pgmap v13950: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:05:42.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:42 smithi114 ceph-mon[30820]: pgmap v13951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:42 smithi116 ceph-mon[35411]: pgmap v13951: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:05:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:44 smithi114 ceph-mon[30820]: pgmap v13952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:44 smithi116 ceph-mon[35411]: pgmap v13952: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:05:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:05:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:05:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:46 smithi114 ceph-mon[30820]: pgmap v13953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:46 smithi116 ceph-mon[35411]: pgmap v13953: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:05:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-mon[30820]: pgmap v13954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:47 smithi116 ceph-mon[35411]: pgmap v13954: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:05:47.604+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:05:47.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:05:47.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:05:47.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:05:47 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:05:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:48 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:05:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:05:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:48 smithi114 ceph-mon[30820]: pgmap v13955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:48 smithi114 ceph-mon[30820]: pgmap v13956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:05:48.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:05:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:48 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:05:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:05:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:05:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:05:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:05:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:05:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:05:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:05:48.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:05:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:05:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:05:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:05:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:05:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:48 smithi116 ceph-mon[35411]: pgmap v13955: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:05:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:48 smithi116 ceph-mon[35411]: pgmap v13956: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:05:48.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:05:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:50 smithi114 ceph-mon[30820]: pgmap v13957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:05:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:50 smithi116 ceph-mon[35411]: pgmap v13957: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:05:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:52 smithi116 ceph-mon[35411]: pgmap v13958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:05:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:52 smithi114 ceph-mon[30820]: pgmap v13958: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:05:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:54 smithi116 ceph-mon[35411]: pgmap v13959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:05:54.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:54 smithi114 ceph-mon[30820]: pgmap v13959: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:05:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:56 smithi116 ceph-mon[35411]: pgmap v13960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:05:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:56 smithi114 ceph-mon[30820]: pgmap v13960: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:05:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:05:58 smithi114 ceph-mon[30820]: pgmap v13961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:05:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:05:58 smithi116 ceph-mon[35411]: pgmap v13961: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:06:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:00 smithi114 ceph-mon[30820]: pgmap v13962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:00 smithi116 ceph-mon[35411]: pgmap v13962: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:02 smithi114 ceph-mon[30820]: pgmap v13963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:02 smithi116 ceph-mon[35411]: pgmap v13963: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:04.981 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:04 smithi116 ceph-mon[35411]: pgmap v13964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:04 smithi114 ceph-mon[30820]: pgmap v13964: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:07.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:06 smithi114 ceph-mon[30820]: pgmap v13965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:07.200 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:06 smithi116 ceph-mon[35411]: pgmap v13965: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:09.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:08 smithi114 ceph-mon[30820]: pgmap v13966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:09.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:08 smithi116 ceph-mon[35411]: pgmap v13966: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:11.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:10 smithi114 ceph-mon[30820]: pgmap v13967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:11.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:10 smithi116 ceph-mon[35411]: pgmap v13967: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:13.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:12 smithi114 ceph-mon[30820]: pgmap v13968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:13.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:12 smithi116 ceph-mon[35411]: pgmap v13968: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:14 smithi114 ceph-mon[30820]: pgmap v13969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:14 smithi116 ceph-mon[35411]: pgmap v13969: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:16 smithi114 ceph-mon[30820]: pgmap v13970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:16 smithi116 ceph-mon[35411]: pgmap v13970: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:18 smithi114 ceph-mon[30820]: pgmap v13971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:18 smithi116 ceph-mon[35411]: pgmap v13971: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:20 smithi114 ceph-mon[30820]: pgmap v13972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:20 smithi116 ceph-mon[35411]: pgmap v13972: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:22 smithi114 ceph-mon[30820]: pgmap v13973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:22 smithi116 ceph-mon[35411]: pgmap v13973: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:24 smithi114 ceph-mon[30820]: pgmap v13974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:24 smithi116 ceph-mon[35411]: pgmap v13974: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:26 smithi114 ceph-mon[30820]: pgmap v13975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:26 smithi116 ceph-mon[35411]: pgmap v13975: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:28 smithi114 ceph-mon[30820]: pgmap v13976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:28 smithi116 ceph-mon[35411]: pgmap v13976: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:30 smithi114 ceph-mon[30820]: pgmap v13977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:30 smithi116 ceph-mon[35411]: pgmap v13977: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:32 smithi114 ceph-mon[30820]: pgmap v13978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:32 smithi116 ceph-mon[35411]: pgmap v13978: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:34 smithi114 ceph-mon[30820]: pgmap v13979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:34 smithi116 ceph-mon[35411]: pgmap v13979: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:36 smithi114 ceph-mon[30820]: pgmap v13980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:36 smithi116 ceph-mon[35411]: pgmap v13980: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:38 smithi114 ceph-mon[30820]: pgmap v13981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:38 smithi116 ceph-mon[35411]: pgmap v13981: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:40 smithi114 ceph-mon[30820]: pgmap v13982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:40 smithi116 ceph-mon[35411]: pgmap v13982: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:42 smithi114 ceph-mon[30820]: pgmap v13983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:42 smithi116 ceph-mon[35411]: pgmap v13983: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:06:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:44 smithi114 ceph-mon[30820]: pgmap v13984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:44 smithi116 ceph-mon[35411]: pgmap v13984: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:46 smithi114 ceph-mon[30820]: pgmap v13985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:46 smithi116 ceph-mon[35411]: pgmap v13985: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:06:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:06:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:06:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:48 smithi114 ceph-mon[30820]: pgmap v13986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:48 smithi116 ceph-mon[35411]: pgmap v13986: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:06:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-mon[30820]: pgmap v13987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:51.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:06:50.965+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:06:51.167 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:06:51.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:06:51.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:06:50 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:06:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:50 smithi116 ceph-mon[35411]: pgmap v13987: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:06:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:51 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:06:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:06:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:51 smithi114 ceph-mon[30820]: pgmap v13988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:06:52.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:51 smithi114 ceph-mon[30820]: pgmap v13989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:06:52.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:51 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:06:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:51 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:06:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:06:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:06:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:06:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:06:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:06:52.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:06:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:06:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:06:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:06:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:06:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:06:52.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:06:52.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:51 smithi116 ceph-mon[35411]: pgmap v13988: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:06:52.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:51 smithi116 ceph-mon[35411]: pgmap v13989: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:06:52.182 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:51 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:06:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:54 smithi114 ceph-mon[30820]: pgmap v13990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:06:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:54 smithi116 ceph-mon[35411]: pgmap v13990: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:06:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:56 smithi114 ceph-mon[30820]: pgmap v13991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:06:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:56 smithi116 ceph-mon[35411]: pgmap v13991: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:06:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:06:58 smithi114 ceph-mon[30820]: pgmap v13992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:06:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:06:58 smithi116 ceph-mon[35411]: pgmap v13992: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:07:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:00 smithi114 ceph-mon[30820]: pgmap v13993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:07:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:00 smithi116 ceph-mon[35411]: pgmap v13993: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:07:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:02 smithi114 ceph-mon[30820]: pgmap v13994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:07:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:02 smithi116 ceph-mon[35411]: pgmap v13994: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:07:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:04 smithi114 ceph-mon[30820]: pgmap v13995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:04 smithi116 ceph-mon[35411]: pgmap v13995: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:06 smithi114 ceph-mon[30820]: pgmap v13996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:07:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:06 smithi116 ceph-mon[35411]: pgmap v13996: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:07:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:08 smithi114 ceph-mon[30820]: pgmap v13997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:08 smithi116 ceph-mon[35411]: pgmap v13997: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:10 smithi114 ceph-mon[30820]: pgmap v13998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:07:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:10 smithi116 ceph-mon[35411]: pgmap v13998: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:07:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:12 smithi114 ceph-mon[30820]: pgmap v13999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:12 smithi116 ceph-mon[35411]: pgmap v13999: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:14 smithi114 ceph-mon[30820]: pgmap v14000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:07:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:14 smithi116 ceph-mon[35411]: pgmap v14000: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:07:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:16 smithi114 ceph-mon[30820]: pgmap v14001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:07:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:16 smithi116 ceph-mon[35411]: pgmap v14001: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:07:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:18 smithi114 ceph-mon[30820]: pgmap v14002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:18 smithi116 ceph-mon[35411]: pgmap v14002: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:20 smithi114 ceph-mon[30820]: pgmap v14003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:20 smithi116 ceph-mon[35411]: pgmap v14003: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:22 smithi114 ceph-mon[30820]: pgmap v14004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:22 smithi116 ceph-mon[35411]: pgmap v14004: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:24 smithi114 ceph-mon[30820]: pgmap v14005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:24 smithi116 ceph-mon[35411]: pgmap v14005: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:26 smithi114 ceph-mon[30820]: pgmap v14006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:07:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:26 smithi116 ceph-mon[35411]: pgmap v14006: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:07:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:28 smithi114 ceph-mon[30820]: pgmap v14007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:28 smithi116 ceph-mon[35411]: pgmap v14007: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:30 smithi114 ceph-mon[30820]: pgmap v14008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:30 smithi116 ceph-mon[35411]: pgmap v14008: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:32 smithi114 ceph-mon[30820]: pgmap v14009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:32 smithi116 ceph-mon[35411]: pgmap v14009: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:34 smithi114 ceph-mon[30820]: pgmap v14010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:34 smithi116 ceph-mon[35411]: pgmap v14010: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:36 smithi114 ceph-mon[30820]: pgmap v14011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:07:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:36 smithi116 ceph-mon[35411]: pgmap v14011: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:07:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:38 smithi114 ceph-mon[30820]: pgmap v14012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:38 smithi116 ceph-mon[35411]: pgmap v14012: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:40 smithi114 ceph-mon[30820]: pgmap v14013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:40 smithi116 ceph-mon[35411]: pgmap v14013: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:42 smithi114 ceph-mon[30820]: pgmap v14014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:42 smithi116 ceph-mon[35411]: pgmap v14014: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:44 smithi114 ceph-mon[30820]: pgmap v14015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:44 smithi116 ceph-mon[35411]: pgmap v14015: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:45 smithi114 ceph-mon[30820]: pgmap v14016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:07:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:45 smithi116 ceph-mon[35411]: pgmap v14016: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:07:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:48 smithi114 ceph-mon[30820]: pgmap v14017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:48 smithi116 ceph-mon[35411]: pgmap v14017: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:07:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:50 smithi114 ceph-mon[30820]: pgmap v14018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:50 smithi116 ceph-mon[35411]: pgmap v14018: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:52 smithi114 ceph-mon[30820]: pgmap v14019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:07:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:52 smithi116 ceph-mon[35411]: pgmap v14019: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:07:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:07:54.327 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-mon[30820]: pgmap v14020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:54 smithi116 ceph-mon[35411]: pgmap v14020: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:07:54.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:07:54.324+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:07:54.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:07:54.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:07:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:07:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:07:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:07:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:07:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:07:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:07:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:07:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:07:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:07:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:07:56.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:56 smithi114 ceph-mon[30820]: pgmap v14021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:07:56.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:56 smithi114 ceph-mon[30820]: pgmap v14022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:07:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:07:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:07:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:07:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:07:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:07:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:07:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:07:56.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:07:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:07:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:07:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:07:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:56 smithi116 ceph-mon[35411]: pgmap v14021: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:07:56.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:56 smithi116 ceph-mon[35411]: pgmap v14022: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:07:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:07:58 smithi114 ceph-mon[30820]: pgmap v14023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:07:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:07:58 smithi116 ceph-mon[35411]: pgmap v14023: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:08:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:00 smithi114 ceph-mon[30820]: pgmap v14024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:08:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:00 smithi116 ceph-mon[35411]: pgmap v14024: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:08:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:02 smithi114 ceph-mon[30820]: pgmap v14025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:08:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:02 smithi116 ceph-mon[35411]: pgmap v14025: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:08:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:04 smithi114 ceph-mon[30820]: pgmap v14026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:08:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:04 smithi116 ceph-mon[35411]: pgmap v14026: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:08:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:06 smithi114 ceph-mon[30820]: pgmap v14027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:08:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:06 smithi116 ceph-mon[35411]: pgmap v14027: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:08:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:08 smithi114 ceph-mon[30820]: pgmap v14028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:08 smithi116 ceph-mon[35411]: pgmap v14028: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:10 smithi114 ceph-mon[30820]: pgmap v14029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:10 smithi116 ceph-mon[35411]: pgmap v14029: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:12 smithi114 ceph-mon[30820]: pgmap v14030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:12 smithi116 ceph-mon[35411]: pgmap v14030: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:14 smithi114 ceph-mon[30820]: pgmap v14031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:14 smithi116 ceph-mon[35411]: pgmap v14031: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:16 smithi114 ceph-mon[30820]: pgmap v14032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:16 smithi116 ceph-mon[35411]: pgmap v14032: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:18 smithi114 ceph-mon[30820]: pgmap v14033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:18 smithi116 ceph-mon[35411]: pgmap v14033: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:20 smithi114 ceph-mon[30820]: pgmap v14034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:20 smithi116 ceph-mon[35411]: pgmap v14034: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:22 smithi114 ceph-mon[30820]: pgmap v14035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:22 smithi116 ceph-mon[35411]: pgmap v14035: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:24 smithi114 ceph-mon[30820]: pgmap v14036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:24 smithi116 ceph-mon[35411]: pgmap v14036: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:26 smithi114 ceph-mon[30820]: pgmap v14037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:26 smithi116 ceph-mon[35411]: pgmap v14037: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:28 smithi114 ceph-mon[30820]: pgmap v14038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:28 smithi116 ceph-mon[35411]: pgmap v14038: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:30 smithi114 ceph-mon[30820]: pgmap v14039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:30 smithi116 ceph-mon[35411]: pgmap v14039: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:32 smithi114 ceph-mon[30820]: pgmap v14040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:32 smithi116 ceph-mon[35411]: pgmap v14040: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:33 smithi114 ceph-mon[30820]: pgmap v14041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:33 smithi116 ceph-mon[35411]: pgmap v14041: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:35 smithi114 ceph-mon[30820]: pgmap v14042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:35 smithi116 ceph-mon[35411]: pgmap v14042: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:37 smithi114 ceph-mon[30820]: pgmap v14043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:37 smithi116 ceph-mon[35411]: pgmap v14043: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:39 smithi114 ceph-mon[30820]: pgmap v14044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:39 smithi116 ceph-mon[35411]: pgmap v14044: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:42 smithi114 ceph-mon[30820]: pgmap v14045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:42 smithi116 ceph-mon[35411]: pgmap v14045: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:44 smithi114 ceph-mon[30820]: pgmap v14046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:44 smithi116 ceph-mon[35411]: pgmap v14046: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:46 smithi114 ceph-mon[30820]: pgmap v14047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:46 smithi116 ceph-mon[35411]: pgmap v14047: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:48 smithi114 ceph-mon[30820]: pgmap v14048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:48 smithi116 ceph-mon[35411]: pgmap v14048: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:50 smithi114 ceph-mon[30820]: pgmap v14049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:50 smithi116 ceph-mon[35411]: pgmap v14049: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:08:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:52 smithi114 ceph-mon[30820]: pgmap v14050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:52 smithi116 ceph-mon[35411]: pgmap v14050: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:08:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:54 smithi114 ceph-mon[30820]: pgmap v14051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:54 smithi116 ceph-mon[35411]: pgmap v14051: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:08:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:55 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:08:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:55 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:08:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:56 smithi114 ceph-mon[30820]: pgmap v14052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:56 smithi116 ceph-mon[35411]: pgmap v14052: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:08:58.118 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:08:57.684+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:08:58.118 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:08:58.118 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:08:58.118 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:08:58.118 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:08:58.119 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:08:58.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:08:58.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:08:58.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:08:58.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:08:58.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:08:57 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:08:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:58 smithi114 ceph-mon[30820]: pgmap v14053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:08:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:08:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:58 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:08:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:58 smithi116 ceph-mon[35411]: pgmap v14053: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:08:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:08:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:08:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:58 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:08:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:59 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:08:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:08:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:08:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:08:59.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:59 smithi114 ceph-mon[30820]: pgmap v14054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:08:59.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:08:59 smithi114 ceph-mon[30820]: pgmap v14055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:08:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:59 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:08:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:08:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:08:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:08:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:08:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:08:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:08:59.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:08:59.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:08:59.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:59 smithi116 ceph-mon[35411]: pgmap v14054: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:08:59.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:08:59 smithi116 ceph-mon[35411]: pgmap v14055: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:09:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:01 smithi114 ceph-mon[30820]: pgmap v14056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:09:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:01 smithi116 ceph-mon[35411]: pgmap v14056: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:09:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:03 smithi114 ceph-mon[30820]: pgmap v14057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:09:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:03 smithi116 ceph-mon[35411]: pgmap v14057: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:09:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:05 smithi114 ceph-mon[30820]: pgmap v14058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:09:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:05 smithi116 ceph-mon[35411]: pgmap v14058: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:09:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:07 smithi114 ceph-mon[30820]: pgmap v14059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:09:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:07 smithi116 ceph-mon[35411]: pgmap v14059: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:09:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:09 smithi114 ceph-mon[30820]: pgmap v14060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:09:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:09 smithi116 ceph-mon[35411]: pgmap v14060: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:09:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:11 smithi114 ceph-mon[30820]: pgmap v14061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:11 smithi116 ceph-mon[35411]: pgmap v14061: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:12 smithi114 ceph-mon[30820]: pgmap v14062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:12 smithi116 ceph-mon[35411]: pgmap v14062: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:14 smithi114 ceph-mon[30820]: pgmap v14063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:14 smithi116 ceph-mon[35411]: pgmap v14063: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:17.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:16 smithi114 ceph-mon[30820]: pgmap v14064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:17.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:16 smithi116 ceph-mon[35411]: pgmap v14064: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:19.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:18 smithi114 ceph-mon[30820]: pgmap v14065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:19.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:18 smithi116 ceph-mon[35411]: pgmap v14065: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:21.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:20 smithi114 ceph-mon[30820]: pgmap v14066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:21.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:20 smithi116 ceph-mon[35411]: pgmap v14066: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:23.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:22 smithi114 ceph-mon[30820]: pgmap v14067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:23.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:22 smithi116 ceph-mon[35411]: pgmap v14067: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:24 smithi114 ceph-mon[30820]: pgmap v14068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:25.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:24 smithi116 ceph-mon[35411]: pgmap v14068: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:27.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:26 smithi114 ceph-mon[30820]: pgmap v14069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:27.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:26 smithi116 ceph-mon[35411]: pgmap v14069: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:29.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:28 smithi114 ceph-mon[30820]: pgmap v14070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:29.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:28 smithi116 ceph-mon[35411]: pgmap v14070: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:31.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:30 smithi114 ceph-mon[30820]: pgmap v14071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:30 smithi116 ceph-mon[35411]: pgmap v14071: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:33.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:32 smithi114 ceph-mon[30820]: pgmap v14072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:33.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:32 smithi116 ceph-mon[35411]: pgmap v14072: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:35.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:34 smithi114 ceph-mon[30820]: pgmap v14073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:35.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:34 smithi116 ceph-mon[35411]: pgmap v14073: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:36 smithi114 ceph-mon[30820]: pgmap v14074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:37.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:36 smithi116 ceph-mon[35411]: pgmap v14074: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:38 smithi114 ceph-mon[30820]: pgmap v14075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:38 smithi116 ceph-mon[35411]: pgmap v14075: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:40 smithi114 ceph-mon[30820]: pgmap v14076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:40 smithi116 ceph-mon[35411]: pgmap v14076: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:42 smithi114 ceph-mon[30820]: pgmap v14077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:42 smithi116 ceph-mon[35411]: pgmap v14077: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:44 smithi114 ceph-mon[30820]: pgmap v14078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:44 smithi116 ceph-mon[35411]: pgmap v14078: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:46 smithi114 ceph-mon[30820]: pgmap v14079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:46 smithi116 ceph-mon[35411]: pgmap v14079: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:48 smithi114 ceph-mon[30820]: pgmap v14080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:48 smithi116 ceph-mon[35411]: pgmap v14080: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:50 smithi114 ceph-mon[30820]: pgmap v14081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:50 smithi116 ceph-mon[35411]: pgmap v14081: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:09:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:52 smithi114 ceph-mon[30820]: pgmap v14082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:52 smithi116 ceph-mon[35411]: pgmap v14082: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:09:55.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:54 smithi114 ceph-mon[30820]: pgmap v14083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:54 smithi116 ceph-mon[35411]: pgmap v14083: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:56 smithi114 ceph-mon[30820]: pgmap v14084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:56 smithi116 ceph-mon[35411]: pgmap v14084: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:09:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:09:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:09:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:09:58 smithi114 ceph-mon[30820]: pgmap v14085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:09:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:09:58 smithi116 ceph-mon[35411]: pgmap v14085: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:00 smithi114 ceph-mon[30820]: pgmap v14086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:10:01.044+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:10:01.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:10:01.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:10:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:00 smithi116 ceph-mon[35411]: pgmap v14086: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T14:10:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:10:02.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-mon[30820]: pgmap v14087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-mon[30820]: pgmap v14088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:10:02.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:10:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:01 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:10:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:10:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:10:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:10:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:10:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:10:02.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:01 smithi116 ceph-mon[35411]: pgmap v14087: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:01 smithi116 ceph-mon[35411]: pgmap v14088: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:10:02.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:10:04.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:04 smithi114 ceph-mon[30820]: pgmap v14089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:10:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:04 smithi116 ceph-mon[35411]: pgmap v14089: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:10:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:06 smithi114 ceph-mon[30820]: pgmap v14090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:10:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:06 smithi116 ceph-mon[35411]: pgmap v14090: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:10:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:08 smithi114 ceph-mon[30820]: pgmap v14091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:10:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:08 smithi116 ceph-mon[35411]: pgmap v14091: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:10:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:10 smithi114 ceph-mon[30820]: pgmap v14092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:10:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:10 smithi116 ceph-mon[35411]: pgmap v14092: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:10:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:12 smithi114 ceph-mon[30820]: pgmap v14093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:10:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:12 smithi116 ceph-mon[35411]: pgmap v14093: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:10:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:14 smithi114 ceph-mon[30820]: pgmap v14094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:14 smithi116 ceph-mon[35411]: pgmap v14094: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:16 smithi114 ceph-mon[30820]: pgmap v14095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:16 smithi116 ceph-mon[35411]: pgmap v14095: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:18 smithi114 ceph-mon[30820]: pgmap v14096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:18 smithi116 ceph-mon[35411]: pgmap v14096: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:20 smithi114 ceph-mon[30820]: pgmap v14097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:20 smithi116 ceph-mon[35411]: pgmap v14097: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:22 smithi114 ceph-mon[30820]: pgmap v14098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:10:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:22 smithi116 ceph-mon[35411]: pgmap v14098: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:10:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:24 smithi114 ceph-mon[30820]: pgmap v14099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:24 smithi116 ceph-mon[35411]: pgmap v14099: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:26 smithi114 ceph-mon[30820]: pgmap v14100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:26 smithi116 ceph-mon[35411]: pgmap v14100: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:28 smithi114 ceph-mon[30820]: pgmap v14101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:28 smithi116 ceph-mon[35411]: pgmap v14101: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:29 smithi114 ceph-mon[30820]: pgmap v14102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:29 smithi116 ceph-mon[35411]: pgmap v14102: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:32 smithi114 ceph-mon[30820]: pgmap v14103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:10:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:32 smithi116 ceph-mon[35411]: pgmap v14103: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:10:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:34 smithi114 ceph-mon[30820]: pgmap v14104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:34 smithi116 ceph-mon[35411]: pgmap v14104: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:36 smithi114 ceph-mon[30820]: pgmap v14105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:36 smithi116 ceph-mon[35411]: pgmap v14105: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:38 smithi114 ceph-mon[30820]: pgmap v14106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:38 smithi116 ceph-mon[35411]: pgmap v14106: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:40 smithi114 ceph-mon[30820]: pgmap v14107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:40 smithi116 ceph-mon[35411]: pgmap v14107: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:42 smithi114 ceph-mon[30820]: pgmap v14108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:10:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:42 smithi116 ceph-mon[35411]: pgmap v14108: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:10:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:44 smithi114 ceph-mon[30820]: pgmap v14109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:44 smithi116 ceph-mon[35411]: pgmap v14109: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:46 smithi114 ceph-mon[30820]: pgmap v14110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:46 smithi116 ceph-mon[35411]: pgmap v14110: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:48 smithi114 ceph-mon[30820]: pgmap v14111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:48 smithi116 ceph-mon[35411]: pgmap v14111: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:50 smithi114 ceph-mon[30820]: pgmap v14112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:50 smithi116 ceph-mon[35411]: pgmap v14112: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:52 smithi114 ceph-mon[30820]: pgmap v14113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:10:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:52 smithi116 ceph-mon[35411]: pgmap v14113: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:10:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:54 smithi114 ceph-mon[30820]: pgmap v14114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:54 smithi116 ceph-mon[35411]: pgmap v14114: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:10:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:55 smithi114 ceph-mon[30820]: pgmap v14115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:55 smithi116 ceph-mon[35411]: pgmap v14115: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:10:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:10:58 smithi114 ceph-mon[30820]: pgmap v14116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:10:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:10:58 smithi116 ceph-mon[35411]: pgmap v14116: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:00 smithi114 ceph-mon[30820]: pgmap v14117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:00 smithi116 ceph-mon[35411]: pgmap v14117: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: pgmap v14118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:11:02.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:11:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: pgmap v14118: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:11:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:11:02.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:02.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:11:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:03 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T14:11:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:03 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T14:11:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:03 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T14:11:03.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:03 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T14:11:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:03 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T14:11:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:03 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T14:11:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:03 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T14:11:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:03 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T14:11:04.407 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-mon[30820]: pgmap v14119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:04.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:04 smithi116 ceph-mon[35411]: pgmap v14119: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:04.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:11:04.404+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:11:04.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:11:04.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:11:04.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:11:04.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:11:04.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:11:04.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:11:04.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:11:04.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:11:04.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:11:04.670 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:11:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:11:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:11:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:11:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:11:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:11:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:11:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:11:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:06 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:11:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:11:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:11:06.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:11:06.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:11:06.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:11:06.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:11:06.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:11:06.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:11:06.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:11:06.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:11:06.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:11:06.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:11:06.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:11:06.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:06 smithi114 ceph-mon[30820]: pgmap v14120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:11:06.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:06 smithi114 ceph-mon[30820]: pgmap v14121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:11:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:06 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:11:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:11:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:11:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:11:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:11:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:11:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:11:06.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:11:06.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:11:06.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:11:06.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:06 smithi116 ceph-mon[35411]: pgmap v14120: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:11:06.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:06 smithi116 ceph-mon[35411]: pgmap v14121: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:11:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:08 smithi114 ceph-mon[30820]: pgmap v14122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:11:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:08 smithi116 ceph-mon[35411]: pgmap v14122: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:11:10.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:10 smithi114 ceph-mon[30820]: pgmap v14123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:11:10.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:10 smithi116 ceph-mon[35411]: pgmap v14123: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:11:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:12 smithi114 ceph-mon[30820]: pgmap v14124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:11:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:12 smithi116 ceph-mon[35411]: pgmap v14124: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:11:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:13 smithi114 ceph-mon[30820]: pgmap v14125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:11:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:13 smithi116 ceph-mon[35411]: pgmap v14125: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:11:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:15 smithi114 ceph-mon[30820]: pgmap v14126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:11:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:15 smithi116 ceph-mon[35411]: pgmap v14126: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:11:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:17 smithi114 ceph-mon[30820]: pgmap v14127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:17 smithi116 ceph-mon[35411]: pgmap v14127: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:19 smithi114 ceph-mon[30820]: pgmap v14128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:19 smithi116 ceph-mon[35411]: pgmap v14128: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:21 smithi114 ceph-mon[30820]: pgmap v14129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:11:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:21 smithi116 ceph-mon[35411]: pgmap v14129: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:11:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:23 smithi114 ceph-mon[30820]: pgmap v14130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:11:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:23 smithi116 ceph-mon[35411]: pgmap v14130: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:11:25.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:25 smithi114 ceph-mon[30820]: pgmap v14131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:11:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:25 smithi116 ceph-mon[35411]: pgmap v14131: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:11:27.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:27 smithi114 ceph-mon[30820]: pgmap v14132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:27 smithi116 ceph-mon[35411]: pgmap v14132: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:29.920 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:29 smithi114 ceph-mon[30820]: pgmap v14133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:29 smithi116 ceph-mon[35411]: pgmap v14133: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:31 smithi114 ceph-mon[30820]: pgmap v14134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:11:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:31 smithi116 ceph-mon[35411]: pgmap v14134: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:11:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:33 smithi114 ceph-mon[30820]: pgmap v14135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:11:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:33 smithi116 ceph-mon[35411]: pgmap v14135: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:11:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:35 smithi114 ceph-mon[30820]: pgmap v14136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:11:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:35 smithi116 ceph-mon[35411]: pgmap v14136: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:11:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:37 smithi114 ceph-mon[30820]: pgmap v14137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:37 smithi116 ceph-mon[35411]: pgmap v14137: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:39 smithi114 ceph-mon[30820]: pgmap v14138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:39 smithi116 ceph-mon[35411]: pgmap v14138: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:41 smithi114 ceph-mon[30820]: pgmap v14139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:11:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:41 smithi116 ceph-mon[35411]: pgmap v14139: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:11:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:43 smithi114 ceph-mon[30820]: pgmap v14140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:11:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:43 smithi116 ceph-mon[35411]: pgmap v14140: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:11:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:45 smithi114 ceph-mon[30820]: pgmap v14141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:11:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:45 smithi116 ceph-mon[35411]: pgmap v14141: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:11:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:47 smithi114 ceph-mon[30820]: pgmap v14142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:47 smithi116 ceph-mon[35411]: pgmap v14142: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:49 smithi114 ceph-mon[30820]: pgmap v14143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:49 smithi116 ceph-mon[35411]: pgmap v14143: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:51 smithi114 ceph-mon[30820]: pgmap v14144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:11:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:51 smithi116 ceph-mon[35411]: pgmap v14144: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:11:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:53 smithi114 ceph-mon[30820]: pgmap v14145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:11:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:53 smithi116 ceph-mon[35411]: pgmap v14145: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:11:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:55 smithi114 ceph-mon[30820]: pgmap v14146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:11:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:55 smithi116 ceph-mon[35411]: pgmap v14146: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:11:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:57 smithi114 ceph-mon[30820]: pgmap v14147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:57 smithi116 ceph-mon[35411]: pgmap v14147: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:11:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:11:59 smithi114 ceph-mon[30820]: pgmap v14148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:11:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:11:59 smithi116 ceph-mon[35411]: pgmap v14148: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:01 smithi114 ceph-mon[30820]: pgmap v14149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:01 smithi116 ceph-mon[35411]: pgmap v14149: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:03 smithi114 ceph-mon[30820]: pgmap v14150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:03 smithi116 ceph-mon[35411]: pgmap v14150: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:04.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:04 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:12:04.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:04 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:12:06.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:06 smithi114 ceph-mon[30820]: pgmap v14151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:12:06.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:06 smithi116 ceph-mon[35411]: pgmap v14151: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:12:08.130 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:12:07.764+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:12:08.130 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:12:08.130 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:12:08.130 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:12:08.131 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:12:08.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:12:08.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:12:08.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:12:08.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:12:08.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:12:08.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:12:08.132 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:12:07 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:12:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:08 smithi114 ceph-mon[30820]: pgmap v14152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:12:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:12:08.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:12:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:08 smithi116 ceph-mon[35411]: pgmap v14152: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:12:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:12:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:12:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:09 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:12:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:12:09.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:09 smithi114 ceph-mon[30820]: pgmap v14153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:12:09.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:09 smithi114 ceph-mon[30820]: pgmap v14154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:12:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:09 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:12:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:12:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:12:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:12:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:12:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:12:09.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:09 smithi116 ceph-mon[35411]: pgmap v14153: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:09 smithi116 ceph-mon[35411]: pgmap v14154: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:12:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:11 smithi114 ceph-mon[30820]: pgmap v14155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:12:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:11 smithi116 ceph-mon[35411]: pgmap v14155: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:12:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:13 smithi114 ceph-mon[30820]: pgmap v14156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T14:12:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:13 smithi116 ceph-mon[35411]: pgmap v14156: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T14:12:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:15 smithi114 ceph-mon[30820]: pgmap v14157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T14:12:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:15 smithi116 ceph-mon[35411]: pgmap v14157: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4 op/s 2024-09-19T14:12:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:17 smithi114 ceph-mon[30820]: pgmap v14158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T14:12:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:17 smithi116 ceph-mon[35411]: pgmap v14158: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 6 op/s 2024-09-19T14:12:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:19 smithi114 ceph-mon[30820]: pgmap v14159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:12:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:19 smithi116 ceph-mon[35411]: pgmap v14159: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:12:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:21 smithi114 ceph-mon[30820]: pgmap v14160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:12:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:21 smithi116 ceph-mon[35411]: pgmap v14160: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:12:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:23 smithi114 ceph-mon[30820]: pgmap v14161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:23 smithi116 ceph-mon[35411]: pgmap v14161: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:25 smithi114 ceph-mon[30820]: pgmap v14162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:25 smithi116 ceph-mon[35411]: pgmap v14162: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:27 smithi114 ceph-mon[30820]: pgmap v14163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:27 smithi116 ceph-mon[35411]: pgmap v14163: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:29 smithi116 ceph-mon[35411]: pgmap v14164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:29 smithi114 ceph-mon[30820]: pgmap v14164: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:31 smithi116 ceph-mon[35411]: pgmap v14165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:12:31.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:31 smithi114 ceph-mon[30820]: pgmap v14165: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:12:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:33 smithi116 ceph-mon[35411]: pgmap v14166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:33 smithi114 ceph-mon[30820]: pgmap v14166: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:35 smithi116 ceph-mon[35411]: pgmap v14167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:35 smithi114 ceph-mon[30820]: pgmap v14167: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:36 smithi114 ceph-mon[30820]: pgmap v14168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:36 smithi116 ceph-mon[35411]: pgmap v14168: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:38 smithi114 ceph-mon[30820]: pgmap v14169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:38 smithi116 ceph-mon[35411]: pgmap v14169: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:41.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:40 smithi114 ceph-mon[30820]: pgmap v14170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:12:41.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:40 smithi116 ceph-mon[35411]: pgmap v14170: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:12:43.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:42 smithi114 ceph-mon[30820]: pgmap v14171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:43.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:42 smithi116 ceph-mon[35411]: pgmap v14171: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:45.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:44 smithi114 ceph-mon[30820]: pgmap v14172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:45.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:44 smithi116 ceph-mon[35411]: pgmap v14172: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:47.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:46 smithi114 ceph-mon[30820]: pgmap v14173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:47.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:46 smithi116 ceph-mon[35411]: pgmap v14173: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:49.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:48 smithi114 ceph-mon[30820]: pgmap v14174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:49.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:48 smithi116 ceph-mon[35411]: pgmap v14174: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:51.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:50 smithi114 ceph-mon[30820]: pgmap v14175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:12:51.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:50 smithi116 ceph-mon[35411]: pgmap v14175: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:12:53.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:52 smithi114 ceph-mon[30820]: pgmap v14176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:53.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:52 smithi116 ceph-mon[35411]: pgmap v14176: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:12:55.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:54 smithi114 ceph-mon[30820]: pgmap v14177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:55.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:54 smithi116 ceph-mon[35411]: pgmap v14177: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:56 smithi114 ceph-mon[30820]: pgmap v14178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:57.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:56 smithi116 ceph-mon[35411]: pgmap v14178: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:12:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:12:58 smithi114 ceph-mon[30820]: pgmap v14179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:12:59.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:12:58 smithi116 ceph-mon[35411]: pgmap v14179: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:00 smithi114 ceph-mon[30820]: pgmap v14180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:01.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:00 smithi116 ceph-mon[35411]: pgmap v14180: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:02 smithi114 ceph-mon[30820]: pgmap v14181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:02 smithi116 ceph-mon[35411]: pgmap v14181: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:04 smithi114 ceph-mon[30820]: pgmap v14182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:04 smithi116 ceph-mon[35411]: pgmap v14182: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:07 smithi114 ceph-mon[30820]: pgmap v14183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:13:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:07 smithi116 ceph-mon[35411]: pgmap v14183: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:13:08.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:08 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:13:08.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:08 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:13:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:09 smithi114 ceph-mon[30820]: pgmap v14184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:09 smithi116 ceph-mon[35411]: pgmap v14184: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-mon[30820]: pgmap v14185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:13:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:13:11.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:13:11.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:13:11.124+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:13:11.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:13:11.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:13:11.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:13:11.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:13:11.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:13:11.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:13:11.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:13:11.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:13:11.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:13:11 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:13:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:11 smithi116 ceph-mon[35411]: pgmap v14185: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:13:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:13:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:11 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:13:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:12 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:13:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:13:12.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:12 smithi114 ceph-mon[30820]: pgmap v14186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:13:12.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:12 smithi114 ceph-mon[30820]: pgmap v14187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:13:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:12 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:13:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:13:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:13:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:13:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:13:12.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:13:12.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:13:12.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:13:12.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:13:12.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:13:12.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:13:12.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:12 smithi116 ceph-mon[35411]: pgmap v14186: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:13:12.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:12 smithi116 ceph-mon[35411]: pgmap v14187: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:13:14.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:14 smithi114 ceph-mon[30820]: pgmap v14188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:13:14.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:14 smithi116 ceph-mon[35411]: pgmap v14188: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:13:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:16 smithi114 ceph-mon[30820]: pgmap v14189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:13:16.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:16 smithi116 ceph-mon[35411]: pgmap v14189: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:13:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:18 smithi114 ceph-mon[30820]: pgmap v14190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:13:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:18 smithi116 ceph-mon[35411]: pgmap v14190: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:13:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:20 smithi116 ceph-mon[35411]: pgmap v14191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:13:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:20 smithi114 ceph-mon[30820]: pgmap v14191: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:13:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:22 smithi116 ceph-mon[35411]: pgmap v14192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:13:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:22 smithi114 ceph-mon[30820]: pgmap v14192: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:13:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:24 smithi116 ceph-mon[35411]: pgmap v14193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:24 smithi114 ceph-mon[30820]: pgmap v14193: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:26 smithi116 ceph-mon[35411]: pgmap v14194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:26.470 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:26 smithi114 ceph-mon[30820]: pgmap v14194: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:28 smithi114 ceph-mon[30820]: pgmap v14195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:28 smithi116 ceph-mon[35411]: pgmap v14195: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:30 smithi114 ceph-mon[30820]: pgmap v14196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:30 smithi116 ceph-mon[35411]: pgmap v14196: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:31 smithi114 ceph-mon[30820]: pgmap v14197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:13:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:31 smithi116 ceph-mon[35411]: pgmap v14197: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:13:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:33 smithi114 ceph-mon[30820]: pgmap v14198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:33 smithi116 ceph-mon[35411]: pgmap v14198: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:35 smithi114 ceph-mon[30820]: pgmap v14199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:35 smithi116 ceph-mon[35411]: pgmap v14199: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:37 smithi114 ceph-mon[30820]: pgmap v14200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:37 smithi116 ceph-mon[35411]: pgmap v14200: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:39 smithi114 ceph-mon[30820]: pgmap v14201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:39 smithi116 ceph-mon[35411]: pgmap v14201: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:41 smithi114 ceph-mon[30820]: pgmap v14202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:13:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:41 smithi116 ceph-mon[35411]: pgmap v14202: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:13:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:43 smithi114 ceph-mon[30820]: pgmap v14203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:43 smithi116 ceph-mon[35411]: pgmap v14203: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:45 smithi114 ceph-mon[30820]: pgmap v14204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:45 smithi116 ceph-mon[35411]: pgmap v14204: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:47 smithi114 ceph-mon[30820]: pgmap v14205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:47 smithi116 ceph-mon[35411]: pgmap v14205: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:49 smithi114 ceph-mon[30820]: pgmap v14206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:49 smithi116 ceph-mon[35411]: pgmap v14206: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:51 smithi114 ceph-mon[30820]: pgmap v14207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:13:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:51 smithi116 ceph-mon[35411]: pgmap v14207: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:13:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:53 smithi114 ceph-mon[30820]: pgmap v14208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:53 smithi116 ceph-mon[35411]: pgmap v14208: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:55 smithi114 ceph-mon[30820]: pgmap v14209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:55.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:55 smithi116 ceph-mon[35411]: pgmap v14209: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:13:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:57 smithi114 ceph-mon[30820]: pgmap v14210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:57 smithi116 ceph-mon[35411]: pgmap v14210: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:13:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:13:59 smithi114 ceph-mon[30820]: pgmap v14211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:13:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:13:59 smithi116 ceph-mon[35411]: pgmap v14211: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:01 smithi114 ceph-mon[30820]: pgmap v14212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:01 smithi116 ceph-mon[35411]: pgmap v14212: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:03 smithi114 ceph-mon[30820]: pgmap v14213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:03 smithi116 ceph-mon[35411]: pgmap v14213: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:05 smithi114 ceph-mon[30820]: pgmap v14214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:14:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:05 smithi116 ceph-mon[35411]: pgmap v14214: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:14:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:07 smithi114 ceph-mon[30820]: pgmap v14215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:07 smithi116 ceph-mon[35411]: pgmap v14215: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:09 smithi114 ceph-mon[30820]: pgmap v14216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:09 smithi116 ceph-mon[35411]: pgmap v14216: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:11 smithi114 ceph-mon[30820]: pgmap v14217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:11 smithi116 ceph-mon[35411]: pgmap v14217: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:12.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:12 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:14:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:12 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:14:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:13 smithi114 ceph-mon[30820]: pgmap v14218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:13 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:13 smithi116 ceph-mon[35411]: pgmap v14218: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:13 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:14:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:14.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:14:14.484+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:14:14.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:14:14.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:14:14.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:14:14.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:14:14.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:14:14.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:14:14.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:14:14.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:14:14.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:14:14 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:14:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:14:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:14:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:14:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:15 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:14:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:14:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:14:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:14:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:14:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:14:16.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:14:16.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:14:16.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:14:16.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:14:16.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:14:16.170 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:14:16.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:15 smithi114 ceph-mon[30820]: pgmap v14219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:14:16.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:15 smithi114 ceph-mon[30820]: pgmap v14220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:15 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:14:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:14:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:14:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:14:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:14:16.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:14:16.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:14:16.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:14:16.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:14:16.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:14:16.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:14:16.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:14:16.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:15 smithi116 ceph-mon[35411]: pgmap v14219: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:14:16.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:15 smithi116 ceph-mon[35411]: pgmap v14220: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:17 smithi114 ceph-mon[30820]: pgmap v14221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:17 smithi116 ceph-mon[35411]: pgmap v14221: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:19 smithi114 ceph-mon[30820]: pgmap v14222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:19 smithi116 ceph-mon[35411]: pgmap v14222: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:21 smithi114 ceph-mon[30820]: pgmap v14223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:21 smithi116 ceph-mon[35411]: pgmap v14223: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:23 smithi114 ceph-mon[30820]: pgmap v14224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:23 smithi116 ceph-mon[35411]: pgmap v14224: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:14:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:25 smithi114 ceph-mon[30820]: pgmap v14225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:14:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:25 smithi116 ceph-mon[35411]: pgmap v14225: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:14:27.970 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:27 smithi114 ceph-mon[30820]: pgmap v14226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:27 smithi116 ceph-mon[35411]: pgmap v14226: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:29 smithi114 ceph-mon[30820]: pgmap v14227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:14:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:29 smithi116 ceph-mon[35411]: pgmap v14227: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:14:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:31 smithi114 ceph-mon[30820]: pgmap v14228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:31 smithi116 ceph-mon[35411]: pgmap v14228: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:33 smithi114 ceph-mon[30820]: pgmap v14229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:14:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:33 smithi116 ceph-mon[35411]: pgmap v14229: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:14:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:35 smithi114 ceph-mon[30820]: pgmap v14230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:14:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:35 smithi116 ceph-mon[35411]: pgmap v14230: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:14:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:37 smithi114 ceph-mon[30820]: pgmap v14231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:37 smithi116 ceph-mon[35411]: pgmap v14231: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:39 smithi114 ceph-mon[30820]: pgmap v14232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:39 smithi116 ceph-mon[35411]: pgmap v14232: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:41 smithi114 ceph-mon[30820]: pgmap v14233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:41 smithi116 ceph-mon[35411]: pgmap v14233: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:43 smithi114 ceph-mon[30820]: pgmap v14234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:43 smithi116 ceph-mon[35411]: pgmap v14234: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:45 smithi114 ceph-mon[30820]: pgmap v14235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:14:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:45 smithi116 ceph-mon[35411]: pgmap v14235: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:14:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:47 smithi114 ceph-mon[30820]: pgmap v14236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:47 smithi116 ceph-mon[35411]: pgmap v14236: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:49 smithi114 ceph-mon[30820]: pgmap v14237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:49 smithi116 ceph-mon[35411]: pgmap v14237: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:51 smithi114 ceph-mon[30820]: pgmap v14238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:51 smithi116 ceph-mon[35411]: pgmap v14238: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:14:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:53 smithi114 ceph-mon[30820]: pgmap v14239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:53 smithi116 ceph-mon[35411]: pgmap v14239: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:14:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:55 smithi114 ceph-mon[30820]: pgmap v14240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:14:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:55 smithi116 ceph-mon[35411]: pgmap v14240: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:14:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:57 smithi114 ceph-mon[30820]: pgmap v14241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:14:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:57 smithi116 ceph-mon[35411]: pgmap v14241: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:14:59 smithi114 ceph-mon[30820]: pgmap v14242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:14:59 smithi116 ceph-mon[35411]: pgmap v14242: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:01 smithi114 ceph-mon[30820]: pgmap v14243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:01 smithi116 ceph-mon[35411]: pgmap v14243: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:03 smithi114 ceph-mon[30820]: pgmap v14244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:03 smithi116 ceph-mon[35411]: pgmap v14244: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:06.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:05 smithi114 ceph-mon[30820]: pgmap v14245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:06.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:05 smithi116 ceph-mon[35411]: pgmap v14245: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:08.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:07 smithi114 ceph-mon[30820]: pgmap v14246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:07 smithi116 ceph-mon[35411]: pgmap v14246: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:09 smithi114 ceph-mon[30820]: pgmap v14247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:09 smithi116 ceph-mon[35411]: pgmap v14247: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:11 smithi114 ceph-mon[30820]: pgmap v14248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:11 smithi116 ceph-mon[35411]: pgmap v14248: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:13 smithi114 ceph-mon[30820]: pgmap v14249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:13 smithi116 ceph-mon[35411]: pgmap v14249: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:15.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:14 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:15:15.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:14 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:15:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:15 smithi114 ceph-mon[30820]: pgmap v14250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:16.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:15 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:15 smithi116 ceph-mon[35411]: pgmap v14250: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:15 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-mon[30820]: pgmap v14251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:15:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:15:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:15:17.780+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:15:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:15:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:15:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:15:18.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:15:18.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:15:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:15:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:15:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:15:18.170 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:15:17 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:15:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:17 smithi116 ceph-mon[35411]: pgmap v14251: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:15:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:15:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:17 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:15:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:18 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:15:19.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:18 smithi114 ceph-mon[30820]: pgmap v14252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:15:19.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:18 smithi114 ceph-mon[30820]: pgmap v14253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:15:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:18 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:15:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:15:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:15:19.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:18 smithi116 ceph-mon[35411]: pgmap v14252: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:15:19.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:18 smithi116 ceph-mon[35411]: pgmap v14253: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:15:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:20 smithi114 ceph-mon[30820]: pgmap v14254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:15:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:20 smithi116 ceph-mon[35411]: pgmap v14254: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:15:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:22 smithi114 ceph-mon[30820]: pgmap v14255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:15:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:22 smithi116 ceph-mon[35411]: pgmap v14255: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:15:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:24 smithi114 ceph-mon[30820]: pgmap v14256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:15:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:24 smithi116 ceph-mon[35411]: pgmap v14256: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:15:27.220 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:26 smithi114 ceph-mon[30820]: pgmap v14257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:15:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:26 smithi116 ceph-mon[35411]: pgmap v14257: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:15:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:28 smithi114 ceph-mon[30820]: pgmap v14258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:15:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:28 smithi116 ceph-mon[35411]: pgmap v14258: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:15:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:30 smithi114 ceph-mon[30820]: pgmap v14259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:30 smithi116 ceph-mon[35411]: pgmap v14259: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:32 smithi114 ceph-mon[30820]: pgmap v14260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:32 smithi116 ceph-mon[35411]: pgmap v14260: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:34 smithi114 ceph-mon[30820]: pgmap v14261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:34 smithi116 ceph-mon[35411]: pgmap v14261: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:36 smithi114 ceph-mon[30820]: pgmap v14262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:36 smithi116 ceph-mon[35411]: pgmap v14262: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:39.231 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:38 smithi116 ceph-mon[35411]: pgmap v14263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:38 smithi114 ceph-mon[30820]: pgmap v14263: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:40 smithi114 ceph-mon[30820]: pgmap v14264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:40 smithi116 ceph-mon[35411]: pgmap v14264: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:42 smithi114 ceph-mon[30820]: pgmap v14265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:42 smithi116 ceph-mon[35411]: pgmap v14265: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:44 smithi114 ceph-mon[30820]: pgmap v14266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:44 smithi116 ceph-mon[35411]: pgmap v14266: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:46 smithi114 ceph-mon[30820]: pgmap v14267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:46 smithi116 ceph-mon[35411]: pgmap v14267: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:48 smithi114 ceph-mon[30820]: pgmap v14268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:48 smithi116 ceph-mon[35411]: pgmap v14268: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:50 smithi114 ceph-mon[30820]: pgmap v14269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:50 smithi116 ceph-mon[35411]: pgmap v14269: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:15:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:52 smithi114 ceph-mon[30820]: pgmap v14270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:52 smithi116 ceph-mon[35411]: pgmap v14270: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:15:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:54 smithi114 ceph-mon[30820]: pgmap v14271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:54 smithi116 ceph-mon[35411]: pgmap v14271: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:57.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:57 smithi114 ceph-mon[30820]: pgmap v14272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:57 smithi116 ceph-mon[35411]: pgmap v14272: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:15:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:15:59 smithi114 ceph-mon[30820]: pgmap v14273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:15:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:15:59 smithi116 ceph-mon[35411]: pgmap v14273: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:01 smithi114 ceph-mon[30820]: pgmap v14274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:01 smithi116 ceph-mon[35411]: pgmap v14274: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:03 smithi114 ceph-mon[30820]: pgmap v14275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:03 smithi116 ceph-mon[35411]: pgmap v14275: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:05 smithi114 ceph-mon[30820]: pgmap v14276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:05 smithi116 ceph-mon[35411]: pgmap v14276: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:07 smithi116 ceph-mon[35411]: pgmap v14277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:16:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:07 smithi114 ceph-mon[30820]: pgmap v14277: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:16:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:09 smithi116 ceph-mon[35411]: pgmap v14278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:09 smithi114 ceph-mon[30820]: pgmap v14278: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:11 smithi114 ceph-mon[30820]: pgmap v14279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:11 smithi116 ceph-mon[35411]: pgmap v14279: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:13 smithi114 ceph-mon[30820]: pgmap v14280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:13 smithi116 ceph-mon[35411]: pgmap v14280: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:15 smithi114 ceph-mon[30820]: pgmap v14281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:15 smithi116 ceph-mon[35411]: pgmap v14281: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:17 smithi114 ceph-mon[30820]: pgmap v14282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:16:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:17 smithi116 ceph-mon[35411]: pgmap v14282: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:16:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:18 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:16:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:18 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:16:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:19 smithi114 ceph-mon[30820]: pgmap v14283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:19 smithi116 ceph-mon[35411]: pgmap v14283: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-mon[30820]: pgmap v14284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:16:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:16:21.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:16:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:16:21.141+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:16:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:16:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:16:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:16:21.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:16:21.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:16:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:16:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:16:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:16:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:16:21.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:16:21 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:16:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:21 smithi116 ceph-mon[35411]: pgmap v14284: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:16:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:16:21.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:16:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:22 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:16:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:16:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:16:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:16:22.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:16:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:16:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:16:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:16:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:16:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:16:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:16:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:22 smithi114 ceph-mon[30820]: pgmap v14285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:16:22.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:22 smithi114 ceph-mon[30820]: pgmap v14286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:16:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:22 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:16:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:16:22.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:22 smithi116 ceph-mon[35411]: pgmap v14285: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:16:22.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:22 smithi116 ceph-mon[35411]: pgmap v14286: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:16:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:24 smithi114 ceph-mon[30820]: pgmap v14287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:16:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:24 smithi116 ceph-mon[35411]: pgmap v14287: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:16:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:26 smithi114 ceph-mon[30820]: pgmap v14288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:16:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:26 smithi116 ceph-mon[35411]: pgmap v14288: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:16:28.469 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:28 smithi114 ceph-mon[30820]: pgmap v14289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:16:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:28 smithi116 ceph-mon[35411]: pgmap v14289: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:16:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:29 smithi114 ceph-mon[30820]: pgmap v14290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:16:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:29 smithi116 ceph-mon[35411]: pgmap v14290: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:16:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:32 smithi114 ceph-mon[30820]: pgmap v14291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:16:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:32 smithi116 ceph-mon[35411]: pgmap v14291: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:16:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:34 smithi114 ceph-mon[30820]: pgmap v14292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:34 smithi116 ceph-mon[35411]: pgmap v14292: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:36 smithi114 ceph-mon[30820]: pgmap v14293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:36 smithi116 ceph-mon[35411]: pgmap v14293: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:38 smithi114 ceph-mon[30820]: pgmap v14294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:38 smithi116 ceph-mon[35411]: pgmap v14294: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:40.482 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:40 smithi116 ceph-mon[35411]: pgmap v14295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:40 smithi114 ceph-mon[30820]: pgmap v14295: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:41 smithi114 ceph-mon[30820]: pgmap v14296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:16:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:41 smithi116 ceph-mon[35411]: pgmap v14296: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:16:44.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:44 smithi114 ceph-mon[30820]: pgmap v14297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:44 smithi116 ceph-mon[35411]: pgmap v14297: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:46.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:46 smithi114 ceph-mon[30820]: pgmap v14298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:46.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:46 smithi116 ceph-mon[35411]: pgmap v14298: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:48 smithi114 ceph-mon[30820]: pgmap v14299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:48.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:48 smithi116 ceph-mon[35411]: pgmap v14299: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:50.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:50 smithi114 ceph-mon[30820]: pgmap v14300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:50.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:50 smithi116 ceph-mon[35411]: pgmap v14300: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:51 smithi114 ceph-mon[30820]: pgmap v14301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:16:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:51 smithi116 ceph-mon[35411]: pgmap v14301: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:16:54.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:54 smithi114 ceph-mon[30820]: pgmap v14302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:54.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:54 smithi116 ceph-mon[35411]: pgmap v14302: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:56.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:56 smithi114 ceph-mon[30820]: pgmap v14303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:56.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:56 smithi116 ceph-mon[35411]: pgmap v14303: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:16:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:58 smithi114 ceph-mon[30820]: pgmap v14304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:58 smithi116 ceph-mon[35411]: pgmap v14304: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:16:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:16:59 smithi114 ceph-mon[30820]: pgmap v14305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:16:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:16:59 smithi116 ceph-mon[35411]: pgmap v14305: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:02 smithi114 ceph-mon[30820]: pgmap v14306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:02 smithi116 ceph-mon[35411]: pgmap v14306: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:04 smithi114 ceph-mon[30820]: pgmap v14307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:04 smithi116 ceph-mon[35411]: pgmap v14307: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:06 smithi114 ceph-mon[30820]: pgmap v14308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:17:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:06 smithi116 ceph-mon[35411]: pgmap v14308: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:17:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:07 smithi114 ceph-mon[30820]: pgmap v14309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:07 smithi116 ceph-mon[35411]: pgmap v14309: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:10 smithi114 ceph-mon[30820]: pgmap v14310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:10 smithi116 ceph-mon[35411]: pgmap v14310: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:12 smithi114 ceph-mon[30820]: pgmap v14311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:12 smithi116 ceph-mon[35411]: pgmap v14311: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:14 smithi114 ceph-mon[30820]: pgmap v14312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:14 smithi116 ceph-mon[35411]: pgmap v14312: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:15 smithi114 ceph-mon[30820]: pgmap v14313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:17:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:15 smithi116 ceph-mon[35411]: pgmap v14313: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:17:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:18 smithi114 ceph-mon[30820]: pgmap v14314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:18 smithi116 ceph-mon[35411]: pgmap v14314: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:20 smithi114 ceph-mon[30820]: pgmap v14315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:20 smithi116 ceph-mon[35411]: pgmap v14315: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:21 smithi114 ceph-mon[30820]: pgmap v14316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:21 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:17:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:21 smithi116 ceph-mon[35411]: pgmap v14316: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:21 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:17:24.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-mon[30820]: pgmap v14317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:24 smithi116 ceph-mon[35411]: pgmap v14317: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:17:24.501+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:17:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:17:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:17:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:17:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:17:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:17:24.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:17:24.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:17:24.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:17:24.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:17:24.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:17:24 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:17:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:17:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:17:25.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:25 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:17:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:17:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:17:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:25 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:17:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:26 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:17:26.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:26 smithi114 ceph-mon[30820]: pgmap v14318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:17:26.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:26 smithi114 ceph-mon[30820]: pgmap v14319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:26 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:17:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:17:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:17:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:17:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:17:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:17:26.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:17:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:17:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:17:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:17:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:26 smithi116 ceph-mon[35411]: pgmap v14318: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:17:26.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:26 smithi116 ceph-mon[35411]: pgmap v14319: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:27 smithi114 ceph-mon[30820]: pgmap v14320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:27 smithi116 ceph-mon[35411]: pgmap v14320: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:29 smithi114 ceph-mon[30820]: pgmap v14321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:29 smithi116 ceph-mon[35411]: pgmap v14321: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:31 smithi114 ceph-mon[30820]: pgmap v14322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:31 smithi116 ceph-mon[35411]: pgmap v14322: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:33 smithi114 ceph-mon[30820]: pgmap v14323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:33 smithi116 ceph-mon[35411]: pgmap v14323: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:17:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:35 smithi114 ceph-mon[30820]: pgmap v14324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:17:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:35 smithi116 ceph-mon[35411]: pgmap v14324: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:17:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:37 smithi114 ceph-mon[30820]: pgmap v14325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:37 smithi116 ceph-mon[35411]: pgmap v14325: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:39 smithi114 ceph-mon[30820]: pgmap v14326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:39 smithi116 ceph-mon[35411]: pgmap v14326: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:41 smithi114 ceph-mon[30820]: pgmap v14327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:41 smithi116 ceph-mon[35411]: pgmap v14327: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:43 smithi114 ceph-mon[30820]: pgmap v14328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:43 smithi116 ceph-mon[35411]: pgmap v14328: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:45 smithi114 ceph-mon[30820]: pgmap v14329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:17:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:45 smithi116 ceph-mon[35411]: pgmap v14329: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:17:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:47 smithi114 ceph-mon[30820]: pgmap v14330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:47 smithi116 ceph-mon[35411]: pgmap v14330: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:49.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:49 smithi114 ceph-mon[30820]: pgmap v14331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:49 smithi116 ceph-mon[35411]: pgmap v14331: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:51.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:51 smithi114 ceph-mon[30820]: pgmap v14332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:51 smithi116 ceph-mon[35411]: pgmap v14332: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:17:53.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:53 smithi114 ceph-mon[30820]: pgmap v14333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:53 smithi116 ceph-mon[35411]: pgmap v14333: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:55.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:55 smithi114 ceph-mon[30820]: pgmap v14334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:17:55.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:55 smithi116 ceph-mon[35411]: pgmap v14334: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:17:57.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:57 smithi114 ceph-mon[30820]: pgmap v14335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:57.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:57 smithi116 ceph-mon[35411]: pgmap v14335: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:17:59.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:17:59 smithi114 ceph-mon[30820]: pgmap v14336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:17:59.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:17:59 smithi116 ceph-mon[35411]: pgmap v14336: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:01.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:01 smithi114 ceph-mon[30820]: pgmap v14337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:01.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:01 smithi116 ceph-mon[35411]: pgmap v14337: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:03.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:03 smithi114 ceph-mon[30820]: pgmap v14338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:03.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:03 smithi116 ceph-mon[35411]: pgmap v14338: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:05.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:05 smithi114 ceph-mon[30820]: pgmap v14339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:05.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:05 smithi116 ceph-mon[35411]: pgmap v14339: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:07.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:07 smithi114 ceph-mon[30820]: pgmap v14340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:07.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:07 smithi116 ceph-mon[35411]: pgmap v14340: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:09.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:09 smithi114 ceph-mon[30820]: pgmap v14341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:09.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:09 smithi116 ceph-mon[35411]: pgmap v14341: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:11.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:11 smithi114 ceph-mon[30820]: pgmap v14342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:11.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:11 smithi116 ceph-mon[35411]: pgmap v14342: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:13.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:13 smithi114 ceph-mon[30820]: pgmap v14343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:13.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:13 smithi116 ceph-mon[35411]: pgmap v14343: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:15.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:15 smithi114 ceph-mon[30820]: pgmap v14344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:15.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:15 smithi116 ceph-mon[35411]: pgmap v14344: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:17.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:17 smithi114 ceph-mon[30820]: pgmap v14345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:17.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:17 smithi116 ceph-mon[35411]: pgmap v14345: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:19.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:19 smithi114 ceph-mon[30820]: pgmap v14346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:19.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:19 smithi116 ceph-mon[35411]: pgmap v14346: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:21.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:21 smithi114 ceph-mon[30820]: pgmap v14347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:21.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:21 smithi116 ceph-mon[35411]: pgmap v14347: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:23.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:23 smithi114 ceph-mon[30820]: pgmap v14348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:23.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:23 smithi116 ceph-mon[35411]: pgmap v14348: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:24 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:18:25.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:24 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:18:25.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:25 smithi116 ceph-mon[35411]: pgmap v14349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:25 smithi114 ceph-mon[30820]: pgmap v14349: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:27.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:27 smithi116 ceph-mon[35411]: pgmap v14350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-mon[30820]: pgmap v14350: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:18:27.862+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:18:28.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:18:28.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:18:27 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:18:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:28 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:18:28.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:28 smithi116 ceph-mon[35411]: pgmap v14351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:18:28.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:28 smithi116 ceph-mon[35411]: pgmap v14352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:18:28.932 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:18:29.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:28 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:18:29.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:18:29.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:18:29.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:18:29.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:18:29.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:18:29.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:18:29.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:28 smithi114 ceph-mon[30820]: pgmap v14351: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:28 smithi114 ceph-mon[30820]: pgmap v14352: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:18:29.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:18:31.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:30 smithi116 ceph-mon[35411]: pgmap v14353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:18:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:30 smithi114 ceph-mon[30820]: pgmap v14353: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:18:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:32 smithi114 ceph-mon[30820]: pgmap v14354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:18:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:32 smithi116 ceph-mon[35411]: pgmap v14354: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:18:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:34 smithi114 ceph-mon[30820]: pgmap v14355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:18:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:34 smithi116 ceph-mon[35411]: pgmap v14355: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:18:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:36 smithi114 ceph-mon[30820]: pgmap v14356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:18:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:36 smithi116 ceph-mon[35411]: pgmap v14356: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:18:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:38 smithi114 ceph-mon[30820]: pgmap v14357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:18:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:38 smithi116 ceph-mon[35411]: pgmap v14357: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:18:41.232 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:40 smithi116 ceph-mon[35411]: pgmap v14358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:40 smithi114 ceph-mon[30820]: pgmap v14358: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:42 smithi114 ceph-mon[30820]: pgmap v14359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:42 smithi116 ceph-mon[35411]: pgmap v14359: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:44 smithi114 ceph-mon[30820]: pgmap v14360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:44 smithi116 ceph-mon[35411]: pgmap v14360: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:46 smithi114 ceph-mon[30820]: pgmap v14361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:46 smithi116 ceph-mon[35411]: pgmap v14361: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:48 smithi114 ceph-mon[30820]: pgmap v14362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:48 smithi116 ceph-mon[35411]: pgmap v14362: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:50 smithi114 ceph-mon[30820]: pgmap v14363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:50 smithi116 ceph-mon[35411]: pgmap v14363: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:18:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:52 smithi114 ceph-mon[30820]: pgmap v14364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:52 smithi116 ceph-mon[35411]: pgmap v14364: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:18:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:54 smithi114 ceph-mon[30820]: pgmap v14365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:54 smithi116 ceph-mon[35411]: pgmap v14365: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:56 smithi114 ceph-mon[30820]: pgmap v14366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:56 smithi116 ceph-mon[35411]: pgmap v14366: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:18:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:18:58 smithi114 ceph-mon[30820]: pgmap v14367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:18:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:18:58 smithi116 ceph-mon[35411]: pgmap v14367: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:00 smithi114 ceph-mon[30820]: pgmap v14368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:19:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:00 smithi116 ceph-mon[35411]: pgmap v14368: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:19:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:02 smithi114 ceph-mon[30820]: pgmap v14369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:19:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:02 smithi116 ceph-mon[35411]: pgmap v14369: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:19:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:04 smithi114 ceph-mon[30820]: pgmap v14370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:04 smithi116 ceph-mon[35411]: pgmap v14370: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:06 smithi114 ceph-mon[30820]: pgmap v14371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:19:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:06 smithi116 ceph-mon[35411]: pgmap v14371: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:19:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:08 smithi114 ceph-mon[30820]: pgmap v14372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:08 smithi116 ceph-mon[35411]: pgmap v14372: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:11 smithi114 ceph-mon[30820]: pgmap v14373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:19:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:11 smithi116 ceph-mon[35411]: pgmap v14373: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:19:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:13 smithi114 ceph-mon[30820]: pgmap v14374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:19:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:13 smithi116 ceph-mon[35411]: pgmap v14374: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:19:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:15 smithi114 ceph-mon[30820]: pgmap v14375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:15 smithi116 ceph-mon[35411]: pgmap v14375: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:17 smithi114 ceph-mon[30820]: pgmap v14376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:19:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:17 smithi116 ceph-mon[35411]: pgmap v14376: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:19:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:19 smithi114 ceph-mon[30820]: pgmap v14377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:19 smithi116 ceph-mon[35411]: pgmap v14377: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:21 smithi114 ceph-mon[30820]: pgmap v14378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:19:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:21 smithi116 ceph-mon[35411]: pgmap v14378: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:19:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:23 smithi114 ceph-mon[30820]: pgmap v14379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:19:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:23 smithi116 ceph-mon[35411]: pgmap v14379: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:19:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:25 smithi114 ceph-mon[30820]: pgmap v14380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:25 smithi116 ceph-mon[35411]: pgmap v14380: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:27 smithi114 ceph-mon[30820]: pgmap v14381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:19:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:27 smithi116 ceph-mon[35411]: pgmap v14381: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:19:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:28 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:19:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:28 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:19:29.318 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:29 smithi114 ceph-mon[30820]: pgmap v14382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:29 smithi116 ceph-mon[35411]: pgmap v14382: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-mon[30820]: pgmap v14383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:19:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:19:31.220+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:19:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:19:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:19:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:19:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:19:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:19:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:19:31.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:19:31.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:19:31.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:19:31.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:19:31 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:19:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:31 smithi116 ceph-mon[35411]: pgmap v14383: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:19:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:32 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:19:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:19:32.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:32 smithi114 ceph-mon[30820]: pgmap v14384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:32 smithi114 ceph-mon[30820]: pgmap v14385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:19:32.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:19:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:32 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:19:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:19:32.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:32 smithi116 ceph-mon[35411]: pgmap v14384: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:32 smithi116 ceph-mon[35411]: pgmap v14385: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:19:32.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:19:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:33 smithi114 ceph-mon[30820]: pgmap v14386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:19:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:33 smithi116 ceph-mon[35411]: pgmap v14386: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:19:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:35 smithi114 ceph-mon[30820]: pgmap v14387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:19:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:35 smithi116 ceph-mon[35411]: pgmap v14387: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:19:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:37 smithi114 ceph-mon[30820]: pgmap v14388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:19:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:37 smithi116 ceph-mon[35411]: pgmap v14388: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:19:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:39 smithi114 ceph-mon[30820]: pgmap v14389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:19:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:39 smithi116 ceph-mon[35411]: pgmap v14389: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:19:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:41 smithi114 ceph-mon[30820]: pgmap v14390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:19:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:41 smithi116 ceph-mon[35411]: pgmap v14390: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:19:43.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:43 smithi114 ceph-mon[30820]: pgmap v14391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:43 smithi116 ceph-mon[35411]: pgmap v14391: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:45 smithi114 ceph-mon[30820]: pgmap v14392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:19:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:45 smithi116 ceph-mon[35411]: pgmap v14392: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:19:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:47 smithi114 ceph-mon[30820]: pgmap v14393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:19:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:47 smithi116 ceph-mon[35411]: pgmap v14393: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:19:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:49 smithi114 ceph-mon[30820]: pgmap v14394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:49 smithi116 ceph-mon[35411]: pgmap v14394: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:51 smithi114 ceph-mon[30820]: pgmap v14395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:19:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:51 smithi116 ceph-mon[35411]: pgmap v14395: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:19:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:53 smithi114 ceph-mon[30820]: pgmap v14396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:53 smithi116 ceph-mon[35411]: pgmap v14396: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:55 smithi114 ceph-mon[30820]: pgmap v14397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:19:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:55 smithi116 ceph-mon[35411]: pgmap v14397: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:19:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:57 smithi114 ceph-mon[30820]: pgmap v14398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:19:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:57 smithi116 ceph-mon[35411]: pgmap v14398: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:19:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:19:59 smithi114 ceph-mon[30820]: pgmap v14399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:19:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:19:59 smithi116 ceph-mon[35411]: pgmap v14399: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T14:20:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T14:20:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:01 smithi114 ceph-mon[30820]: pgmap v14400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:01 smithi116 ceph-mon[35411]: pgmap v14400: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:03 smithi114 ceph-mon[30820]: pgmap v14401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:03 smithi116 ceph-mon[35411]: pgmap v14401: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:05 smithi114 ceph-mon[30820]: pgmap v14402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:20:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:05 smithi116 ceph-mon[35411]: pgmap v14402: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:20:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:07 smithi114 ceph-mon[30820]: pgmap v14403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:20:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:07 smithi116 ceph-mon[35411]: pgmap v14403: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:20:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:09 smithi114 ceph-mon[30820]: pgmap v14404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:09 smithi116 ceph-mon[35411]: pgmap v14404: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:11 smithi114 ceph-mon[30820]: pgmap v14405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:11 smithi116 ceph-mon[35411]: pgmap v14405: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:13 smithi114 ceph-mon[30820]: pgmap v14406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:13 smithi116 ceph-mon[35411]: pgmap v14406: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:15 smithi114 ceph-mon[30820]: pgmap v14407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:20:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:15 smithi116 ceph-mon[35411]: pgmap v14407: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:20:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:17 smithi114 ceph-mon[30820]: pgmap v14408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:20:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:17 smithi116 ceph-mon[35411]: pgmap v14408: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:20:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:19 smithi114 ceph-mon[30820]: pgmap v14409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:19 smithi116 ceph-mon[35411]: pgmap v14409: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:21.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:21 smithi114 ceph-mon[30820]: pgmap v14410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:21 smithi116 ceph-mon[35411]: pgmap v14410: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:23 smithi114 ceph-mon[30820]: pgmap v14411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:23 smithi116 ceph-mon[35411]: pgmap v14411: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:25 smithi114 ceph-mon[30820]: pgmap v14412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:20:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:25 smithi116 ceph-mon[35411]: pgmap v14412: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:20:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:27 smithi114 ceph-mon[30820]: pgmap v14413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:20:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:27 smithi116 ceph-mon[35411]: pgmap v14413: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:20:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:29 smithi114 ceph-mon[30820]: pgmap v14414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:29 smithi116 ceph-mon[35411]: pgmap v14414: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:31 smithi114 ceph-mon[30820]: pgmap v14415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:31 smithi116 ceph-mon[35411]: pgmap v14415: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:32 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:20:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:32 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:20:33.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:20:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:33 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:20:33.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:33 smithi114 ceph-mon[30820]: pgmap v14416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:20:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:33 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:20:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:33 smithi116 ceph-mon[35411]: pgmap v14416: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:20:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:20:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:20:34.581+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:20:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:20:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:20:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:20:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:20:34.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:20:34.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:20:34.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:20:34.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:20:34.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:20:34 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:20:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:34 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:20:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:35 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:20:35.917 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:20:35.918 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:20:35.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:20:35.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:20:35.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:20:35.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:20:35.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:20:35.919 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:20:35.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:35 smithi114 ceph-mon[30820]: pgmap v14417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:20:35.919 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:35 smithi114 ceph-mon[30820]: pgmap v14418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:20:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:35 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:20:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:20:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:20:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:20:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:35 smithi116 ceph-mon[35411]: pgmap v14417: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:35 smithi116 ceph-mon[35411]: pgmap v14418: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:20:37.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:37 smithi114 ceph-mon[30820]: pgmap v14419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:20:37.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:37 smithi116 ceph-mon[35411]: pgmap v14419: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:20:39.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:39 smithi114 ceph-mon[30820]: pgmap v14420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:20:39.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:39 smithi116 ceph-mon[35411]: pgmap v14420: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:20:41.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:41 smithi114 ceph-mon[30820]: pgmap v14421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:20:41.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:41 smithi116 ceph-mon[35411]: pgmap v14421: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:20:43.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:43 smithi114 ceph-mon[30820]: pgmap v14422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:20:43.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:43 smithi116 ceph-mon[35411]: pgmap v14422: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:20:45.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:45 smithi114 ceph-mon[30820]: pgmap v14423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:20:45.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:45 smithi116 ceph-mon[35411]: pgmap v14423: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:20:47.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:47 smithi114 ceph-mon[30820]: pgmap v14424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:20:47.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:47 smithi116 ceph-mon[35411]: pgmap v14424: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:20:49.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:49 smithi116 ceph-mon[35411]: pgmap v14425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:49 smithi114 ceph-mon[30820]: pgmap v14425: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:51.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:51 smithi116 ceph-mon[35411]: pgmap v14426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:52.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:51 smithi114 ceph-mon[30820]: pgmap v14426: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:20:53.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:53 smithi116 ceph-mon[35411]: pgmap v14427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:53 smithi114 ceph-mon[30820]: pgmap v14427: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:20:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:55 smithi114 ceph-mon[30820]: pgmap v14428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:20:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:55 smithi116 ceph-mon[35411]: pgmap v14428: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:20:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:57 smithi114 ceph-mon[30820]: pgmap v14429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:20:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:57 smithi116 ceph-mon[35411]: pgmap v14429: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:20:59.969 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:20:59 smithi114 ceph-mon[30820]: pgmap v14430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:20:59 smithi116 ceph-mon[35411]: pgmap v14430: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:01 smithi114 ceph-mon[30820]: pgmap v14431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:01 smithi116 ceph-mon[35411]: pgmap v14431: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:04.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:03 smithi114 ceph-mon[30820]: pgmap v14432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:04.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:03 smithi116 ceph-mon[35411]: pgmap v14432: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:06 smithi114 ceph-mon[30820]: pgmap v14433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:21:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:06 smithi116 ceph-mon[35411]: pgmap v14433: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:21:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:08 smithi114 ceph-mon[30820]: pgmap v14434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:08 smithi116 ceph-mon[35411]: pgmap v14434: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:10 smithi114 ceph-mon[30820]: pgmap v14435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:10 smithi116 ceph-mon[35411]: pgmap v14435: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:12 smithi114 ceph-mon[30820]: pgmap v14436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:12 smithi116 ceph-mon[35411]: pgmap v14436: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:14 smithi114 ceph-mon[30820]: pgmap v14437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:14 smithi116 ceph-mon[35411]: pgmap v14437: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:16 smithi114 ceph-mon[30820]: pgmap v14438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:21:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:16 smithi116 ceph-mon[35411]: pgmap v14438: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:21:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:18 smithi114 ceph-mon[30820]: pgmap v14439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:18 smithi116 ceph-mon[35411]: pgmap v14439: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:20.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:20 smithi114 ceph-mon[30820]: pgmap v14440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:20.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:20 smithi116 ceph-mon[35411]: pgmap v14440: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:22 smithi114 ceph-mon[30820]: pgmap v14441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:22 smithi116 ceph-mon[35411]: pgmap v14441: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:24 smithi114 ceph-mon[30820]: pgmap v14442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:21:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:24 smithi116 ceph-mon[35411]: pgmap v14442: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:21:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:26 smithi114 ceph-mon[30820]: pgmap v14443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:21:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:26 smithi116 ceph-mon[35411]: pgmap v14443: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:21:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:27 smithi114 ceph-mon[30820]: pgmap v14444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:27 smithi116 ceph-mon[35411]: pgmap v14444: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:29.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:29 smithi114 ceph-mon[30820]: pgmap v14445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:21:29.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:29 smithi116 ceph-mon[35411]: pgmap v14445: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:21:31.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:31 smithi114 ceph-mon[30820]: pgmap v14446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:31.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:31 smithi116 ceph-mon[35411]: pgmap v14446: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:33.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:33 smithi116 ceph-mon[35411]: pgmap v14447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:33 smithi114 ceph-mon[30820]: pgmap v14447: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:35.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: pgmap v14448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:35.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:35 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: pgmap v14448: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:21:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:35 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-19T14:21:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:36 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T14:21:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:36 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T14:21:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:36 smithi116 ceph-mon[35411]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T14:21:36.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:36 smithi116 ceph-mon[35411]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T14:21:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:36 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi116 to 176.0M 2024-09-19T14:21:37.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:36 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi116 to 184588185: error parsing value: Value '184588185' is below minimum 939524096 2024-09-19T14:21:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:36 smithi114 ceph-mon[30820]: Adjusting osd_memory_target on smithi114 to 176.4M 2024-09-19T14:21:37.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:36 smithi114 ceph-mon[30820]: Unable to set osd_memory_target on smithi114 to 185021849: error parsing value: Value '185021849' is below minimum 939524096 2024-09-19T14:21:37.943 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-mon[30820]: pgmap v14449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:37 smithi116 ceph-mon[35411]: pgmap v14449: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:21:37.940+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:21:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:21:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:21:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:21:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:21:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:21:38.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:21:38.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:21:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:21:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:21:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:21:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:21:38.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:21:37 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:21:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:38 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:21:39.167 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:21:39.168 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:38 smithi114 ceph-mon[30820]: pgmap v14450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:38 smithi114 ceph-mon[30820]: pgmap v14451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:21:39.169 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:21:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:38 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:21:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:21:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:21:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:21:39.179 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:21:39.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:38 smithi116 ceph-mon[35411]: pgmap v14450: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:38 smithi116 ceph-mon[35411]: pgmap v14451: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:21:39.181 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:21:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:41 smithi114 ceph-mon[30820]: pgmap v14452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:21:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:41 smithi116 ceph-mon[35411]: pgmap v14452: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:21:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:43 smithi114 ceph-mon[30820]: pgmap v14453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:21:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:43 smithi116 ceph-mon[35411]: pgmap v14453: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:21:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:45 smithi114 ceph-mon[30820]: pgmap v14454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:21:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:45 smithi116 ceph-mon[35411]: pgmap v14454: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:21:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:47 smithi114 ceph-mon[30820]: pgmap v14455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:21:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:47 smithi116 ceph-mon[35411]: pgmap v14455: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:21:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:49 smithi114 ceph-mon[30820]: pgmap v14456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:21:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:49 smithi116 ceph-mon[35411]: pgmap v14456: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:21:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:51 smithi114 ceph-mon[30820]: pgmap v14457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:21:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:51 smithi116 ceph-mon[35411]: pgmap v14457: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:21:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:53 smithi114 ceph-mon[30820]: pgmap v14458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:53 smithi116 ceph-mon[35411]: pgmap v14458: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:21:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:55 smithi114 ceph-mon[30820]: pgmap v14459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:55 smithi116 ceph-mon[35411]: pgmap v14459: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:57 smithi114 ceph-mon[30820]: pgmap v14460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:57 smithi116 ceph-mon[35411]: pgmap v14460: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:21:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:21:59 smithi114 ceph-mon[30820]: pgmap v14461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:21:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:21:59 smithi116 ceph-mon[35411]: pgmap v14461: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:01 smithi114 ceph-mon[30820]: pgmap v14462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:22:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:01 smithi116 ceph-mon[35411]: pgmap v14462: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:22:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:03 smithi114 ceph-mon[30820]: pgmap v14463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:22:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:03 smithi116 ceph-mon[35411]: pgmap v14463: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:22:05.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:05 smithi114 ceph-mon[30820]: pgmap v14464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:05 smithi116 ceph-mon[35411]: pgmap v14464: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:07.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:07 smithi114 ceph-mon[30820]: pgmap v14465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:22:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:07 smithi116 ceph-mon[35411]: pgmap v14465: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:22:09.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:09 smithi114 ceph-mon[30820]: pgmap v14466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:09.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:09 smithi116 ceph-mon[35411]: pgmap v14466: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:11.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:11 smithi114 ceph-mon[30820]: pgmap v14467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:22:11.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:11 smithi116 ceph-mon[35411]: pgmap v14467: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:22:13.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:13 smithi114 ceph-mon[30820]: pgmap v14468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:22:13.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:13 smithi116 ceph-mon[35411]: pgmap v14468: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:22:15.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:15 smithi114 ceph-mon[30820]: pgmap v14469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:15.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:15 smithi116 ceph-mon[35411]: pgmap v14469: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:17.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:17 smithi114 ceph-mon[30820]: pgmap v14470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:22:17.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:17 smithi116 ceph-mon[35411]: pgmap v14470: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:22:19.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:19 smithi114 ceph-mon[30820]: pgmap v14471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:22:19.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:19 smithi116 ceph-mon[35411]: pgmap v14471: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:22:21.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:21 smithi114 ceph-mon[30820]: pgmap v14472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:22:21.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:21 smithi116 ceph-mon[35411]: pgmap v14472: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:22:23.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:23 smithi114 ceph-mon[30820]: pgmap v14473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:22:23.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:23 smithi116 ceph-mon[35411]: pgmap v14473: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:22:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:25 smithi114 ceph-mon[30820]: pgmap v14474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:22:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:25 smithi116 ceph-mon[35411]: pgmap v14474: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:22:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:27 smithi114 ceph-mon[30820]: pgmap v14475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:22:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:27 smithi116 ceph-mon[35411]: pgmap v14475: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:22:29.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:29 smithi114 ceph-mon[30820]: pgmap v14476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:29 smithi116 ceph-mon[35411]: pgmap v14476: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:31 smithi114 ceph-mon[30820]: pgmap v14477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:22:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:31 smithi116 ceph-mon[35411]: pgmap v14477: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:22:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:33 smithi114 ceph-mon[30820]: pgmap v14478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:22:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:33 smithi116 ceph-mon[35411]: pgmap v14478: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:22:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:35 smithi114 ceph-mon[30820]: pgmap v14479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:35 smithi116 ceph-mon[35411]: pgmap v14479: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:37 smithi114 ceph-mon[30820]: pgmap v14480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:22:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:37 smithi116 ceph-mon[35411]: pgmap v14480: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:22:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:38 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:22:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:38 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:22:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:39 smithi114 ceph-mon[30820]: pgmap v14481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:39 smithi116 ceph-mon[35411]: pgmap v14481: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-mon[30820]: pgmap v14482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:22:41.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:22:41.300+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:22:41.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:22:41.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:22:41.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:22:41.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:22:41.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:22:41.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:22:41.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:22:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:22:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:22:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:22:41.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:22:41 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:22:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:41 smithi116 ceph-mon[35411]: pgmap v14482: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:22:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:22:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:22:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:22:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:22:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:22:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:22:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:43 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:22:43.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:22:43.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:43 smithi114 ceph-mon[30820]: pgmap v14483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:22:43.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:43 smithi114 ceph-mon[30820]: pgmap v14484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:22:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:43 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:22:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:22:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:22:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:22:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:22:43.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:43 smithi116 ceph-mon[35411]: pgmap v14483: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:43 smithi116 ceph-mon[35411]: pgmap v14484: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:22:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:45 smithi114 ceph-mon[30820]: pgmap v14485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:22:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:45 smithi116 ceph-mon[35411]: pgmap v14485: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:22:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:47 smithi114 ceph-mon[30820]: pgmap v14486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:22:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:47 smithi116 ceph-mon[35411]: pgmap v14486: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:22:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:49 smithi114 ceph-mon[30820]: pgmap v14487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:22:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:49 smithi116 ceph-mon[35411]: pgmap v14487: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:22:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:51 smithi114 ceph-mon[30820]: pgmap v14488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:22:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:51 smithi116 ceph-mon[35411]: pgmap v14488: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:22:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:53 smithi114 ceph-mon[30820]: pgmap v14489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:22:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:53 smithi116 ceph-mon[35411]: pgmap v14489: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:22:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:55 smithi114 ceph-mon[30820]: pgmap v14490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:55 smithi116 ceph-mon[35411]: pgmap v14490: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:22:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:57 smithi114 ceph-mon[30820]: pgmap v14491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:22:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:57 smithi116 ceph-mon[35411]: pgmap v14491: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:22:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:22:59 smithi114 ceph-mon[30820]: pgmap v14492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:22:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:22:59 smithi116 ceph-mon[35411]: pgmap v14492: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:01 smithi114 ceph-mon[30820]: pgmap v14493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:01 smithi116 ceph-mon[35411]: pgmap v14493: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:03.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:03 smithi114 ceph-mon[30820]: pgmap v14494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:03.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:03 smithi116 ceph-mon[35411]: pgmap v14494: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:05 smithi116 ceph-mon[35411]: pgmap v14495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:05 smithi114 ceph-mon[30820]: pgmap v14495: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:07 smithi114 ceph-mon[30820]: pgmap v14496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:23:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:07 smithi116 ceph-mon[35411]: pgmap v14496: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:23:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:09 smithi114 ceph-mon[30820]: pgmap v14497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:09 smithi116 ceph-mon[35411]: pgmap v14497: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:11 smithi114 ceph-mon[30820]: pgmap v14498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:11 smithi116 ceph-mon[35411]: pgmap v14498: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:13 smithi114 ceph-mon[30820]: pgmap v14499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:13 smithi116 ceph-mon[35411]: pgmap v14499: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:15 smithi114 ceph-mon[30820]: pgmap v14500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:15 smithi116 ceph-mon[35411]: pgmap v14500: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:17 smithi114 ceph-mon[30820]: pgmap v14501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:23:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:17 smithi116 ceph-mon[35411]: pgmap v14501: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:23:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:19 smithi114 ceph-mon[30820]: pgmap v14502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:19 smithi116 ceph-mon[35411]: pgmap v14502: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:21 smithi114 ceph-mon[30820]: pgmap v14503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:21 smithi116 ceph-mon[35411]: pgmap v14503: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:22 smithi114 ceph-mon[30820]: pgmap v14504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:22 smithi116 ceph-mon[35411]: pgmap v14504: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:24.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:24 smithi114 ceph-mon[30820]: pgmap v14505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:24.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:24 smithi116 ceph-mon[35411]: pgmap v14505: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:26.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:26 smithi114 ceph-mon[30820]: pgmap v14506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:23:26.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:26 smithi116 ceph-mon[35411]: pgmap v14506: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:23:28.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:28 smithi114 ceph-mon[30820]: pgmap v14507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:28.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:28 smithi116 ceph-mon[35411]: pgmap v14507: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:30.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:30 smithi114 ceph-mon[30820]: pgmap v14508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:30.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:30 smithi116 ceph-mon[35411]: pgmap v14508: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:32.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:32 smithi114 ceph-mon[30820]: pgmap v14509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:32.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:32 smithi116 ceph-mon[35411]: pgmap v14509: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:34.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:34 smithi114 ceph-mon[30820]: pgmap v14510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:34.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:34 smithi116 ceph-mon[35411]: pgmap v14510: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:36.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:36 smithi114 ceph-mon[30820]: pgmap v14511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T14:23:36.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:36 smithi116 ceph-mon[35411]: pgmap v14511: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-19T14:23:38.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:38 smithi114 ceph-mon[30820]: pgmap v14512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:38.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:38 smithi116 ceph-mon[35411]: pgmap v14512: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:23:40.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:40 smithi114 ceph-mon[30820]: pgmap v14513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:40.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:40 smithi116 ceph-mon[35411]: pgmap v14513: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:42 smithi114 ceph-mon[30820]: pgmap v14514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:42.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:42 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:23:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:42 smithi116 ceph-mon[35411]: pgmap v14514: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:23:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:42 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:23:44.663 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-mon[30820]: pgmap v14515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:44.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:44 smithi116 ceph-mon[35411]: pgmap v14515: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:23:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:23:44.661+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:23:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:23:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:23:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:23:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:23:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:23:44.918 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:23:44.919 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:23:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:23:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:23:44.920 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:23:44 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:23:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:45 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:23:45.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:23:45.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:23:45.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:23:45.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:23:45.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:45 smithi114 ceph-mon[30820]: pgmap v14516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:23:45.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:45 smithi114 ceph-mon[30820]: pgmap v14517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:23:45.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:23:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:45 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:23:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:23:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:23:45.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:45 smithi116 ceph-mon[35411]: pgmap v14516: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 6 op/s 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:45 smithi116 ceph-mon[35411]: pgmap v14517: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:23:45.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:23:48.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:47 smithi114 ceph-mon[30820]: pgmap v14518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:23:48.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:47 smithi116 ceph-mon[35411]: pgmap v14518: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:23:50.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:49 smithi114 ceph-mon[30820]: pgmap v14519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:23:50.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:49 smithi116 ceph-mon[35411]: pgmap v14519: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:23:52.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:51 smithi114 ceph-mon[30820]: pgmap v14520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:23:52.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:51 smithi116 ceph-mon[35411]: pgmap v14520: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:23:54.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:53 smithi114 ceph-mon[30820]: pgmap v14521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:23:54.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:53 smithi116 ceph-mon[35411]: pgmap v14521: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:23:56.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:55 smithi114 ceph-mon[30820]: pgmap v14522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:23:56.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:55 smithi116 ceph-mon[35411]: pgmap v14522: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:23:58.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:57 smithi114 ceph-mon[30820]: pgmap v14523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:23:58.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:57 smithi116 ceph-mon[35411]: pgmap v14523: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:00.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:23:59 smithi114 ceph-mon[30820]: pgmap v14524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:00.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:23:59 smithi116 ceph-mon[35411]: pgmap v14524: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:02 smithi114 ceph-mon[30820]: pgmap v14525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:02 smithi116 ceph-mon[35411]: pgmap v14525: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:04 smithi114 ceph-mon[30820]: pgmap v14526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:04 smithi116 ceph-mon[35411]: pgmap v14526: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:06 smithi114 ceph-mon[30820]: pgmap v14527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:24:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:06 smithi116 ceph-mon[35411]: pgmap v14527: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:24:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:08 smithi114 ceph-mon[30820]: pgmap v14528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:08 smithi116 ceph-mon[35411]: pgmap v14528: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:10 smithi114 ceph-mon[30820]: pgmap v14529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:10 smithi116 ceph-mon[35411]: pgmap v14529: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:12 smithi114 ceph-mon[30820]: pgmap v14530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:12 smithi116 ceph-mon[35411]: pgmap v14530: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:14.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:14 smithi114 ceph-mon[30820]: pgmap v14531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:14.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:14 smithi116 ceph-mon[35411]: pgmap v14531: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:16.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:16 smithi114 ceph-mon[30820]: pgmap v14532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:24:16.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:16 smithi116 ceph-mon[35411]: pgmap v14532: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:24:18.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:18 smithi114 ceph-mon[30820]: pgmap v14533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:18.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:18 smithi116 ceph-mon[35411]: pgmap v14533: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:19 smithi114 ceph-mon[30820]: pgmap v14534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:19 smithi116 ceph-mon[35411]: pgmap v14534: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:21 smithi114 ceph-mon[30820]: pgmap v14535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:21 smithi116 ceph-mon[35411]: pgmap v14535: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:23 smithi114 ceph-mon[30820]: pgmap v14536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:23 smithi116 ceph-mon[35411]: pgmap v14536: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:26.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:25 smithi114 ceph-mon[30820]: pgmap v14537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:24:26.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:25 smithi116 ceph-mon[35411]: pgmap v14537: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:24:28.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:27 smithi114 ceph-mon[30820]: pgmap v14538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:28.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:27 smithi116 ceph-mon[35411]: pgmap v14538: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:30.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:29 smithi114 ceph-mon[30820]: pgmap v14539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:30.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:29 smithi116 ceph-mon[35411]: pgmap v14539: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:32.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:31 smithi114 ceph-mon[30820]: pgmap v14540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:32.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:31 smithi116 ceph-mon[35411]: pgmap v14540: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:34.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:33 smithi114 ceph-mon[30820]: pgmap v14541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:24:34.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:33 smithi116 ceph-mon[35411]: pgmap v14541: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:24:36.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:35 smithi114 ceph-mon[30820]: pgmap v14542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:24:36.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:35 smithi116 ceph-mon[35411]: pgmap v14542: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 6 op/s 2024-09-19T14:24:38.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:37 smithi114 ceph-mon[30820]: pgmap v14543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:38.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:37 smithi116 ceph-mon[35411]: pgmap v14543: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:40.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:39 smithi114 ceph-mon[30820]: pgmap v14544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:24:40.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:39 smithi116 ceph-mon[35411]: pgmap v14544: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-09-19T14:24:42.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:41 smithi114 ceph-mon[30820]: pgmap v14545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:42.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:41 smithi116 ceph-mon[35411]: pgmap v14545: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:24:44.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:43 smithi114 ceph-mon[30820]: pgmap v14546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:44.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:43 smithi116 ceph-mon[35411]: pgmap v14546: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:24:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:45 smithi114 ceph-mon[30820]: pgmap v14547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:24:46.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:45 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:24:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:45 smithi116 ceph-mon[35411]: pgmap v14547: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:24:46.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:45 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:24:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:47.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:47 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:47.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:47 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-mon[30820]: pgmap v14548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:24:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:24:48.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:48.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:24:48.020+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:24:48.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:24:48.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:24:48.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:24:48.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:24:48.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:24:48.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:24:48.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:24:48.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:24:48.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:24:48.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:24:48 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:24:48.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:48 smithi116 ceph-mon[35411]: pgmap v14548: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:24:48.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:24:48.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:24:48.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:48 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:24:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:49 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:24:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:24:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:24:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:24:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:24:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:24:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:24:49.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:24:49.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:24:49.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:49 smithi114 ceph-mon[30820]: pgmap v14549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:24:49.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:49 smithi114 ceph-mon[30820]: pgmap v14550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:49 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:24:49.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:24:49.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:24:49.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:49 smithi116 ceph-mon[35411]: pgmap v14549: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 5 op/s 2024-09-19T14:24:49.682 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:49 smithi116 ceph-mon[35411]: pgmap v14550: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:24:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:51 smithi114 ceph-mon[30820]: pgmap v14551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:24:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:51 smithi116 ceph-mon[35411]: pgmap v14551: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:24:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:53 smithi114 ceph-mon[30820]: pgmap v14552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:24:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:53 smithi116 ceph-mon[35411]: pgmap v14552: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:24:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:55 smithi114 ceph-mon[30820]: pgmap v14553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:24:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:55 smithi116 ceph-mon[35411]: pgmap v14553: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:24:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:57 smithi114 ceph-mon[30820]: pgmap v14554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:24:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:57 smithi116 ceph-mon[35411]: pgmap v14554: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:24:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:24:59 smithi114 ceph-mon[30820]: pgmap v14555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:24:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:24:59 smithi116 ceph-mon[35411]: pgmap v14555: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:25:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:01 smithi114 ceph-mon[30820]: pgmap v14556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:01 smithi116 ceph-mon[35411]: pgmap v14556: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:03 smithi114 ceph-mon[30820]: pgmap v14557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:03 smithi116 ceph-mon[35411]: pgmap v14557: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:05 smithi114 ceph-mon[30820]: pgmap v14558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:05 smithi116 ceph-mon[35411]: pgmap v14558: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:07 smithi114 ceph-mon[30820]: pgmap v14559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:07 smithi116 ceph-mon[35411]: pgmap v14559: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:09 smithi114 ceph-mon[30820]: pgmap v14560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:09 smithi116 ceph-mon[35411]: pgmap v14560: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:11 smithi114 ceph-mon[30820]: pgmap v14561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:11 smithi116 ceph-mon[35411]: pgmap v14561: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:13 smithi114 ceph-mon[30820]: pgmap v14562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:13 smithi116 ceph-mon[35411]: pgmap v14562: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:15 smithi114 ceph-mon[30820]: pgmap v14563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:15 smithi116 ceph-mon[35411]: pgmap v14563: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:17 smithi114 ceph-mon[30820]: pgmap v14564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:17 smithi116 ceph-mon[35411]: pgmap v14564: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:19.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:19 smithi114 ceph-mon[30820]: pgmap v14565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:19 smithi116 ceph-mon[35411]: pgmap v14565: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:21 smithi114 ceph-mon[30820]: pgmap v14566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:21 smithi116 ceph-mon[35411]: pgmap v14566: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:22.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:22 smithi114 ceph-mon[30820]: pgmap v14567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:22.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:22 smithi116 ceph-mon[35411]: pgmap v14567: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:25.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:25 smithi114 ceph-mon[30820]: pgmap v14568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:25.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:25 smithi116 ceph-mon[35411]: pgmap v14568: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:27.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:27 smithi114 ceph-mon[30820]: pgmap v14569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:27.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:27 smithi116 ceph-mon[35411]: pgmap v14569: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:29.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:29 smithi114 ceph-mon[30820]: pgmap v14570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:29.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:29 smithi116 ceph-mon[35411]: pgmap v14570: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:31.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:31 smithi114 ceph-mon[30820]: pgmap v14571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:31.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:31 smithi116 ceph-mon[35411]: pgmap v14571: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:33.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:33 smithi114 ceph-mon[30820]: pgmap v14572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:33.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:33 smithi116 ceph-mon[35411]: pgmap v14572: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:35.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:35 smithi114 ceph-mon[30820]: pgmap v14573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:35.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:35 smithi116 ceph-mon[35411]: pgmap v14573: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:37.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:37 smithi114 ceph-mon[30820]: pgmap v14574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:37.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:37 smithi116 ceph-mon[35411]: pgmap v14574: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:39.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:39 smithi114 ceph-mon[30820]: pgmap v14575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:39.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:39 smithi116 ceph-mon[35411]: pgmap v14575: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:41.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:41 smithi114 ceph-mon[30820]: pgmap v14576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:41.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:41 smithi116 ceph-mon[35411]: pgmap v14576: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:43.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:43 smithi114 ceph-mon[30820]: pgmap v14577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:43.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:43 smithi116 ceph-mon[35411]: pgmap v14577: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:25:45.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:45 smithi114 ceph-mon[30820]: pgmap v14578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:45.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:45 smithi116 ceph-mon[35411]: pgmap v14578: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:47.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:47 smithi114 ceph-mon[30820]: pgmap v14579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:47.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:47 smithi116 ceph-mon[35411]: pgmap v14579: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:25:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:49 smithi114 ceph-mon[30820]: pgmap v14580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:49.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:25:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:49.418 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:49 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:49 smithi116 ceph-mon[35411]: pgmap v14580: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:25:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:25:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:49.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:49.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:49 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:51.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-mon[30820]: pgmap v14581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:51.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:25:51.380+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:25:51.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:25:51.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:25:51.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:25:51.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:25:51.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:25:51.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:25:51.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:25:51.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:25:51.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:25:51.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:25:51 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:25:51.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:51 smithi116 ceph-mon[35411]: pgmap v14581: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:25:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:25:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:25:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:52.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:25:52.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:25:52.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:25:53.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:53 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:25:53.418 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:25:53.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:53 smithi114 ceph-mon[30820]: pgmap v14582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:25:53.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:53 smithi114 ceph-mon[30820]: pgmap v14583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:25:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:53 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:25:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:25:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:25:53.429 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:25:53.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:53 smithi116 ceph-mon[35411]: pgmap v14582: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:25:53.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:53 smithi116 ceph-mon[35411]: pgmap v14583: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:25:55.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:55 smithi114 ceph-mon[30820]: pgmap v14584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:25:55.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:55 smithi116 ceph-mon[35411]: pgmap v14584: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:25:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:57 smithi114 ceph-mon[30820]: pgmap v14585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:25:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:57 smithi116 ceph-mon[35411]: pgmap v14585: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5 op/s 2024-09-19T14:25:59.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:25:59 smithi114 ceph-mon[30820]: pgmap v14586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:25:59.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:25:59 smithi116 ceph-mon[35411]: pgmap v14586: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:26:01.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:01 smithi114 ceph-mon[30820]: pgmap v14587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:26:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:01 smithi116 ceph-mon[35411]: pgmap v14587: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:26:03.430 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:03 smithi116 ceph-mon[35411]: pgmap v14588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:26:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:03 smithi114 ceph-mon[30820]: pgmap v14588: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:26:05.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:05 smithi116 ceph-mon[35411]: pgmap v14589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:05 smithi114 ceph-mon[30820]: pgmap v14589: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:07.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:07 smithi116 ceph-mon[35411]: pgmap v14590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:07 smithi114 ceph-mon[30820]: pgmap v14590: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:09 smithi114 ceph-mon[30820]: pgmap v14591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:09 smithi116 ceph-mon[35411]: pgmap v14591: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:11 smithi114 ceph-mon[30820]: pgmap v14592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:11 smithi116 ceph-mon[35411]: pgmap v14592: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:13 smithi114 ceph-mon[30820]: pgmap v14593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:13 smithi116 ceph-mon[35411]: pgmap v14593: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:15.482 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:15 smithi116 ceph-mon[35411]: pgmap v14594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:15 smithi114 ceph-mon[30820]: pgmap v14594: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:17 smithi114 ceph-mon[30820]: pgmap v14595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:17 smithi116 ceph-mon[35411]: pgmap v14595: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:19 smithi114 ceph-mon[30820]: pgmap v14596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:19 smithi116 ceph-mon[35411]: pgmap v14596: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:21 smithi114 ceph-mon[30820]: pgmap v14597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:21 smithi116 ceph-mon[35411]: pgmap v14597: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:23 smithi114 ceph-mon[30820]: pgmap v14598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:23 smithi116 ceph-mon[35411]: pgmap v14598: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:25 smithi114 ceph-mon[30820]: pgmap v14599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:25 smithi116 ceph-mon[35411]: pgmap v14599: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:27 smithi114 ceph-mon[30820]: pgmap v14600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:27 smithi116 ceph-mon[35411]: pgmap v14600: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:29 smithi114 ceph-mon[30820]: pgmap v14601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:29 smithi116 ceph-mon[35411]: pgmap v14601: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:31 smithi114 ceph-mon[30820]: pgmap v14602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:31 smithi116 ceph-mon[35411]: pgmap v14602: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:33 smithi114 ceph-mon[30820]: pgmap v14603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:33 smithi116 ceph-mon[35411]: pgmap v14603: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:35 smithi114 ceph-mon[30820]: pgmap v14604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:35 smithi116 ceph-mon[35411]: pgmap v14604: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:37 smithi114 ceph-mon[30820]: pgmap v14605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:37 smithi116 ceph-mon[35411]: pgmap v14605: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:39 smithi114 ceph-mon[30820]: pgmap v14606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:39 smithi116 ceph-mon[35411]: pgmap v14606: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:41 smithi114 ceph-mon[30820]: pgmap v14607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:41 smithi116 ceph-mon[35411]: pgmap v14607: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:43.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:43 smithi114 ceph-mon[30820]: pgmap v14608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:43.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:43 smithi116 ceph-mon[35411]: pgmap v14608: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:45 smithi114 ceph-mon[30820]: pgmap v14609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:45 smithi116 ceph-mon[35411]: pgmap v14609: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:47 smithi114 ceph-mon[30820]: pgmap v14610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:47 smithi116 ceph-mon[35411]: pgmap v14610: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:49 smithi114 ceph-mon[30820]: pgmap v14611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:49 smithi116 ceph-mon[35411]: pgmap v14611: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-19T14:26:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:51 smithi114 ceph-mon[30820]: pgmap v14612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:51 smithi116 ceph-mon[35411]: pgmap v14612: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:26:52.555 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:52 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:26:52.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:52 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:26:53.562 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:53 smithi116 ceph-mon[35411]: pgmap v14613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:53.657 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:53 smithi114 ceph-mon[30820]: pgmap v14613: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:26:54.834 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:26:54.740+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:26:54.834 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:26:54.834 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:26:54.835 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:26:54.836 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:26:54.836 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:26:54.836 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:26:54.836 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:26:54.836 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:26:54.836 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:26:54.836 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:26:54 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:26:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:55 smithi114 ceph-mon[30820]: pgmap v14614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:55 smithi116 ceph-mon[35411]: pgmap v14614: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:26:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:56 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:26:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:26:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:26:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:26:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:26:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:26:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:26:56.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:26:56.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:26:56.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:26:56.670 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:26:56.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:56.670 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:56 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:56.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:56 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:26:56.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:26:56.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:26:56.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:26:56.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:26:56.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:26:56.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:26:56.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:26:56.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:26:56.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:26:56.682 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:26:56.683 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:26:56.683 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:26:56.683 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:26:56.683 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:26:56.683 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:56.683 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:56 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:57.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:57 smithi114 ceph-mon[30820]: pgmap v14615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:57.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:57.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:57.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:26:57.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:26:57.570 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:57 smithi116 ceph-mon[35411]: pgmap v14615: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 7 op/s 2024-09-19T14:26:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:26:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:26:57.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:26:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:58 smithi114 ceph-mon[30820]: pgmap v14616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:26:58.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:26:58 smithi114 ceph-mon[30820]: pgmap v14617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:26:58.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:58 smithi116 ceph-mon[35411]: pgmap v14616: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:26:58.680 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:26:58 smithi116 ceph-mon[35411]: pgmap v14617: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:27:00.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:00 smithi114 ceph-mon[30820]: pgmap v14618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:27:00.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:00 smithi116 ceph-mon[35411]: pgmap v14618: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:27:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:02 smithi114 ceph-mon[30820]: pgmap v14619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:27:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:02 smithi116 ceph-mon[35411]: pgmap v14619: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:27:04.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:04 smithi114 ceph-mon[30820]: pgmap v14620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:27:04.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:04 smithi116 ceph-mon[35411]: pgmap v14620: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:27:06.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:06 smithi114 ceph-mon[30820]: pgmap v14621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:27:06.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:06 smithi116 ceph-mon[35411]: pgmap v14621: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-09-19T14:27:08.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:08 smithi114 ceph-mon[30820]: pgmap v14622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:27:08.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:08 smithi116 ceph-mon[35411]: pgmap v14622: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:27:10.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:10 smithi114 ceph-mon[30820]: pgmap v14623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:10.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:10 smithi116 ceph-mon[35411]: pgmap v14623: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:12 smithi114 ceph-mon[30820]: pgmap v14624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:12 smithi116 ceph-mon[35411]: pgmap v14624: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:13 smithi114 ceph-mon[30820]: pgmap v14625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:13 smithi116 ceph-mon[35411]: pgmap v14625: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:16.383 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:15 smithi116 ceph-mon[35411]: pgmap v14626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:16.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:15 smithi114 ceph-mon[30820]: pgmap v14626: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:18.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:17 smithi114 ceph-mon[30820]: pgmap v14627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:18.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:17 smithi116 ceph-mon[35411]: pgmap v14627: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:20.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:19 smithi114 ceph-mon[30820]: pgmap v14628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:20.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:19 smithi116 ceph-mon[35411]: pgmap v14628: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:22.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:21 smithi114 ceph-mon[30820]: pgmap v14629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:22.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:21 smithi116 ceph-mon[35411]: pgmap v14629: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:24.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:23 smithi114 ceph-mon[30820]: pgmap v14630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:24.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:23 smithi116 ceph-mon[35411]: pgmap v14630: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:26.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:25 smithi114 ceph-mon[30820]: pgmap v14631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:26.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:25 smithi116 ceph-mon[35411]: pgmap v14631: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:28.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:27 smithi114 ceph-mon[30820]: pgmap v14632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:28.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:27 smithi116 ceph-mon[35411]: pgmap v14632: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:30.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:29 smithi114 ceph-mon[30820]: pgmap v14633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:30.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:29 smithi116 ceph-mon[35411]: pgmap v14633: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:32.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:31 smithi114 ceph-mon[30820]: pgmap v14634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:32.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:31 smithi116 ceph-mon[35411]: pgmap v14634: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:34.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:33 smithi114 ceph-mon[30820]: pgmap v14635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:34.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:33 smithi116 ceph-mon[35411]: pgmap v14635: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:36.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:35 smithi114 ceph-mon[30820]: pgmap v14636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:36.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:35 smithi116 ceph-mon[35411]: pgmap v14636: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:38.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:37 smithi114 ceph-mon[30820]: pgmap v14637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:38.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:37 smithi116 ceph-mon[35411]: pgmap v14637: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:40.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:39 smithi114 ceph-mon[30820]: pgmap v14638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:40.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:39 smithi116 ceph-mon[35411]: pgmap v14638: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:42.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:41 smithi114 ceph-mon[30820]: pgmap v14639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:42.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:41 smithi116 ceph-mon[35411]: pgmap v14639: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:44.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:43 smithi114 ceph-mon[30820]: pgmap v14640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:44.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:43 smithi116 ceph-mon[35411]: pgmap v14640: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:46.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:45 smithi114 ceph-mon[30820]: pgmap v14641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:46.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:46 smithi116 ceph-mon[35411]: pgmap v14641: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:48.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:48 smithi114 ceph-mon[30820]: pgmap v14642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:48.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:48 smithi116 ceph-mon[35411]: pgmap v14642: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:50.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:50 smithi114 ceph-mon[30820]: pgmap v14643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:50.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:50 smithi116 ceph-mon[35411]: pgmap v14643: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:52.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:52 smithi114 ceph-mon[30820]: pgmap v14644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:52.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:52 smithi116 ceph-mon[35411]: pgmap v14644: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:27:54.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:54 smithi114 ceph-mon[30820]: pgmap v14645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:54.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:54 smithi116 ceph-mon[35411]: pgmap v14645: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:27:56.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:56 smithi114 ceph-mon[30820]: pgmap v14646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:56.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:56 smithi116 ceph-mon[35411]: pgmap v14646: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:27:57.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:57 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:27:57.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:57 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:27:58.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:27:58 smithi114 ceph-mon[30820]: pgmap v14647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:27:58.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:27:58 smithi116 ceph-mon[35411]: pgmap v14647: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:00.417 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-mon[30820]: pgmap v14648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:28:00.212+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:28:00.418 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:28:00.419 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:28:00.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:28:00.420 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:28:00 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:28:00.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:00 smithi116 ceph-mon[35411]: pgmap v14648: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:01.420 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:01 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:28:01.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:28:01.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:28:01.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:28:01.420 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:28:01.421 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:01 smithi114 ceph-mon[30820]: pgmap v14649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:01 smithi114 ceph-mon[30820]: pgmap v14650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:28:01.422 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:28:01.429 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:01 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:28:01.430 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:01 smithi116 ceph-mon[35411]: pgmap v14649: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:01 smithi116 ceph-mon[35411]: pgmap v14650: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:28:01.431 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:28:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:03 smithi114 ceph-mon[30820]: pgmap v14651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:28:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:03 smithi116 ceph-mon[35411]: pgmap v14651: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:28:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:05 smithi114 ceph-mon[30820]: pgmap v14652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:28:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:05 smithi116 ceph-mon[35411]: pgmap v14652: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:28:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:07 smithi114 ceph-mon[30820]: pgmap v14653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:28:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:07 smithi116 ceph-mon[35411]: pgmap v14653: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:28:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:09 smithi114 ceph-mon[30820]: pgmap v14654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:28:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:09 smithi116 ceph-mon[35411]: pgmap v14654: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:28:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:11 smithi114 ceph-mon[30820]: pgmap v14655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:28:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:11 smithi116 ceph-mon[35411]: pgmap v14655: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:28:13.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:13 smithi114 ceph-mon[30820]: pgmap v14656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:13.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:13 smithi116 ceph-mon[35411]: pgmap v14656: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:15.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:15 smithi114 ceph-mon[30820]: pgmap v14657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:15.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:15 smithi116 ceph-mon[35411]: pgmap v14657: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:17.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:17 smithi114 ceph-mon[30820]: pgmap v14658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:17.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:17 smithi116 ceph-mon[35411]: pgmap v14658: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:19.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:19 smithi114 ceph-mon[30820]: pgmap v14659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:19.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:19 smithi116 ceph-mon[35411]: pgmap v14659: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:21.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:21 smithi114 ceph-mon[30820]: pgmap v14660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:28:21.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:21 smithi116 ceph-mon[35411]: pgmap v14660: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:28:23.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:23 smithi114 ceph-mon[30820]: pgmap v14661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:23.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:23 smithi116 ceph-mon[35411]: pgmap v14661: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:25.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:25 smithi114 ceph-mon[30820]: pgmap v14662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:25.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:25 smithi116 ceph-mon[35411]: pgmap v14662: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:27.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:27 smithi114 ceph-mon[30820]: pgmap v14663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:27.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:27 smithi116 ceph-mon[35411]: pgmap v14663: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:29.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:29 smithi114 ceph-mon[30820]: pgmap v14664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:29.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:29 smithi116 ceph-mon[35411]: pgmap v14664: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:31.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:31 smithi114 ceph-mon[30820]: pgmap v14665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:28:31.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:31 smithi116 ceph-mon[35411]: pgmap v14665: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:28:33.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:33 smithi114 ceph-mon[30820]: pgmap v14666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:33.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:33 smithi116 ceph-mon[35411]: pgmap v14666: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:35.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:35 smithi114 ceph-mon[30820]: pgmap v14667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:35.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:35 smithi116 ceph-mon[35411]: pgmap v14667: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:37.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:37 smithi114 ceph-mon[30820]: pgmap v14668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:37.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:37 smithi116 ceph-mon[35411]: pgmap v14668: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:39.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:39 smithi114 ceph-mon[30820]: pgmap v14669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:39.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:39 smithi116 ceph-mon[35411]: pgmap v14669: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:41.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:41 smithi114 ceph-mon[30820]: pgmap v14670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:28:41.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:41 smithi116 ceph-mon[35411]: pgmap v14670: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:28:42.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:42 smithi114 ceph-mon[30820]: pgmap v14671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:42.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:42 smithi116 ceph-mon[35411]: pgmap v14671: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:45.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:45 smithi114 ceph-mon[30820]: pgmap v14672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:45.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:45 smithi116 ceph-mon[35411]: pgmap v14672: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:47.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:47 smithi114 ceph-mon[30820]: pgmap v14673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:47.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:47 smithi116 ceph-mon[35411]: pgmap v14673: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:49.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:49 smithi114 ceph-mon[30820]: pgmap v14674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:49.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:49 smithi116 ceph-mon[35411]: pgmap v14674: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:51.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:51 smithi114 ceph-mon[30820]: pgmap v14675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:28:51.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:51 smithi116 ceph-mon[35411]: pgmap v14675: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:28:53.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:53 smithi114 ceph-mon[30820]: pgmap v14676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:53.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:53 smithi116 ceph-mon[35411]: pgmap v14676: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:28:55.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:55 smithi114 ceph-mon[30820]: pgmap v14677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:55.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:55 smithi116 ceph-mon[35411]: pgmap v14677: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:57.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:57 smithi114 ceph-mon[30820]: pgmap v14678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:57.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:57 smithi116 ceph-mon[35411]: pgmap v14678: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:28:59.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:28:59 smithi114 ceph-mon[30820]: pgmap v14679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:28:59.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:28:59 smithi116 ceph-mon[35411]: pgmap v14679: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-mon[30820]: pgmap v14680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:01.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:29:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:29:01.525+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:29:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:29:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:29:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:29:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:29:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:29:01.668 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:29:01.669 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:29:01 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:29:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:01 smithi116 ceph-mon[35411]: pgmap v14680: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:01.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:29:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:29:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:29:02.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:02 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:29:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:29:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:29:02.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:02 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:29:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:03 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:29:03.667 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:29:03.668 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:03 smithi114 ceph-mon[30820]: pgmap v14681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:29:03.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:03 smithi114 ceph-mon[30820]: pgmap v14682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:03 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:29:03.679 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:29:03.680 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:03 smithi116 ceph-mon[35411]: pgmap v14681: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 6 op/s 2024-09-19T14:29:03.681 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:03 smithi116 ceph-mon[35411]: pgmap v14682: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:05.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:05 smithi114 ceph-mon[30820]: pgmap v14683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:05.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:05 smithi116 ceph-mon[35411]: pgmap v14683: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:07.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:07 smithi114 ceph-mon[30820]: pgmap v14684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:07.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:07 smithi116 ceph-mon[35411]: pgmap v14684: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:09.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:09 smithi114 ceph-mon[30820]: pgmap v14685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:09.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:09 smithi116 ceph-mon[35411]: pgmap v14685: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:11.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:11 smithi114 ceph-mon[30820]: pgmap v14686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:11.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:11 smithi116 ceph-mon[35411]: pgmap v14686: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:29:12.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:12 smithi114 ceph-mon[30820]: pgmap v14687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:29:12.679 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:12 smithi116 ceph-mon[35411]: pgmap v14687: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:29:14.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:14 smithi114 ceph-mon[30820]: pgmap v14688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:14.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:14 smithi116 ceph-mon[35411]: pgmap v14688: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:16.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:16 smithi114 ceph-mon[30820]: pgmap v14689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:16.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:16 smithi116 ceph-mon[35411]: pgmap v14689: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:18.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:18 smithi114 ceph-mon[30820]: pgmap v14690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:18.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:18 smithi116 ceph-mon[35411]: pgmap v14690: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:20.918 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:20 smithi114 ceph-mon[30820]: pgmap v14691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:20.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:20 smithi116 ceph-mon[35411]: pgmap v14691: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:22.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:22 smithi114 ceph-mon[30820]: pgmap v14692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:29:22.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:22 smithi116 ceph-mon[35411]: pgmap v14692: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:29:24.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:24 smithi114 ceph-mon[30820]: pgmap v14693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:24.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:24 smithi116 ceph-mon[35411]: pgmap v14693: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:26.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:26 smithi114 ceph-mon[30820]: pgmap v14694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:26.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:26 smithi116 ceph-mon[35411]: pgmap v14694: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:28.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:28 smithi114 ceph-mon[30820]: pgmap v14695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:28.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:28 smithi116 ceph-mon[35411]: pgmap v14695: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:30.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:30 smithi114 ceph-mon[30820]: pgmap v14696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:30.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:30 smithi116 ceph-mon[35411]: pgmap v14696: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:32.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:32 smithi114 ceph-mon[30820]: pgmap v14697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:29:32.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:32 smithi116 ceph-mon[35411]: pgmap v14697: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:29:34.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:34 smithi114 ceph-mon[30820]: pgmap v14698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:34.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:34 smithi116 ceph-mon[35411]: pgmap v14698: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:36.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:36 smithi114 ceph-mon[30820]: pgmap v14699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:36.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:36 smithi116 ceph-mon[35411]: pgmap v14699: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:38.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:38 smithi114 ceph-mon[30820]: pgmap v14700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:38.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:38 smithi116 ceph-mon[35411]: pgmap v14700: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:40.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:40 smithi114 ceph-mon[30820]: pgmap v14701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:40.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:40 smithi116 ceph-mon[35411]: pgmap v14701: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:42.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:42 smithi114 ceph-mon[30820]: pgmap v14702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:29:42.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:42 smithi116 ceph-mon[35411]: pgmap v14702: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:29:44.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:44 smithi114 ceph-mon[30820]: pgmap v14703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:44.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:44 smithi116 ceph-mon[35411]: pgmap v14703: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:46.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:46 smithi114 ceph-mon[30820]: pgmap v14704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:46.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:46 smithi116 ceph-mon[35411]: pgmap v14704: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:48.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:48 smithi114 ceph-mon[30820]: pgmap v14705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:48.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:48 smithi116 ceph-mon[35411]: pgmap v14705: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:50.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:50 smithi114 ceph-mon[30820]: pgmap v14706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:50.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:50 smithi116 ceph-mon[35411]: pgmap v14706: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:29:52.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:52 smithi114 ceph-mon[30820]: pgmap v14707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:29:52.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:52 smithi116 ceph-mon[35411]: pgmap v14707: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:29:54.917 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:54 smithi114 ceph-mon[30820]: pgmap v14708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:54.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:54 smithi116 ceph-mon[35411]: pgmap v14708: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:56.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:56 smithi116 ceph-mon[35411]: pgmap v14709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:57.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:56 smithi114 ceph-mon[30820]: pgmap v14709: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:29:58.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:29:58 smithi116 ceph-mon[35411]: pgmap v14710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:29:59.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:29:58 smithi114 ceph-mon[30820]: pgmap v14710: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:30:00.929 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:00 smithi116 ceph-mon[35411]: pgmap v14711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:30:00.930 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:00 smithi116 ceph-mon[35411]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T14:30:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:00 smithi114 ceph-mon[30820]: pgmap v14711: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-09-19T14:30:01.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:00 smithi114 ceph-mon[30820]: overall HEALTH_WARN 1 hosts fail cephadm check; 1/3 mons down, quorum a,b; Reduced data availability: 3 pgs inactive, 3 pgs down, 2 pgs stale 2024-09-19T14:30:02.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:01 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:30:02.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:01 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-19T14:30:03.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:02 smithi114 ceph-mon[30820]: pgmap v14712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:30:03.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:02 smithi116 ceph-mon[35411]: pgmap v14712: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:30:05.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-mon[30820]: pgmap v14713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: 2024-09-19T14:30:04.820+0000 7fe90d200640 -1 log_channel(cephadm) log [ERR] : Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: Traceback (most recent call last): 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: yield 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await asyncio.wait_for( 2024-09-19T14:30:05.168 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return fut.result() 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: _, session = await loop.create_connection( 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise exceptions[0] 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: sock = await self._connect_sock( 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: await self.sock_connect(sock, address) 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: return await fut 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:30:05.169 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Sep 19 14:30:04 smithi114 ceph-fface640-7651-11ef-bceb-c7b262605968-mgr-x[31044]: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:30:05.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:04 smithi116 ceph-mon[35411]: pgmap v14713: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:30:06.069 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:05 smithi114 ceph-mon[30820]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:30:06.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: Traceback (most recent call last): 2024-09-19T14:30:06.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:30:06.069 INFO:journalctl@ceph.mon.a.smithi114.stdout: yield 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await asyncio.wait_for( 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: return fut.result() 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: _, session = await loop.create_connection( 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise exceptions[0] 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: sock = await self._connect_sock( 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:30:06.070 INFO:journalctl@ceph.mon.a.smithi114.stdout: await self.sock_connect(sock, address) 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: return await fut 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:05 smithi114 ceph-mon[30820]: pgmap v14714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:05 smithi114 ceph-mon[30820]: pgmap v14715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:30:06.071 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:05 smithi114 ceph-mon[30820]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:30:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:05 smithi116 ceph-mon[35411]: Can't communicate with remote host `172.21.15.118`, possibly because the host is not reachable or python3 is not installed on the host. [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:30:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: Traceback (most recent call last): 2024-09-19T14:30:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 195, in redirect_log 2024-09-19T14:30:06.180 INFO:journalctl@ceph.mon.b.smithi116.stdout: yield 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 172, in _remote_connection 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await asyncio.wait_for( 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/tasks.py", line 479, in wait_for 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: return fut.result() 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 430, in _connect 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: _, session = await loop.create_connection( 2024-09-19T14:30:06.181 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1065, in create_connection 2024-09-19T14:30:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise exceptions[0] 2024-09-19T14:30:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 1050, in create_connection 2024-09-19T14:30:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: sock = await self._connect_sock( 2024-09-19T14:30:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/base_events.py", line 961, in _connect_sock 2024-09-19T14:30:06.182 INFO:journalctl@ceph.mon.b.smithi116.stdout: await self.sock_connect(sock, address) 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 500, in sock_connect 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: return await fut 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: File "/lib64/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: raise OSError(err, f'Connect call failed {address}') 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout: OSError: [Errno 113] Connect call failed ('172.21.15.118', 22) 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:05 smithi116 ceph-mon[35411]: pgmap v14714: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6 op/s 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:05 smithi116 ceph-mon[35411]: pgmap v14715: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-09-19T14:30:06.183 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:05 smithi116 ceph-mon[35411]: from='mgr.14150 172.21.15.114:0/333341980' entity='mgr.x' 2024-09-19T14:30:08.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:07 smithi114 ceph-mon[30820]: pgmap v14716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:30:08.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:07 smithi116 ceph-mon[35411]: pgmap v14716: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:30:10.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:09 smithi114 ceph-mon[30820]: pgmap v14717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:30:10.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:09 smithi116 ceph-mon[35411]: pgmap v14717: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4 op/s 2024-09-19T14:30:12.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:11 smithi114 ceph-mon[30820]: pgmap v14718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:30:12.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:11 smithi116 ceph-mon[35411]: pgmap v14718: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:30:14.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:13 smithi114 ceph-mon[30820]: pgmap v14719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:30:14.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:13 smithi116 ceph-mon[35411]: pgmap v14719: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6 op/s 2024-09-19T14:30:16.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:15 smithi114 ceph-mon[30820]: pgmap v14720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:30:16.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:15 smithi116 ceph-mon[35411]: pgmap v14720: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 5 op/s 2024-09-19T14:30:18.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:17 smithi114 ceph-mon[30820]: pgmap v14721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:30:18.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:17 smithi116 ceph-mon[35411]: pgmap v14721: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 6 op/s 2024-09-19T14:30:20.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:19 smithi114 ceph-mon[30820]: pgmap v14722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:30:20.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:19 smithi116 ceph-mon[35411]: pgmap v14722: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:30:22.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:21 smithi114 ceph-mon[30820]: pgmap v14723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:30:22.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:21 smithi116 ceph-mon[35411]: pgmap v14723: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-19T14:30:24.167 INFO:journalctl@ceph.mon.a.smithi114.stdout:Sep 19 14:30:23 smithi114 ceph-mon[30820]: pgmap v14724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:30:24.179 INFO:journalctl@ceph.mon.b.smithi116.stdout:Sep 19 14:30:23 smithi116 ceph-mon[35411]: pgmap v14724: 129 pgs: 3 down, 2 stale+active+clean, 124 active+clean; 9.4 GiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-19T14:30:25.184 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-09-19T14:30:25.185 DEBUG:teuthology.task.console_log:Killing console logger for smithi114 2024-09-19T14:30:25.186 DEBUG:teuthology.task.console_log:Killing console logger for smithi116 2024-09-19T14:30:25.186 DEBUG:teuthology.task.console_log:Killing console logger for smithi118 2024-09-19T14:30:25.186 DEBUG:teuthology.exit:Finished running handlers